A closer look at
our living planet with Tower Services

What is STACD?

Geospatial datasets have complex lineages that are crucial for reproducibility and understanding data provenance, yet current metadata standards like STAC (SpatioTemporal Asset Catalog) provide limited support for capturing complete processing workflows. We propose STACD (STAC extension with DAGs), an extension to STAC specifications that incorporates Directed Acyclic Graph (DAG) representations along with defining algorithms and version changes in the workflows. We also provide a reference implementation on Apache Airflow to demonstrate STACD capabilities such as selective recomputation when some datasets or algorithms in a DAG are updated, complete lineage construction for a dataset, and opportunities for improved collaboration and distributed processing that arise with this standard.

How a run flow through the platform?

All compute goes through Airflow and STACD. Onboarding a service needs three YAML files: DAG, algorithm repo and dataset repo.

Key Features

• Selective Recomputation: When an algorithm is updated, STACD re-runs only the affected steps instead of processing the entire workflow again.
• Automated Workflows: Developers define pipelines using simple YAML configurations, which automatically compile into executable Apache Airflow DAGs.
• Full Data Provenance: Tracks dataset versions and history across multiple computational tools and isolated Docker containers.

Real-World Impact

STACD serves as the data orchestration backbone for the CoRE Stack project at IIT Delhi, powering drone canopy processing, bioacoustics monitoring, and landscape mapping across more than 1,000 villages in India.

Overview

All services share one compute backbone. GPU-equipped compute nodes run the heavy workloads, Kubernetes orchestrates containers across those nodes, Airflow runs every compute job as a DAG, and STACD generates those DAGs so services do not hand-write them. Users never talk to Airflow directly: they use a service UI, and the service triggers the job and tracks it.

How a Compute Job Runs

1. The user works in a service's UI, reached through Nginx.
2. The service triggers an Airflow DAG generated by STACD.
3. Airflow hands the tasks to Kubernetes, which schedules them on GPU nodes.
4. The service polls the DAG run until it ends in success or failed.
5. On success, output is written to data/<app-name>/.
6. The user browses and downloads results in the shared FileBrowser.

Components

Design Principles

• One path for all compute: every job goes through Airflow, so runs are tracked and reproducible.
• Separation of concerns: UI services trigger jobs; they do not run them.
• Uniform outputs: one folder per app under data/.