Skip to content

Python API

LintDataset

LintDataset validates an xarray dataset against the metadata rules required for publication to the ESA Open Science Catalog. Run it before publishing to catch issues early.

from deep_code.tools.lint import LintDataset

# Lint a Zarr dataset from the DeepESDL public/team bucket by ID
result = LintDataset(dataset_id="my-dataset.zarr").lint_dataset()

# Or lint an already-opened xarray.Dataset
import xarray as xr
ds = xr.open_zarr("s3://my-bucket/my-dataset.zarr")
result = LintDataset(dataset=ds).lint_dataset()

print(result)

Validation rules

Two custom rules are checked:

Rule What is validated If missing during publish
deepcode/dataset-description Dataset has a description attribute in ds.attrs. Falls back to "No description available."
deepcode/variable-gcmd-keyword-url Every data variable has a gcmd_keyword_url attribute. You are prompted to enter a URL for each affected variable interactively.

Run LintDataset beforehand to spot metadata gaps before they interrupt the publish flow.

The same check is also available from the CLI — see Lint a dataset.

Publisher

deep_code.tools.publish.Publisher is the main entry point.

from deep_code.tools.publish import Publisher

publisher = Publisher(
    dataset_config_path="dataset.yaml",
    workflow_config_path="workflow.yaml",
    environment="staging",  # "production" | "staging" | "testing"
)

# Generate files locally (no PR)
publisher.publish(write_to_file=True, mode="all")

# Or open a PR directly
publisher.publish(write_to_file=False, mode="dataset")

mode controls what is published:

mode What is published
"dataset" OSC STAC collection, variable catalogs, product/variable base catalogs, project collection.
"workflow" OGC API workflow and experiment records, workflow/experiment base catalogs.
"all" Both of the above (default).

OscDatasetStacGenerator

OscDatasetStacGenerator can also be used directly when you need more control over individual artifacts.

from deep_code.utils.dataset_stac_generator import ItemConfig, OscDatasetStacGenerator

generator = OscDatasetStacGenerator(
    collection_id="my-collection",
    workflow_id="my-workflow",
    workflow_title="My Workflow",
    license_type="CC-BY-4.0",
    items_config=[ItemConfig(dataset_id="my-dataset.zarr", item_id="my-item")],
    osc_themes=["cryosphere"],
    osc_region="Global",
    osc_status="completed",
    # Optional: override the default project identifier.
    # Controls osc:project on the collection and the link to the project collection.
    osc_project="deep-earth-system-data-lab",
    # Optional, only used for the S3-hosted STAC item: absolute URL of the Zarr
    # store. If omitted, it is looked up from the dataset's item in the PRR STAC API.
    access_link=None,
)

# Without stac_catalog_s3_root the collection links to the dataset's PRR
# collection (https://eoresults.esa.int/stac/collections/my-collection),
# which must already exist.
collection = generator.build_dataset_stac_collection(mode="dataset")

osc_project parameter

osc_project defaults to "deep-earth-system-data-lab" and is used in three places:

  1. Sets osc:project on the OSC extension of the generated STAC collection.
  2. Generates the related link from the product collection to the project collection (../../projects/{osc_project}/collection.json).
  3. Determines the file path of the project collection when publishing (projects/{osc_project}/collection.json).

Automatic project collection creation

When Publisher.publish_dataset() runs, it checks whether projects/{osc_project}/collection.json already exists in the catalog repository:

  • Missing — a minimal STAC Collection is created for the project and a child link is appended to projects/catalog.json so it is reachable from the catalog root.
  • Exists — the existing collection is updated with a child link to the new product and related links for its themes (same behaviour as before).

This means publishing to a new project does not require manual catalog setup.

STAC Catalog and Item generation

Only needed for datasets that are not in PRR:

# Build the S3 STAC hierarchy (dict keyed by S3 path)
file_dict = generator.build_zarr_stac_catalog_file_dict(
    stac_catalog_s3_root="s3://bucket/stac/my-collection/"
)
# file_dict contains:
#   "s3://bucket/stac/my-collection/catalog.json"
#   "s3://bucket/stac/my-collection/my-collection/items/my-item.json"

The item's assets point to access_link, or, if it is not passed to the generator, to the Zarr store served by PRR, looked up from https://eoresults.esa.int/stac/collections/{collection_id}/items/{item_id}.

See STAC Catalog on S3 for details on the generated structure.

PRR collection generation

Build a self-contained PRR (Project Results Repository) Collection → Item → Assets tree as local files. The high-level helper reads the same dataset config as the CLI:

from deep_code.tools.prr import generate_prr_collection

out_dir = generate_prr_collection("dataset.yaml", output_dir="./prr")
# ./prr/<collection_id>/collection.json  +  ./prr/<collection_id>/items/<item_id>.json

Or drive the generator directly:

generator.save_prr_collection("./prr")   # writes a self-contained tree

The Item includes the datacube extension; the Collection declares the OSC, Scientific, Processing, Themes and CF extensions and the PRR-mandatory fields. See PRR collection output and Generate a PRR collection.