Save cube to team storage
Save a datacube to your team storage¶
A DeepESDL example notebook¶
This notebook demonstrates how save a generated datacube to your team storage. In the example the CCI data store is used, for details concerning the CCI store are given in example notebook 03_Generate_CCI_cubes.ipynb
Please, also refer to the DeepESDL documentation and visit the platform's website for further information!
Brockmann Consult, 2024
This notebook runs with the python environment deepesdl-xcube-1.7.0
, please checkout the documentation for help on changing the environment.
# mandatory imports
import datetime
import os
import matplotlib.pyplot as plt
import shapely.geometry
from xcube.core.store import new_data_store
Configure matplotlib to display graphs inline directly in the notebook and set a sensible default figure size.
%matplotlib inline
plt.rcParams["figure.figsize"] = 16, 8
Provide mandatory parameters to instantiate the store class:
store = new_data_store("ccizarr")
Let's open an example dataset:
def open_zarrstore(filename, time_range, variables, bbox):
min_lat, min_lon, max_lat, max_lon = bbox
ds = store.open_data(filename)
subset = ds.sel(
lat=slice(min_lat, max_lat),
lon=slice(min_lon, max_lon),
time=slice(time_range[0], time_range[1]),
)
subset = subset[variables]
return subset
dataset = open_zarrstore(
"ESACCI-LST-L3C-LST-MODISA-0.01deg_1DAILY_DAY-2002-2018-fv3.00.zarr",
time_range=[datetime.datetime(2016, 10, 1), datetime.datetime(2016, 10, 2)],
variables=["lst"],
bbox=[35, -10, 70, 30],
)
dataset
<xarray.Dataset> Size: 224MB Dimensions: (time: 2, lat: 3500, lon: 4000) Coordinates: * lat (lat) float32 14kB 35.01 35.01 35.02 35.03 ... 69.97 69.98 69.99 * lon (lon) float32 16kB -9.995 -9.985 -9.975 ... 29.98 29.98 29.99 * time (time) datetime64[ns] 16B 2016-10-01 2016-10-02 Data variables: lst (time, lat, lon) float64 224MB dask.array<chunksize=(1, 1500, 1000), meta=np.ndarray> Attributes: (12/42) Conventions: CF-1.8 catalogue_url: https://catalogue.ceda.ac.uk/uuid/6babb8d9a8d... cdm_data_type: grid comment: These data were produced as part of the ESA L... creator_email: djg20@le.ac.uk creator_name: University of Leicester Surface Temperature G... ... ... summary: This file contains level L3C global land surf... time_coverage_duration: PT04M59S time_coverage_end: 19700101T000001Z time_coverage_resolution: P1D time_coverage_start: 19700101T000001Z title: ESA LST CCI land surface temperature data at ...
Plot one time stamp of the dataset for a analysed_sst in order to take a brief look at the dataset:
dataset.lst.isel(time=0).plot.imshow(cmap="plasma")
<matplotlib.image.AxesImage at 0x7f7ebf4a8950>
To store the cube in your teams user space, please first retrieve the details from your environment variables as the following:
S3_USER_STORAGE_KEY = os.environ["S3_USER_STORAGE_KEY"]
S3_USER_STORAGE_SECRET = os.environ["S3_USER_STORAGE_SECRET"]
S3_USER_STORAGE_BUCKET = os.environ["S3_USER_STORAGE_BUCKET"]
You need to instantiate a s3 datastore pointing to the team bucket:
team_store = new_data_store(
"s3",
root=S3_USER_STORAGE_BUCKET,
storage_options=dict(
anon=False, key=S3_USER_STORAGE_KEY, secret=S3_USER_STORAGE_SECRET
),
)
If you have stored no data to your user space, the returned list will be empty:
list(team_store.get_data_ids())
['SST.levels', 'amazonas_v8.zarr', 'amazonas_v9.zarr', 'noise_trajectory.zarr', 'reanalysis-era5-single-levels-monthly-means-subset-2001-2010_TMH.zarr']
Save in levels format to generate data pyramid for efficient visualisation
team_store.write_data(
dataset, "analysed_lst.levels", replace=True, use_saved_levels=True
)
'analysed_lst.levels'
If you list the content of you datastore again, you will now see the newly written dataset in the list:
list(team_store.get_data_ids())
['SST.levels', 'amazonas_v8.zarr', 'amazonas_v9.zarr', 'analysed_lst.levels', 'noise_trajectory.zarr', 'reanalysis-era5-single-levels-monthly-means-subset-2001-2010_TMH.zarr']
Open data from your team storage:
ml_dataset = team_store.open_data("analysed_lst.levels")
ml_dataset
<xcube.core.mldataset.fs.FsMultiLevelDataset at 0x7f7ebe387290>
Check, how many levels were written:
ml_dataset.num_levels
2
Print metadata details for each multilevel dataset
for level in range(ml_dataset.num_levels):
dataset_i = ml_dataset.get_dataset(level)
display(dataset_i)
<xarray.Dataset> Size: 224MB Dimensions: (lat: 3500, lon: 4000, time: 2) Coordinates: * lat (lat) float32 14kB 35.01 35.01 35.02 35.03 ... 69.97 69.98 69.99 * lon (lon) float32 16kB -9.995 -9.985 -9.975 ... 29.98 29.98 29.99 * time (time) datetime64[ns] 16B 2016-10-01 2016-10-02 Data variables: lst (time, lat, lon) float64 224MB dask.array<chunksize=(1, 2000, 2000), meta=np.ndarray> Attributes: (12/42) Conventions: CF-1.8 catalogue_url: https://catalogue.ceda.ac.uk/uuid/6babb8d9a8d... cdm_data_type: grid comment: These data were produced as part of the ESA L... creator_email: djg20@le.ac.uk creator_name: University of Leicester Surface Temperature G... ... ... summary: This file contains level L3C global land surf... time_coverage_duration: PT04M59S time_coverage_end: 19700101T000001Z time_coverage_resolution: P1D time_coverage_start: 19700101T000001Z title: ESA LST CCI land surface temperature data at ...
<xarray.Dataset> Size: 56MB Dimensions: (lat: 1750, lon: 2000, time: 2) Coordinates: * lat (lat) float32 7kB 35.01 35.02 35.05 35.06 ... 69.94 69.96 69.98 * lon (lon) float32 8kB -9.995 -9.975 -9.955 -9.935 ... 29.95 29.96 29.98 * time (time) datetime64[ns] 16B 2016-10-01 2016-10-02 Data variables: lst (time, lat, lon) float64 56MB dask.array<chunksize=(1, 1750, 2000), meta=np.ndarray> Attributes: (12/42) Conventions: CF-1.8 catalogue_url: https://catalogue.ceda.ac.uk/uuid/6babb8d9a8d... cdm_data_type: grid comment: These data were produced as part of the ESA L... creator_email: djg20@le.ac.uk creator_name: University of Leicester Surface Temperature G... ... ... summary: This file contains level L3C global land surf... time_coverage_duration: PT04M59S time_coverage_end: 19700101T000001Z time_coverage_resolution: P1D time_coverage_start: 19700101T000001Z title: ESA LST CCI land surface temperature data at ...
To work with a certain level:
dataset_1 = ml_dataset.get_dataset(1)
dataset_1
<xarray.Dataset> Size: 56MB Dimensions: (lat: 1750, lon: 2000, time: 2) Coordinates: * lat (lat) float32 7kB 35.01 35.02 35.05 35.06 ... 69.94 69.96 69.98 * lon (lon) float32 8kB -9.995 -9.975 -9.955 -9.935 ... 29.95 29.96 29.98 * time (time) datetime64[ns] 16B 2016-10-01 2016-10-02 Data variables: lst (time, lat, lon) float64 56MB dask.array<chunksize=(1, 1750, 2000), meta=np.ndarray> Attributes: (12/42) Conventions: CF-1.8 catalogue_url: https://catalogue.ceda.ac.uk/uuid/6babb8d9a8d... cdm_data_type: grid comment: These data were produced as part of the ESA L... creator_email: djg20@le.ac.uk creator_name: University of Leicester Surface Temperature G... ... ... summary: This file contains level L3C global land surf... time_coverage_duration: PT04M59S time_coverage_end: 19700101T000001Z time_coverage_resolution: P1D time_coverage_start: 19700101T000001Z title: ESA LST CCI land surface temperature data at ...
To delete a dataset from your team user space:
team_store.delete_data("analysed_lst.levels")