Chunking of datasets

How to save a data cube with a desired chunking¶

A DeepESDL example notebook¶

This notebook demonstrates how modify the chunking of a dataset before persisting it.

Please, also refer to the DeepESDL documentation and visit the platform's website for further information!

Brockmann Consult, 2024

---

This notebook runs with the python environment deepesdl-xcube-1.7.0, please checkout the documentation for help on changing the environment.

First, lets create a small cube, which we can later on append data to. We will use ESA CCI data for this. Please head over to 3 - Generate CCI data cubes to get more details about the xcube-cci data store :)

In [1]:

import datetime
import os

from xcube.core.store import new_data_store

import datetime import os from xcube.core.store import new_data_store

In [2]:

store = new_data_store("ccizarr")

store = new_data_store("ccizarr")

Next, we create a cube containing only 4 days of data:

In [3]:

def open_zarrstore(filename, time_range, variables):
    ds = store.open_data(filename)
    subset = ds.sel(time=slice(time_range[0], time_range[1]))
    subset = subset[variables]

    return subset


dataset = open_zarrstore(
    "ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0-1981-2016-v02.0-fv01.0.zarr",
    time_range=[datetime.datetime(2015, 10, 1), datetime.datetime(2015, 10, 5)],
    variables=["analysed_sst"],
)

def open_zarrstore(filename, time_range, variables): ds = store.open_data(filename) subset = ds.sel(time=slice(time_range[0], time_range[1])) subset = subset[variables] return subset dataset = open_zarrstore( "ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0-1981-2016-v02.0-fv01.0.zarr", time_range=[datetime.datetime(2015, 10, 1), datetime.datetime(2015, 10, 5)], variables=["analysed_sst"], )

In [4]:

dataset

dataset

Out[4]:

<xarray.Dataset> Size: 33MB
Dimensions:       (time: 4, lat: 720, lon: 1440)
Coordinates:
  * lat           (lat) float32 3kB -89.88 -89.62 -89.38 ... 89.38 89.62 89.88
  * lon           (lon) float32 6kB -179.9 -179.6 -179.4 ... 179.4 179.6 179.9
  * time          (time) datetime64[ns] 32B 2015-10-01T12:00:00 ... 2015-10-0...
Data variables:
    analysed_sst  (time, lat, lon) float64 33MB dask.array<chunksize=(4, 720, 720), meta=np.ndarray>
Attributes: (12/47)
    Conventions:                CF-1.4
    acknowledgment:             Funded by ESA
    cdm_data_type:              grid
    comment:                    
    creator_email:              science.leader@esa-sst-cci.org
    creator_name:               SST_cci
    ...                         ...
    summary:                    An ensemble product with input from a number ...
    time_coverage_end:          20170101T000000Z
    time_coverage_start:        20161231T000000Z
    title:                      Global SST Ensemble, L4 GMPE
    uuid:                       dc0c5b25-93bf-4943-aba1-7f0de9109620
    westernmost_longitude:      -180.0

xarray.Dataset

• Dimensions:
  • time: 4
  • lat: 720
  • lon: 1440
• Coordinates: (3)
  • lat
    (lat)
    float32
    -89.88 -89.62 ... 89.62 89.88

    axis :

    Y

    reference_datum :

    geographical coordinates, WGS84 projection

    standard_name :

    latitude

    units :

    degrees_north

    valid_max :

    90.0

    valid_min :

    -90.0

    array([-89.875, -89.625, -89.375, ...,  89.375,  89.625,  89.875],
          dtype=float32)

  • lon
    (lon)
    float32
    -179.9 -179.6 ... 179.6 179.9

    axis :

    X

    reference_datum :

    geographical coordinates, WGS84 projection

    standard_name :

    longitude

    units :

    degrees_east

    valid_max :

    180.0

    valid_min :

    -180.0

    array([-179.875, -179.625, -179.375, ...,  179.375,  179.625,  179.875],
          dtype=float32)

  • time
    (time)
    datetime64[ns]
    2015-10-01T12:00:00 ... 2015-10-...

    axis :

    T

    long_name :

    reference time of sst field

    standard_name :

    time

    array(['2015-10-01T12:00:00.000000000', '2015-10-02T12:00:00.000000000',
           '2015-10-03T12:00:00.000000000', '2015-10-04T12:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (1)
  • analysed_sst
    (time, lat, lon)
    float64
    dask.array<chunksize=(4, 720, 720), meta=np.ndarray>

    long_name :

    median SST from GMPE

    source :

    SST CCI V2, SST CCI V1.1, MyOcean, AVHRR_OI, CMC, HadISST.2.2.0.0_r0, HadISST.2.2.0.0_r1, HadISST.2.2.0.0_r2, HadISST.2.2.0.0_r3, HadISST.2.2.0.0_r4, HadISST.2.2.0.0_r5, HadISST.2.2.0.0_r6, HadISST.2.2.0.0_r7, HadISST.2.2.0.0_r8, HadISST.2.2.0.0_r9, MGDSST

    standard_name :

    sea_surface_foundation_temperature

    units :

    kelvin

    valid_max :

    4500

    valid_min :

    -300

    Array Chunk Bytes 31.64 MiB 15.82 MiB Shape (4, 720, 1440) (4, 720, 720) Dask graph 2 chunks in 3 graph layers Data type float64 numpy.ndarray

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index([-89.875, -89.625, -89.375, -89.125, -88.875, -88.625, -88.375, -88.125,
           -87.875, -87.625,
           ...
            87.625,  87.875,  88.125,  88.375,  88.625,  88.875,  89.125,  89.375,
            89.625,  89.875],
          dtype='float32', name='lat', length=720))

  • lon
    PandasIndex

    PandasIndex(Index([-179.875, -179.625, -179.375, -179.125, -178.875, -178.625, -178.375,
           -178.125, -177.875, -177.625,
           ...
            177.625,  177.875,  178.125,  178.375,  178.625,  178.875,  179.125,
            179.375,  179.625,  179.875],
          dtype='float32', name='lon', length=1440))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2015-10-01 12:00:00', '2015-10-02 12:00:00',
                   '2015-10-03 12:00:00', '2015-10-04 12:00:00'],
                  dtype='datetime64[ns]', name='time', freq=None))

• Attributes: (47)

  Conventions :

  CF-1.4

  acknowledgment :

  Funded by ESA

  cdm_data_type :

  grid

  comment :

  creator_email :

  science.leader@esa-sst-cci.org

  creator_name :

  SST_cci

  creator_url :

  http://www.esa-sst-cci.org

  date_created :

  20180711T172223Z

  easternmost_longitude :

  180.0

  file_quality_level :

  3

  gds_version_id :

  2.0

  geospatial_lat_resolution :

  0.25 degree

  geospatial_lat_units :

  degrees north

  geospatial_lon_resolution :

  0.25 degree

  geospatial_lon_units :

  degrees east

  history :

  NULL

  id :

  UKMO-L4LRens-GLOB-GMPE

  institution :

  UKMO

  keywords :

  Oceans > Ocean Temperature > Sea Surface Temperature

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Science Keywords

  license :

  Creative Commons Licence by attribution (https://creativecommons.org/licenses/by/4.0/

  metadata_conventions :

  Unidata Observation Dataset v1.0

  metadata_link :

  not available

  naming_authority :

  org.ghrsst

  netcdf_version_id :

  3.6

  northernmost_latitude :

  90.0

  platform :

  processing_level :

  GMPE

  product_version :

  3.0

  project :

  Climate Change Initiative - European Space Agency

  publisher_email :

  science.leader@esa-sst-cci.org

  publisher_name :

  ESACCI

  publisher_url :

  http://www.esa-sst-cci.org

  references :

  Martin et al., Deep Sea Research II, 2011

  sensor :

  source :

  SST CCI V2, SST CCI V1.1, MyOcean, AVHRR_OI, CMC, HadISST.2.2.0.0_r0, HadISST.2.2.0.0_r1, HadISST.2.2.0.0_r2, HadISST.2.2.0.0_r3, HadISST.2.2.0.0_r4, HadISST.2.2.0.0_r5, HadISST.2.2.0.0_r6, HadISST.2.2.0.0_r7, HadISST.2.2.0.0_r8, HadISST.2.2.0.0_r9, MGDSST

  southernmost_latitude :

  -90.0

  spatial_resolution :

  0.25 degree

  standard_name_vocabulary :

  NetCDF Climate and Forecast (CF) Metadata Convention

  start_time :

  20161231T000000Z

  stop_time :

  20170101T000000Z

  summary :

  An ensemble product with input from a number of L4 SST analyses

  time_coverage_end :

  20170101T000000Z

  time_coverage_start :

  20161231T000000Z

  title :

  Global SST Ensemble, L4 GMPE

  uuid :

  dc0c5b25-93bf-4943-aba1-7f0de9109620

  westernmost_longitude :

  -180.0

In the example above, we can see that the variable analysed_sst is chunked as follows: (4, 720, 720). This means, each chunk contains 4 time values, 720 lat values and 720 lon values per chunk. Variables, which contain 1 time value and many spatial dimensions in one chunk are optimal for visualisation/plotting of one time stamp.

For analysing long time series, it is benificial to chunk a dataset accordingly, so the chunks contain more values of the time dimension and less of the spatial dimensions.

In [5]:

# time optimised chunking - please note, this is just an example
time_chunksize = 1
x_chunksize = 120  # or lon
y_chunksize = 120  # or lat

# time optimised chunking - please note, this is just an example time_chunksize = 1 x_chunksize = 120 # or lon y_chunksize = 120 # or lat

Now the chunking is applied to all variables, but skipping crs if present:

In [6]:

dataset.data_vars

dataset.data_vars

Out[6]:

Data variables:
    analysed_sst  (time, lat, lon) float64 33MB dask.array<chunksize=(4, 720, 720), meta=np.ndarray>

In [7]:

for var_name in dataset.data_vars:
    if var_name != "crs" and "_bounds" not in var_name:
        print(var_name)
        dataset[var_name] = dataset[var_name].chunk(
            {"time": time_chunksize, "lat": y_chunksize, "lon": x_chunksize}
        )

for var_name in dataset.data_vars: if var_name != "crs" and "_bounds" not in var_name: print(var_name) dataset[var_name] = dataset[var_name].chunk( {"time": time_chunksize, "lat": y_chunksize, "lon": x_chunksize} )

analysed_sst

To save a copy of a cube with a specific chunking, the encoding must be adjusted acordingly.

In [8]:

encoding_dict = dict()

encoding_dict = dict()

We want to ensure that the coordinate variables are stored in the best performant way, so we ensure that they are not chunked. This can be specified via the encoding:

In [9]:

coords_encoding = {k: dict(chunks=v.shape) for k, v in dataset.coords.items()}

coords_encoding = {k: dict(chunks=v.shape) for k, v in dataset.coords.items()}

In [10]:

coords_encoding

coords_encoding

Out[10]:

{'lat': {'chunks': (720,)},
 'lon': {'chunks': (1440,)},
 'time': {'chunks': (4,)}}

Specify the chunking the data variables encoding and ensuring that empty chunks are not written to disk by adding write_empty_chunks=False. This saves space on disk. Again, skipping crs if present.

In [11]:

vars_encoding = {
    k: dict(chunks=(time_chunksize, y_chunksize, x_chunksize), write_empty_chunks=False)
    for k, v in dataset.data_vars.items()
    if k != "crs"
}

vars_encoding = { k: dict(chunks=(time_chunksize, y_chunksize, x_chunksize), write_empty_chunks=False) for k, v in dataset.data_vars.items() if k != "crs" }

In [12]:

vars_encoding

vars_encoding

Out[12]:

{'analysed_sst': {'chunks': (1, 120, 120), 'write_empty_chunks': False}}

Next, combining both dictionaries to form the encoding for the entire dataset.

In [13]:

encoding_dict.update(coords_encoding)
encoding_dict.update(vars_encoding)

encoding_dict.update(coords_encoding) encoding_dict.update(vars_encoding)

In [14]:

encoding_dict

encoding_dict

Out[14]:

{'lat': {'chunks': (720,)},
 'lon': {'chunks': (1440,)},
 'time': {'chunks': (4,)},
 'analysed_sst': {'chunks': (1, 120, 120), 'write_empty_chunks': False}}

Next, save it to the team s3 storage:

To store the cube in your teams user space, please first retrieve the details from your environment variables as the following:

In [15]:

S3_USER_STORAGE_KEY = os.environ["S3_USER_STORAGE_KEY"]
S3_USER_STORAGE_SECRET = os.environ["S3_USER_STORAGE_SECRET"]
S3_USER_STORAGE_BUCKET = os.environ["S3_USER_STORAGE_BUCKET"]

S3_USER_STORAGE_KEY = os.environ["S3_USER_STORAGE_KEY"] S3_USER_STORAGE_SECRET = os.environ["S3_USER_STORAGE_SECRET"] S3_USER_STORAGE_BUCKET = os.environ["S3_USER_STORAGE_BUCKET"]

You need to instantiate a s3 datastore pointing to the team bucket:

In [16]:

team_store = new_data_store(
    "s3",
    root=S3_USER_STORAGE_BUCKET,
    storage_options=dict(
        anon=False, key=S3_USER_STORAGE_KEY, secret=S3_USER_STORAGE_SECRET
    ),
)

team_store = new_data_store( "s3", root=S3_USER_STORAGE_BUCKET, storage_options=dict( anon=False, key=S3_USER_STORAGE_KEY, secret=S3_USER_STORAGE_SECRET ), )

If you have stored no data to your user space, the returned list will be empty:

In [17]:

list(team_store.get_data_ids())

list(team_store.get_data_ids())

Out[17]:

['SST.levels',
 'amazonas_v8.zarr',
 'amazonas_v9.zarr',
 'noise_trajectory.zarr',
 'reanalysis-era5-single-levels-monthly-means-subset-2001-2010_TMH.zarr',
 's2-demo-cube.zarr']

In [18]:

output_id = "analysed_sst.zarr"

output_id = "analysed_sst.zarr"

Now let's write the data to the team s3 storage and remember to specify the encoding while doing so:

In [19]:

team_store.write_data(dataset, output_id, encoding=encoding_dict, replace=True)

team_store.write_data(dataset, output_id, encoding=encoding_dict, replace=True)

Out[19]:

'analysed_sst.zarr'

If you list the content of you datastore again, you will now see the newly written dataset in the list:

In [20]:

list(team_store.get_data_ids())

list(team_store.get_data_ids())

Out[20]:

['SST.levels',
 'amazonas_v8.zarr',
 'amazonas_v9.zarr',
 'analysed_sst.zarr',
 'noise_trajectory.zarr',
 'reanalysis-era5-single-levels-monthly-means-subset-2001-2010_TMH.zarr',
 's2-demo-cube.zarr']

Let's verify that our chunking has been applied:

In [21]:

ds = team_store.open_data(output_id)

ds = team_store.open_data(output_id)

In [22]:

ds

ds

Out[22]:

<xarray.Dataset> Size: 33MB
Dimensions:       (time: 4, lat: 720, lon: 1440)
Coordinates:
  * lat           (lat) float32 3kB -89.88 -89.62 -89.38 ... 89.38 89.62 89.88
  * lon           (lon) float32 6kB -179.9 -179.6 -179.4 ... 179.4 179.6 179.9
  * time          (time) datetime64[ns] 32B 2015-10-01T12:00:00 ... 2015-10-0...
Data variables:
    analysed_sst  (time, lat, lon) float64 33MB dask.array<chunksize=(1, 120, 120), meta=np.ndarray>
Attributes: (12/47)
    Conventions:                CF-1.4
    acknowledgment:             Funded by ESA
    cdm_data_type:              grid
    comment:                    
    creator_email:              science.leader@esa-sst-cci.org
    creator_name:               SST_cci
    ...                         ...
    summary:                    An ensemble product with input from a number ...
    time_coverage_end:          20170101T000000Z
    time_coverage_start:        20161231T000000Z
    title:                      Global SST Ensemble, L4 GMPE
    uuid:                       dc0c5b25-93bf-4943-aba1-7f0de9109620
    westernmost_longitude:      -180.0

xarray.Dataset

• Dimensions:
  • time: 4
  • lat: 720
  • lon: 1440
• Coordinates: (3)
  • lat
    (lat)
    float32
    -89.88 -89.62 ... 89.62 89.88

    axis :

    Y

    reference_datum :

    geographical coordinates, WGS84 projection

    standard_name :

    latitude

    units :

    degrees_north

    valid_max :

    90.0

    valid_min :

    -90.0

    array([-89.875, -89.625, -89.375, ...,  89.375,  89.625,  89.875],
          dtype=float32)

  • lon
    (lon)
    float32
    -179.9 -179.6 ... 179.6 179.9

    axis :

    X

    reference_datum :

    geographical coordinates, WGS84 projection

    standard_name :

    longitude

    units :

    degrees_east

    valid_max :

    180.0

    valid_min :

    -180.0

    array([-179.875, -179.625, -179.375, ...,  179.375,  179.625,  179.875],
          dtype=float32)

  • time
    (time)
    datetime64[ns]
    2015-10-01T12:00:00 ... 2015-10-...

    axis :

    T

    long_name :

    reference time of sst field

    standard_name :

    time

    array(['2015-10-01T12:00:00.000000000', '2015-10-02T12:00:00.000000000',
           '2015-10-03T12:00:00.000000000', '2015-10-04T12:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (1)
  • analysed_sst
    (time, lat, lon)
    float64
    dask.array<chunksize=(1, 120, 120), meta=np.ndarray>

    long_name :

    median SST from GMPE

    source :

    SST CCI V2, SST CCI V1.1, MyOcean, AVHRR_OI, CMC, HadISST.2.2.0.0_r0, HadISST.2.2.0.0_r1, HadISST.2.2.0.0_r2, HadISST.2.2.0.0_r3, HadISST.2.2.0.0_r4, HadISST.2.2.0.0_r5, HadISST.2.2.0.0_r6, HadISST.2.2.0.0_r7, HadISST.2.2.0.0_r8, HadISST.2.2.0.0_r9, MGDSST

    standard_name :

    sea_surface_foundation_temperature

    units :

    kelvin

    valid_max :

    4500

    valid_min :

    -300

    Array Chunk Bytes 31.64 MiB 112.50 kiB Shape (4, 720, 1440) (1, 120, 120) Dask graph 288 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index([-89.875, -89.625, -89.375, -89.125, -88.875, -88.625, -88.375, -88.125,
           -87.875, -87.625,
           ...
            87.625,  87.875,  88.125,  88.375,  88.625,  88.875,  89.125,  89.375,
            89.625,  89.875],
          dtype='float32', name='lat', length=720))

  • lon
    PandasIndex

    PandasIndex(Index([-179.875, -179.625, -179.375, -179.125, -178.875, -178.625, -178.375,
           -178.125, -177.875, -177.625,
           ...
            177.625,  177.875,  178.125,  178.375,  178.625,  178.875,  179.125,
            179.375,  179.625,  179.875],
          dtype='float32', name='lon', length=1440))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2015-10-01 12:00:00', '2015-10-02 12:00:00',
                   '2015-10-03 12:00:00', '2015-10-04 12:00:00'],
                  dtype='datetime64[ns]', name='time', freq=None))

• Attributes: (47)

  Conventions :

  CF-1.4

  acknowledgment :

  Funded by ESA

  cdm_data_type :

  grid

  comment :

  creator_email :

  science.leader@esa-sst-cci.org

  creator_name :

  SST_cci

  creator_url :

  http://www.esa-sst-cci.org

  date_created :

  20180711T172223Z

  easternmost_longitude :

  180.0

  file_quality_level :

  3

  gds_version_id :

  2.0

  geospatial_lat_resolution :

  0.25 degree

  geospatial_lat_units :

  degrees north

  geospatial_lon_resolution :

  0.25 degree

  geospatial_lon_units :

  degrees east

  history :

  NULL

  id :

  UKMO-L4LRens-GLOB-GMPE

  institution :

  UKMO

  keywords :

  Oceans > Ocean Temperature > Sea Surface Temperature

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Science Keywords

  license :

  Creative Commons Licence by attribution (https://creativecommons.org/licenses/by/4.0/

  metadata_conventions :

  Unidata Observation Dataset v1.0

  metadata_link :

  not available

  naming_authority :

  org.ghrsst

  netcdf_version_id :

  3.6

  northernmost_latitude :

  90.0

  platform :

  processing_level :

  GMPE

  product_version :

  3.0

  project :

  Climate Change Initiative - European Space Agency

  publisher_email :

  science.leader@esa-sst-cci.org

  publisher_name :

  ESACCI

  publisher_url :

  http://www.esa-sst-cci.org

  references :

  Martin et al., Deep Sea Research II, 2011

  sensor :

  source :

  SST CCI V2, SST CCI V1.1, MyOcean, AVHRR_OI, CMC, HadISST.2.2.0.0_r0, HadISST.2.2.0.0_r1, HadISST.2.2.0.0_r2, HadISST.2.2.0.0_r3, HadISST.2.2.0.0_r4, HadISST.2.2.0.0_r5, HadISST.2.2.0.0_r6, HadISST.2.2.0.0_r7, HadISST.2.2.0.0_r8, HadISST.2.2.0.0_r9, MGDSST

  southernmost_latitude :

  -90.0

  spatial_resolution :

  0.25 degree

  standard_name_vocabulary :

  NetCDF Climate and Forecast (CF) Metadata Convention

  start_time :

  20161231T000000Z

  stop_time :

  20170101T000000Z

  summary :

  An ensemble product with input from a number of L4 SST analyses

  time_coverage_end :

  20170101T000000Z

  time_coverage_start :

  20161231T000000Z

  title :

  Global SST Ensemble, L4 GMPE

  uuid :

  dc0c5b25-93bf-4943-aba1-7f0de9109620

  westernmost_longitude :

  -180.0

Looks good, now let's clean up the example cube :)

In [23]:

team_store.delete_data(output_id)

team_store.delete_data(output_id)

In [ ]: