Hands-on - Virtualize NetCDF from ESGF

This example uses data from the Earth System Grid Federation THREDDS Data Server.

This is a quicker example for hands-on experience. For a full walkthrough, view previous example on the NASA-NEX-GDDP CMIP6 data.

Thank you to Raphael Hagen for contributing this example!

Step 1: Import necessary functions and classes

import icechunk
from obstore.store import HTTPStore
from virtualizarr import open_virtual_dataset
from virtualizarr.parsers import HDFParser
from virtualizarr.registry import ObjectStoreRegistry

Step 2: Define data location

bucket = "https://esgf-data.ucar.edu"
path = "thredds/fileServer/esg_dataroot/CMIP6/CMIP/NCAR/CESM2/historical/r3i1p1f1/day/tas/gn/v20190308/tas_day_CESM2_historical_r3i1p1f1_gn_19200101-19291231.nc"

Step 3: Create an ObjectStore and an ObjectStoreRegistry

store = HTTPStore.from_url(bucket)
registry = ObjectStoreRegistry({bucket: store})

Step 4: Create an instance of the HDFParser

parser = HDFParser()

Step 5: Create a virtual dataset via open_virtual_dataset

vds = open_virtual_dataset(
    url=f"{bucket}/{path}",
    parser=parser,
    registry=registry,
    loadable_variables=["lat", "lon", "time", "time_bnds", "lat_bnds", "lon_bnds"],
)

/home/jovyan/Code/virtual-zarr/esip-2025/.pixi/envs/default/lib/python3.13/site-packages/numcodecs/zarr3.py:145: UserWarning: Numcodecs codecs are not in the Zarr version 3 specification and may not be supported by other zarr implementations.
  super().__init__(**codec_config)
/home/jovyan/Code/virtual-zarr/esip-2025/.pixi/envs/default/lib/python3.13/site-packages/xarray/conventions.py:204: SerializationWarning: variable 'tas' has multiple fill values {np.float64(1.0000000200408773e+20), np.float64(1e+20)} defined, decoding all values to NaN.
  var = coder.decode(var, name=name)

vds

<xarray.Dataset> Size: 807MB
Dimensions:    (lat: 192, lon: 288, time: 3650, nbnd: 2)
Coordinates:
  * lat        (lat) float64 2kB -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
  * lon        (lon) float64 2kB 0.0 1.25 2.5 3.75 ... 355.0 356.2 357.5 358.8
  * time       (time) object 29kB 1920-01-01 00:00:00 ... 1929-12-31 00:00:00
Dimensions without coordinates: nbnd
Data variables:
    time_bnds  (time, nbnd) object 58kB ...
    lat_bnds   (lat, nbnd) float32 2kB ...
    lon_bnds   (lon, nbnd) float32 2kB ...
    tas        (time, lat, lon) float32 807MB ManifestArray<shape=(3650, 192,...
Attributes: (12/45)
    Conventions:            CF-1.7 CMIP-6.2
    activity_id:            CMIP
    case_id:                17
    cesm_casename:          b.e21.BHIST.f09_g17.CMIP6-historical.003
    contact:                cesm_cmip6@ucar.edu
    creation_date:          2019-01-18T16:46:02Z
    ...                     ...
    sub_experiment:         none
    sub_experiment_id:      none
    branch_time_in_parent:  240900.0
    branch_time_in_child:   674885.0
    branch_method:          standard
    further_info_url:       https://furtherinfo.es-doc.org/CMIP6.NCAR.CESM2.h...

xarray.Dataset

• Dimensions:
  • lat: 192
  • lon: 288
  • time: 3650
  • nbnd: 2
• Coordinates: (3)
  • lat
    (lat)
    float64
    -90.0 -89.06 -88.12 ... 89.06 90.0

    axis :

    Y

    bounds :

    lat_bnds

    standard_name :

    latitude

    title :

    Latitude

    type :

    double

    units :

    degrees_north

    valid_max :

    90.0

    valid_min :

    -90.0

    array([-90.      , -89.057592, -88.115183, -87.172775, -86.230366, -85.287958,
           -84.34555 , -83.403141, -82.460733, -81.518325, -80.575916, -79.633508,
           -78.691099, -77.748691, -76.806283, -75.863874, -74.921466, -73.979058,
           -73.036649, -72.094241, -71.151832, -70.209424, -69.267016, -68.324607,
           -67.382199, -66.439791, -65.497382, -64.554974, -63.612565, -62.670157,
           -61.727749, -60.78534 , -59.842932, -58.900524, -57.958115, -57.015707,
           -56.073298, -55.13089 , -54.188482, -53.246073, -52.303665, -51.361257,
           -50.418848, -49.47644 , -48.534031, -47.591623, -46.649215, -45.706806,
           -44.764398, -43.82199 , -42.879581, -41.937173, -40.994764, -40.052356,
           -39.109948, -38.167539, -37.225131, -36.282723, -35.340314, -34.397906,
           -33.455497, -32.513089, -31.570681, -30.628272, -29.685864, -28.743455,
           -27.801047, -26.858639, -25.91623 , -24.973822, -24.031414, -23.089005,
           -22.146597, -21.204188, -20.26178 , -19.319372, -18.376963, -17.434555,
           -16.492147, -15.549738, -14.60733 , -13.664921, -12.722513, -11.780105,
           -10.837696,  -9.895288,  -8.95288 ,  -8.010471,  -7.068063,  -6.125654,
            -5.183246,  -4.240838,  -3.298429,  -2.356021,  -1.413613,  -0.471204,
             0.471204,   1.413613,   2.356021,   3.298429,   4.240838,   5.183246,
             6.125654,   7.068063,   8.010471,   8.95288 ,   9.895288,  10.837696,
            11.780105,  12.722513,  13.664921,  14.60733 ,  15.549738,  16.492147,
            17.434555,  18.376963,  19.319372,  20.26178 ,  21.204188,  22.146597,
            23.089005,  24.031414,  24.973822,  25.91623 ,  26.858639,  27.801047,
            28.743455,  29.685864,  30.628272,  31.570681,  32.513089,  33.455497,
            34.397906,  35.340314,  36.282723,  37.225131,  38.167539,  39.109948,
            40.052356,  40.994764,  41.937173,  42.879581,  43.82199 ,  44.764398,
            45.706806,  46.649215,  47.591623,  48.534031,  49.47644 ,  50.418848,
            51.361257,  52.303665,  53.246073,  54.188482,  55.13089 ,  56.073298,
            57.015707,  57.958115,  58.900524,  59.842932,  60.78534 ,  61.727749,
            62.670157,  63.612565,  64.554974,  65.497382,  66.439791,  67.382199,
            68.324607,  69.267016,  70.209424,  71.151832,  72.094241,  73.036649,
            73.979058,  74.921466,  75.863874,  76.806283,  77.748691,  78.691099,
            79.633508,  80.575916,  81.518325,  82.460733,  83.403141,  84.34555 ,
            85.287958,  86.230366,  87.172775,  88.115183,  89.057592,  90.      ])

  • lon
    (lon)
    float64
    0.0 1.25 2.5 ... 356.2 357.5 358.8

    axis :

    X

    bounds :

    lon_bnds

    standard_name :

    longitude

    title :

    Longitude

    type :

    double

    units :

    degrees_east

    valid_max :

    360.0

    valid_min :

    0.0

    array([  0.  ,   1.25,   2.5 , ..., 356.25, 357.5 , 358.75], shape=(288,))

  • time
    (time)
    object
    1920-01-01 00:00:00 ... 1929-12-...

    axis :

    T

    bounds :

    time_bnds

    standard_name :

    time

    title :

    time

    type :

    double

    array([cftime.DatetimeNoLeap(1920, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1920, 1, 2, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1920, 1, 3, 0, 0, 0, 0, has_year_zero=True), ...,
           cftime.DatetimeNoLeap(1929, 12, 29, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1929, 12, 30, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1929, 12, 31, 0, 0, 0, 0, has_year_zero=True)],
          shape=(3650,), dtype=object)

• Data variables: (4)
  • time_bnds
    (time, nbnd)
    object
    ...

    [7300 values with dtype=object]

  • lat_bnds
    (lat, nbnd)
    float32
    ...

    units :

    degrees_north

    [384 values with dtype=float32]

  • lon_bnds
    (lon, nbnd)
    float32
    ...

    units :

    degrees_east

    [576 values with dtype=float32]

  • tas
    (time, lat, lon)
    float32
    ManifestArray<shape=(3650, 192, ...

    _FillValue :

    AAAAgB2vFUQ=

    cell_measures :

    area: areacella

    cell_methods :

    area: time: mean

    comment :

    near-surface (usually, 2 meter) air temperature

    coordinates :

    time lat lon

    description :

    near-surface (usually, 2 meter) air temperature

    frequency :

    day

    id :

    tas

    long_name :

    Near-Surface Air Temperature

    mipTable :

    day

    missing_value :

    1e+20

    out_name :

    tas

    prov :

    day ((isd.003))

    realm :

    atmos

    standard_name :

    air_temperature

    time :

    time

    time_label :

    time-mean

    time_title :

    Temporal mean

    title :

    Near-Surface Air Temperature

    type :

    real

    units :

    K

    variable_id :

    tas

    ManifestArray<shape=(3650, 192, 288), dtype=float32, chunks=(1, 192, 288)>

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index([             -90.0, -89.05759162303664,  -88.1151832460733,
           -87.17277486910994,  -86.2303664921466, -85.28795811518324,
            -84.3455497382199, -83.40314136125654, -82.46073298429319,
           -81.51832460732984,
           ...
            81.51832460732984,   82.4607329842932,  83.40314136125653,
            84.34554973821989,  85.28795811518324,   86.2303664921466,
            87.17277486910996,  88.11518324607329,  89.05759162303664,
                         90.0],
          dtype='float64', name='lat', length=192))

  • lon
    PandasIndex

    PandasIndex(Index([   0.0,   1.25,    2.5,   3.75,    5.0,   6.25,    7.5,   8.75,   10.0,
            11.25,
           ...
            347.5, 348.75,  350.0, 351.25,  352.5, 353.75,  355.0, 356.25,  357.5,
           358.75],
          dtype='float64', name='lon', length=288))

  • time
    PandasIndex

    PandasIndex(CFTimeIndex([1920-01-01 00:00:00, 1920-01-02 00:00:00, 1920-01-03 00:00:00,
                 1920-01-04 00:00:00, 1920-01-05 00:00:00, 1920-01-06 00:00:00,
                 1920-01-07 00:00:00, 1920-01-08 00:00:00, 1920-01-09 00:00:00,
                 1920-01-10 00:00:00,
                 ...
                 1929-12-22 00:00:00, 1929-12-23 00:00:00, 1929-12-24 00:00:00,
                 1929-12-25 00:00:00, 1929-12-26 00:00:00, 1929-12-27 00:00:00,
                 1929-12-28 00:00:00, 1929-12-29 00:00:00, 1929-12-30 00:00:00,
                 1929-12-31 00:00:00],
                dtype='object', length=3650, calendar='noleap', freq='D'))

• Attributes: (45)

  Conventions :

  CF-1.7 CMIP-6.2

  activity_id :

  CMIP

  case_id :

  17

  cesm_casename :

  b.e21.BHIST.f09_g17.CMIP6-historical.003

  contact :

  cesm_cmip6@ucar.edu

  creation_date :

  2019-01-18T16:46:02Z

  data_specs_version :

  01.00.29

  experiment :

  all-forcing simulation of the recent past

  experiment_id :

  historical

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  day

  grid :

  native 0.9x1.25 finite volume grid (192x288 latxlon)

  grid_label :

  gn

  initialization_index :

  1

  institution :

  National Center for Atmospheric Research, Climate and Global Dynamics Laboratory, 1850 Table Mesa Drive, Boulder, CO 80305, USA

  institution_id :

  NCAR

  license :

  CMIP6 model data produced by <The National Center for Atmospheric Research> is licensed under a Creative Commons Attribution-[]ShareAlike 4.0 International License (https://creativecommons.org/licenses/). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file)[]. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  mip_era :

  CMIP6

  model_doi_url :

  https://doi.org/10.5065/D67H1H0V

  nominal_resolution :

  100 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  piControl

  parent_mip_era :

  CMIP6

  parent_source_id :

  CESM2

  parent_time_units :

  days since 0001-01-01 00:00:00

  parent_variant_label :

  r1i1p1f1

  physics_index :

  1

  product :

  model-output

  realization_index :

  3

  realm :

  atmos

  source :

  CESM2 (2017): atmosphere: CAM6 (0.9x1.25 finite volume grid; 288 x 192 longitude/latitude; 32 levels; top level 2.25 mb); ocean: POP2 (320x384 longitude/latitude; 60 levels; top grid cell 0-10 m); sea_ice: CICE5.1 (same grid as ocean); land: CLM5 0.9x1.25 finite volume grid; 288 x 192 longitude/latitude; 32 levels; top level 2.25 mb); aerosol: MAM4 (0.9x1.25 finite volume grid; 288 x 192 longitude/latitude; 32 levels; top level 2.25 mb); atmoschem: MAM4 (0.9x1.25 finite volume grid; 288 x 192 longitude/latitude; 32 levels; top level 2.25 mb); landIce: CISM2.1; ocnBgchem: MARBL (320x384 longitude/latitude; 60 levels; top grid cell 0-10 m)

  source_id :

  CESM2

  source_type :

  AOGCM BGC

  table_id :

  day

  tracking_id :

  hdl:21.14100/51927b42-a61c-4086-86fa-9f6b761ac919

  variable_id :

  tas

  variant_info :

  CMIP6 20th century experiments (1850-2014) with CAM6, interactive land (CLM5), coupled ocean (POP2) with biogeochemistry (MARBL), interactive sea ice (CICE5.1), and non-evolving land ice (CISM2.1)

  variant_label :

  r3i1p1f1

  sub_experiment :

  none

  sub_experiment_id :

  none

  branch_time_in_parent :

  240900.0

  branch_time_in_child :

  674885.0

  branch_method :

  standard

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.NCAR.CESM2.historical.none.r3i1p1f1

Step 6: Serialize as an (in-memory) Icechunk store

Usually you’d want to use a persistent storage format, but let’s not waste disk space on an example.

icechunk_store = icechunk.in_memory_storage()
repo = icechunk.Repository.create(icechunk_store)
session = repo.writable_session("main")
vds.vz.to_icechunk(session.store)
session.commit("Create virtual store")

'Y308PPAWNJ1Q054MB8JG'