Problem while reading a self-made netcdf file with io.return_xarray_mfdataset

[simon3122]

Hello,

My problem is as follows: I want to read a self-made netcdf file with io.return_xarray_mfdataset.
The netcdf header gives:

group: netcdf4 {
dimensions:
y = 3454 ;
x = 5422 ;
t = 1 ;
variables:
float nav_lat(y, x) ;
nav_lat:axis = "Y" ;
nav_lat:standard_name = "latitude" ;
nav_lat:long_name = "Latitude" ;
nav_lat:units = "degrees_north" ;
nav_lat:nav_model = "grid_T" ;
float nav_lon(y, x) ;
nav_lon:axis = "X" ;
nav_lon:standard_name = "longitude" ;
nav_lon:long_name = "Longitude" ;
nav_lon:units = "degrees_east" ;
nav_lon:nav_model = "grid_T" ;
double time_centered(t) ;
time_centered:standard_name = "time" ;
time_centered:long_name = "Time axis" ;
time_centered:title = "Time" ;
time_centered:time_origin = "1958-01-01 00:00:00" ;
time_centered:bounds = "time_centered_bounds" ;
time_centered:units = "seconds since 1958-01-01" ;
time_centered:calendar = "gregorian" ;
double t(t) ;
t:axis = "T" ;
t:standard_name = "time" ;
t:long_name = "Time axis" ;
t:title = "Time" ;
t:time_origin = "1958-01-01 00:00:00" ;
t:bounds = "time_counter_bounds" ;
t:units = "seconds since 1958-01-01" ;
t:calendar = "gregorian" ;
float sossheig(t, y, x) ;
sossheig:_FillValue = 0.f ;
sossheig:long_name = "sea surface height" ;
sossheig:units = "m" ;
sossheig:online_operation = "average" ;
sossheig:interval_operation = "40s" ;
sossheig:interval_write = "5d" ;
sossheig:coordinates = "nav_lon nav_lat time_centered" ;
} // group netcdf4

My code is:
from oocgcm.core import io
chunks = (1727, 2711)
xr_chunks_tmean = {'y': chunks[-2], 'x': chunks[-1], 't':1}
vmean_xrt =io.return_xarray_mfdataset(filemean, chunks=xr_chunks_tmean)[vdict[vkey]['vname']][:]
I get the error output
ValueError: some chunks keys are not dimensions on this object: ['y', 'x', 't']

[simon3122]

This problem was when reading a Netcdf4 file.
But is not anymore when reading a Netcdf3 file, written with these options.

ds.to_netcdf(filenam,
'w',
format='NETCDF3_64BIT'
engine='scipy',
encoding={ vkey2:{'dtype':'float32'}})

[lesommer]

Hi Simon,
could you please look in more detail and let me know what is the vkey/vname that returns the exception ?
thanks.

[simon3122]

I am working on NEMO data with the following variables:
vkey = 'sea level'
vkey2 = vdict[vkey]['vname']
The dictionary is defined as follows:
vdict['sea level']={'vname': 'sossheig', ...

If it can provide information, here are my options to output in Netcdf4 (which raises a problem at the following reading stage)
ds.to_netcdf(filenam,'w',format='NETCDF4', engine='netcdf4', encoding={ vkey2:{'_FillValue':0,'dtype':'float32'}})

[lesommer]

I suspect this related to the options used in reading netcdf files in core.io.
NB : these options have changed in the trunk (see)

1. could you try to open your newly created dataset directly from xarray methods ?
2. try without specifying chunk size for 't' dimension {'y': chunks[-2], 'x': chunks[-1]}

[simon3122]

First, I noticed that I can not create the xarray.dataset: the error comes immediately from the open_mfdataset function

Here are my results:

1. Replacing
   io.return_xarray_mfdataset(filemean,chunks=xr_chunks_tmean)
   with
   xr.open_mfdataset(filemean,chunks=xr_chunks_tmean,engine='netcdf4',lock=False)

   does not change the result

2. Leaving out the 't' dimension provides this error:
   ValueError: some chunks keys are not dimensions on this object: ['y', 'x']

I can add another result : when specifying chunk=None in the xarray function, the dataset again can not be read properly:
print xr.open_mfdataset(filemean,chunks=None,engine='netcdf4',lock=False)
gives

<xarray.Dataset>
Dimensions:  ()
Coordinates:
    *empty*
Data variables:
    *empty*

[simon3122]

The blocking version was when writing the file
ds.to_netcdf(filenam,'w','NETCDF4', 'netcdf4'...}})
which gives the following header

group: netcdf4 {
  dimensions:
    y = 3454 ;
    x = 5422 ;

In fact, it works when writing the file
ds.to_netcdf(filenam,'w',format='NETCDF4', engine='netcdf4'...}})
which gives the following header

dimensions:
    y = 3454 ;
    x = 5422 ;

The first version seems to create an inner netcdf4 group which might not be a standard Netcdf4 file.
I understand that Xarray reading function may not read properly the first version (with an inner group) if the option 'group' is not filled.