UnicodeDecodeError

[ocean-data-challenges]

UnicodeDecodeError

[jejjohnson]

Seconded. I had the same issue. I cannot open it using the xr.backends.PydapDataStore.open code. It seems we're not the only one with this problem. See issue here. Perhaps @mballaro might have a solution?

Demo Snippet

credentials_path = "/home/johnsonj/credentials/aviso.yaml"
my_aviso_session = load_aviso_credentials(credentials_path)
url_alongtrack = 'https://tds.aviso.altimetry.fr/thredds/dodsC/2021a-SSH-mapping-OSE-along-track-data'
url_map = 'https://tds.aviso.altimetry.fr/thredds/dodsC/2021a-SSH-mapping-OSE-grid-data'
url_ds_alg = f'{url_alongtrack}/dt_gulfstream_alg_phy_l3_20161201-20180131_285-315_23-53.nc'
store_ds_alg = xr.backends.PydapDataStore.open(url_ds_alg, session=my_aviso_session)

Error Message

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
Input In [90], in <cell line: 7>()
      5 # url_ds_alg = f'{url_alongtrack}/dt_gulfstream_alg_phy_l3_20161201-20180131_285-315_23-53.nc'
      6 url_ds_alg = f'{url_alongtrack}/dt_gulfstream_alg_phy_l3_20161201-20180131_285-315_23-53.nc'
----> 7 store_ds_alg = xr.backends.PydapDataStore.open(url_ds_alg, session=my_aviso_session)

File ~/miniconda3/envs/oi_torch_py39/lib/python3.9/site-packages/xarray/backends/pydap_.py:93, in PydapDataStore.open(cls, url, session)
     90 @classmethod
     91 def open(cls, url, session=None):
---> 93     ds = pydap.client.open_url(url, session=session)
     94     return cls(ds)

File ~/miniconda3/envs/oi_torch_py39/lib/python3.9/site-packages/pydap/client.py:66, in open_url(url, application, session, output_grid, timeout)
     58 def open_url(url, application=None, session=None, output_grid=True,
     59              timeout=DEFAULT_TIMEOUT):
     60     """
     61     Open a remote URL, returning a dataset.
     62 
     63     set output_grid to False to retrieve only main arrays and
     64     never retrieve coordinate axes.
     65     """
---> 66     dataset = DAPHandler(url, application, session, output_grid,
     67                          timeout).dataset
     69     # attach server-side functions
     70     dataset.functions = Functions(url, application, session)

File ~/miniconda3/envs/oi_torch_py39/lib/python3.9/site-packages/pydap/handlers/dap.py:57, in DAPHandler.__init__(self, url, application, session, output_grid, timeout)
     55 if not r.charset:
     56     r.charset = 'ascii'
---> 57 dds = r.text
     59 dasurl = urlunsplit((scheme, netloc, path + '.das', query, fragment))
     60 r = GET(dasurl, application, session, timeout=timeout)

File ~/miniconda3/envs/oi_torch_py39/lib/python3.9/site-packages/webob/response.py:622, in Response._text__get(self)
    620 decoding = self.charset or self.default_body_encoding
    621 body = self.body
--> 622 return body.decode(decoding, self.unicode_errors)

UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128)

---

Temporary Solution

I simply download it using the standard wget commands in bash. Did the minor changes.

# grab arguments
dir=$3
username=$1
password=$2

# make train directory
train_dir=$dir/train
echo train_dir

# make ref directory
ref_dir=$dir/ref

# make test directory
test_dir=$dir/test

# download all datasets
wget --user $username --password $password --directory-prefix=$train_dir 'https://tds.aviso.altimetry.fr/thredds/fileServer/2021a-SSH-mapping-OSE-along-track-data/dt_gulfstream_alg_phy_l3_20161201-20180131_285-315_23-53.nc'

# GULFSTREAM (SARAL/Altika obs)
wget --user $username --password $password --directory-prefix=$train_dir 'https://tds.aviso.altimetry.fr/thredds/fileServer/2021a-SSH-mapping-OSE-along-track-data/dt_gulfstream_alg_phy_l3_20161201-20180131_285-315_23-53.nc'

# GULFSTREAM (Jason-3 obs)
wget --user $username --password $password --directory-prefix=$train_dir
'https://tds.aviso.altimetry.fr/thredds/fileServer/2021a-SSH-mapping-OSE-along-track-data/dt_gulfstream_j3_phy_l3_20161201-20180131_285-315_23-53.nc'

# GULFSTREAM (Sentinel-3A obs)
wget --user $username --password $password --directory-prefix=$train_dir 'https://tds.aviso.altimetry.fr/thredds/fileServer/2021a-SSH-mapping-OSE-along-track-data/dt_gulfstream_s3a_phy_l3_20161201-20180131_285-315_23-53.nc'


# GULFSTREAM (Jason-2 obs)
wget --user $username --password $password --directory-prefix=$train_dir 'https://tds.aviso.altimetry.fr/thredds/fileServer/2021a-SSH-mapping-OSE-along-track-data/dt_gulfstream_j2g_phy_l3_20161201-20180131_285-315_23-53.nc'


# GULFSTREAM (Jason-2n obs)
wget --user $username --password $password --directory-prefix=$train_dir 'https://tds.aviso.altimetry.fr/thredds/fileServer/2021a-SSH-mapping-OSE-along-track-data/dt_gulfstream_j2n_phy_l3_20161201-20180131_285-315_23-53.nc'

# GULFSTREAM (Cryosat-2 obs)
wget --user $username --password $password --directory-prefix=$train_dir 'https://tds.aviso.altimetry.fr/thredds/fileServer/2021a-SSH-mapping-OSE-along-track-data/dt_gulfstream_h2g_phy_l3_20161201-20180131_285-315_23-53.nc'

# GULFSTREAM (Haiyang-2 obs)
wget --user $username --password $password --directory-prefix=$train_dir 'https://tds.aviso.altimetry.fr/thredds/fileServer/2021a-SSH-mapping-OSE-along-track-data/dt_gulfstream_h2g_phy_l3_20161201-20180131_285-315_23-53.nc'


# download reference map data
wget --user $username --password $password --directory-prefix=$ref_dir 'https://tds.aviso.altimetry.fr/thredds/fileServer/2021a-SSH-mapping-OSE-grid-data/mdt.nc'

# download test data
wget --user $username --password $password --directory-prefix=$test_dir 'https://tds.aviso.altimetry.fr/thredds/fileServer/2021a-SSH-mapping-OSE-along-track-data/dt_gulfstream_c2_phy_l3_20161201-20180131_285-315_23-53.nc'

Demo Usage:

bash dl_data.sh <aviso-username> <aviso-password> <directory-to-save>