csdms/bmi-topography

Fetch and cache land elevation data from OpenTopography through an API, BMI, or CLI

bmi-topography

bmi-topography is a Python library for fetching and caching land elevation data using the OpenTopography REST API.

The bmi-topography library provides access to the following global raster datasets:

• SRTMGL3 (SRTM GL3 90m)
• SRTMGL1 (SRTM GL1 30m)
• SRTMGL1_E (SRTM GL1 Ellipsoidal 30m)
• AW3D30 (ALOS World 3D 30m)
• AW3D30_E (ALOS World 3D Ellipsoidal, 30m)
• SRTM15Plus (Global Bathymetry SRTM15+ V2.1)
• NASADEM (NASADEM Global DEM)
• COP30 (Copernicus Global DSM 30m)
• COP90 (Copernicus Global DSM 90m)

The library includes an API and a CLI that accept the dataset type, a latitude-longitude bounding box, and the output file format. Data are downloaded from OpenTopography and cached locally. The cache is checked before downloading new data. Data from a cached file can optionally be loaded into an xarray DataArray through rioxarray.

The bmi-topography API is wrapped with a Basic Model Interface (BMI), which provides a standard set of functions for coupling with data or models that also expose a BMI. More information on the BMI can found in its documentation.

Installation

Install the latest stable release of bmi-topography with pip:

pip install bmi-topography

or with conda:

conda install -c conda-forge bmi-topography

The bmi-topography library can also be built and installed from source. The library uses several other open source libraries, so a convenient way of building and installing it is within a conda environment. After cloning or downloading the bmi-topography repository, change into the repository directory and set up a conda environment with the included environment file:

conda env create --file=environment.yml

Then build and install bmi-topography from source with

pip install -e .

API key

To better understand usage, OpenTopography requires an API key to access datasets they host. Getting an API key is easy, and it's free: just follow the instructions in the link above.

Once you have an API key, there are three ways to use it with bmi-topography:

1. parameter: Pass the API key as a string through the api_key parameter.
2. environment variable: In the shell, set the OPENTOPOGRAPHY_API_KEY environment variable to the API key value.
3. dot file: Put the API key in the file .opentopography.txt in the current directory or in your home directory.

If you attempt to use bmi-topography to access an OpenTopography dataset without an API key, you'll get a error like this:

requests.exceptions.HTTPError: 401 Client Error: This dataset requires an API Key for access.

Examples

A brief example of using the bmi-topography API is given in the following steps.

Start a Python session and import the Topography class:

>>> from bmi_topography import Topography

For convenience, a set of default parameter values for Topography are included in the class definition. Copy these and modify them with custom values:

>>> params = Topography.DEFAULT.copy()
>>> params["south"] = 39.93
>>> params["north"] = 40.00
>>> params["west"] = -105.33
>>> params["east"] = -105.26
>>> params
{'dem_type': 'SRTMGL3',
 'south': 39.93,
 'north': 40.0,
 'west': -105.33,
 'east': -105.26,
 'output_format': 'GTiff',
 'cache_dir': '~/.bmi_topography'}

These coordinate values represent an area around Boulder, Colorado.

Make a instance of Topography with these parameters:

>>> boulder = Topography(**params)

then fetch the data from OpenTopography:

>>> boulder.fetch()
PosixPath('/Users/mpiper/.bmi_topography/SRTMGL3_39.93_-105.33_40.0_-105.26.tif')

This step might take a few moments, and it will increase for requests of larger areas. Note that the file has been saved to a local cache directory.

Load the data into an xarray DataArray for further work:

>>> boulder.load()
<xarray.DataArray 'SRTMGL3' (band: 1, y: 84, x: 84)>
array([[[2052, 2035, ..., 1645, 1643],
        [2084, 2059, ..., 1643, 1642],
        ...,
        [2181, 2170, ..., 1764, 1763],
        [2184, 2179, ..., 1773, 1769]]], dtype=int16)
Coordinates:
  * band         (band) int64 1
  * x            (x) float64 -105.3 -105.3 -105.3 ... -105.3 -105.3 -105.3
  * y            (y) float64 40.0 40.0 40.0 40.0 ... 39.93 39.93 39.93 39.93
    spatial_ref  int64 0
Attributes:
    _FillValue:    0.0
    scale_factor:  1.0
    add_offset:    0.0
    units:         meters
    location:      node

Note that coordinate reference system information is stored in the spatial_ref non-dimension coordinate:

>>> boulder.da.spatial_ref
<xarray.DataArray 'spatial_ref' ()>
array(0)
Coordinates:
    spatial_ref  int64 0
Attributes:
    crs_wkt:                      GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["...
    semi_major_axis:              6378137.0
    semi_minor_axis:              6356752.314245179
    inverse_flattening:           298.257223563
    reference_ellipsoid_name:     WGS 84
    longitude_of_prime_meridian:  0.0
    prime_meridian_name:          Greenwich
    geographic_crs_name:          WGS 84
    grid_mapping_name:            latitude_longitude
    spatial_ref:                  GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["...
    GeoTransform:                 -105.33041666668363 0.000833333333333144 0....

Display the elevations with the default xarray DataArray plot method.

>>> import matplotlib.pyplot as plt
>>> boulder.da.plot()
>>> plt.show()

For examples with more detail, see the two Jupyter Notebooks, Python script, and shell script included in the examples directory of the bmi-topography repository.

User and developer documentation for bmi-topography is available at https://bmi-topography.readthedocs.io.

bmi-topography's Issues

Fix documentation build

Previously, I had been using pandoc to translate the Markdown files in the root directory of the repository into sphinx-friendly reStructuredText files for the documentation. In the move to nox for managing automated tasks, this got lost. It's now breaking. I need to evaluate if this workflow is still needed and fix it.

Support all available OpenTopography global raster datasets

Hi Mark,

Big fan of bmi-topography here. Would it be feasible to support NASADEM and copernicus as well?
See: https://opentopography.org/news/nasadem-and-copernicus-dem

Benjamin

Causes of a 401 error

If Topography receives a 401 status when trying to fetch a dataset, it sets the reason to "This dataset requires an API Key for access". Is this the only reason for a 401? What if a bad key is being used (e.g. Topography found a key but there's a typo in it, or it's an old key that has been revoked)? What if a demo key is used but it's reached its maximum number of downloads?

Replace multiple CLI options with a single --config-file option

The bmi-topography CLI includes the following options for configuring a call to the OpenTopography REST API:

• dem_type
• south
• north
• west
• east
• output_format

It's cumbersome to set all of these options. The CLI would easier to use and script if values for these options were instead placed in a text configuration file (e.g., yaml or toml), then referenced with an option such as --config-file.

Raise exception with helpful message for missing API key

Currently, if a request is made of the OpenTopography API without an API key, Topography raises a requests.HTTPError exception that shows the status code and the URL. It would be good to help a user know why the request failed by adding a message about needing an API key.

Test matrix of datasets and output formats

The current version of the test_fetch function is from when only one DEM type and one output format were available in the Topography class. With the addition of new dem types (e.g., #13) and output formats (e.g., #14), a matrix of tests should be set up to test each combination.

Warn user when request exceeds maximum download size from Opentopography

OpenTopography has a ceiling on the size of a data request--see #15 for an example. The Topography data component should warn a user when an instance is created that would request an area exceeding this threshold.

Underscores in bmi-topography command line options

I'm wondering about the _s in the bmi-topography command line options (e.g. --no_fetch, --api_key). It seems more standard to use -s in such cases (e.g. --no-fetch, --api-key).

Server down

Hi Mark,

I noticed the open-topography server is not always online. Would it be possible to check for that when running bmi-topography?

Cheers,
Benjamin

Update contributing guidelines in line with migration from `make` to `nox`

Reference: Issue #51 Use nox instead of make for building and testing the project & commit #e313b21

Following CSDMS best practices, I've updated bmi-topography to use nox instead of make for building and testing the project.

There is no makefile anymore. CI pipeline is now using nox (Flexible test automation) package.

Affected line(s):

bmi-topography/CONTRIBUTING.md

Lines 64 to 72 in 993df5f

	3. Install your local copy into a conda environment. A conda enviroment file is
	supplied at the root of the repository. Assuming you have conda installed,
	this is how you set up your fork for local development:
	``` {.shell}
	$ cd bmi-topography
	$ conda env create --file=environment.yml
	$ conda activate topography
	$ make install

bmi-topography/CONTRIBUTING.md

Lines 83 to 91 in 993df5f

	5. When you're done making changes, check that your changes pass
	ruff and the tests:
	``` {.shell}
	$ make lint
	$ make test
	```
	Both ruff and pytest are included in the environment.

Proposed change(s):

• Remove $ make install line from
  

  bmi-topography/CONTRIBUTING.md

  Line 72 in 993df5f

  $ make install

• Change pre-commit tests recommendation:

5. When you're done making changes, check that your changes pass ruff and the tests:
   5.1 Check available sessions (test stages). See nox docs -> list all available sessions

   $ nox --list
   

   5.2 Run linter and tests.(s) See nox docs -> run a particular set of sessions

   $ nox -s format
   $ nox -s test 
   

Support download of ALOS data

The OpenTopography API, on which bmi-topography is built, supports not only STRM, but also ALOS. bmi-topography should be enhanced to include downloads of

• ALOS World 3D 30m
• ALOS World 3D (Ellipsoidal)

from the OpenTopography API.

Error when fetching

Thanks for this exceptional library,

would you help why I get this error,

HTTPError: 400 Client Error: 400 for url: https://portal.opentopography.org/API/globaldem?demtype=SRTMGL3&south=5&north=45&west=10&east=70&outputFormat=GTiff

This is the code snippet.

topo = Topography(
    dem_type="SRTMGL3",
    south=5,
    north=45,
    west=10,
    east=70,
    output_format="GTiff",
    cache_dir="."
    )

fname = topo.fetch()
print(fname)

test_load fails on Windows with python=3.10

The test_load test is failing on Windows, python=3.10, but only when run through GitHub Actions. It passes when run locally. It also passes on Windows, python<3.10, when run through Actions.

This may be an upstream problem with xarray or rioxarray. Here's the stack trace:

================================== FAILURES ===================================
__________________________________ test_load __________________________________
Traceback (most recent call last):
  File "C:\Miniconda\envs\topography\lib\site-packages\xarray\core\common.py", line 284, in __setattr__
    object.__setattr__(self, name, value)
AttributeError: 'DataArray' object has no attribute '_file_obj'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\a\bmi-topography\bmi-topography\tests\test_topography.py", line 106, in test_load
    topo.load()
  File "D:\a\bmi-topography\bmi-topography\bmi_topography\topography.py", line 177, in load
    self._da = rioxarray.open_rasterio(_filename)
  File "C:\Miniconda\envs\topography\lib\site-packages\rioxarray\_io.py", line 775, in open_rasterio
    result._file_obj = manager
  File "C:\Miniconda\envs\topography\lib\site-packages\xarray\core\common.py", line 292, in __setattr__
    raise AttributeError(
AttributeError: cannot set attribute '_file_obj' on a 'DataArray' object. Use __setitem__ styleassignment (e.g., `ds['name'] = ...`) instead of assigning variables.

Request smaller dataset when testing

When testing, Topography.DEFAULTS are used, which fetches a dataset of about 5 MB. To be kind to OpenTopography, we should request a smaller dataset, on the order of KB.

Update docs and examples to use API key

The documentation and examples should be updated to show how to use an API key from OpenTopography.

Add funding acknowledgement

Reference Greg and Eric's EarthCube award.

Allow directionality in BMI grid spacing

If negative grid spacing values are allowed in the BMI (see csdms/bmi#110), then bmi-topography should allow them, as well.

See discussion in #47 (comment).

Making this change would allow 4d080a6 to be reverted.

Move project metadata from setup.cfg to pyproject.toml

A pyproject.toml file seems to be preferred over setup.cfg for static project metadata. Move as much as possible from the latter to the former.

Change from xarray to rioxarray

The xarray.open_rasterio method has been deprecated in favor of rioxarray. The Topography class should be updated accordingly.

Change property name from dataarray to da

The name dataarray is verbose, and da is used in the xarray documentation.

Fix broken rtfd configuration file

Docs builds are broken because of an invalid configuration setting. The error:

Error

Problem in your project's configuration. Invalid configuration option "python.system_packages". This configuration key has been deprecated and removed. Refer to https://blog.readthedocs.com/drop-support-system-packages/ to read more about this change and how to upgrade your config file.

Visualize example data in README and docs

In the README and docs, Topography is used to fetch sample data. The data should be visualized. (See the bmi-geotiff docs, for example.)

Extension of AAIGrid

When you download a DEM using the output_format = "AAIGrid”, the data format of the file is OK, but the extension of it still says “.tif” which is a little confusing.

Update format job in Actions

black has a new syntax, so the following warning is shown when running the format job:

Unexpected input(s) 'args', valid inputs are ['options', 'src', 'black_args', 'version']

We should update the format job to fix this.

Add API_key field

Beginning 2022-02-01, OpenTopography will require an API key to access its global datasets.

The Topography class will need a API_Key property to satisfy this requirement.

Add new OT global raster datasets

OpenTopography has made available two more global raster datasets:

• EU_DTM (DTM 30m)
• GEDI_L3 (DTM 1000m)

The Topography data component should be updated to access these new datasets.

Group data component cache files under a single directory

CSDMS now has several data components. Most cache data in a local directory to avoid fetching data that has already been downloaded; e.g., the Topography data component uses the directory ~/.bmi-topography. Instead of having multiple hidden directories in a user's home directory (one for each data component), we should group the cache directories under one hidden directory, ~/.csdms. The cache directory for the Topography component would then be ~/.csdms/bmi-topography; for GeoTiff, ~/.csdms/bmi-geotiff, etc.

This is issue is intended as a placeholder for all data components, and is motivated by a discussion in csdms/bmi-wavewatch3#4.

Set minimum Python >= 3.10

In keeping with our informal policy of supporting the latest three minor Python versions, the minimum Python version should be bumped to 3.10.