weiji14/deepicedrain

Mapping and monitoring deep subglacial water activity in Antarctica using remote sensing and machine learning, with ICESat-2!

Kia Ora 👋

I am a Geospatial Data Scientist from Malaysia 🇲🇾 working remotely and thinking about where to go next! My background is in Geography, with a particular interest in Earth Observation using satellites 🛰️, having done cool research on Antarctic ice dynamics during my PhD in Aotearoa/New Zealand 🥝. Specifically, I work on applying Deep Learning methods to Remote Sensing datasets, keeping busy on the leading edge of data-driven research in the Earth Sciences.

These are some of the hats I wear:

• ☁️ Machine Learning Engineer in the GeoAI team at Development Seed
• 🌏 Core Developer of PyGMT: A Pythonic Interface to the Generic Mapping Tools

I'm currently:

• 🦀 Learning Rust, and trying to get into the GeoRust ecosystem
• ☯️ Developing a Python package called zen3geo for building Earth Observation and geospatial data pipelines
• 🔭 Making open source contributions to the Pytorch ecosystem and Pangeo community, all in the PyData world!

Want to connect?

• 💻 I'm open to building partnerships with folks working on open geospatial tools, specifically with a Remote Sensing focus, so reach out!
• 📫 Track me on Mastodon as @weiji14, or find some other links on my website
• 👀 Be sure to check out my cool Computer Vision projects:
  • DeepBedMap - Super Resolution Generative Adversarial Network to resolve the bed topography of Antarctica
  • DeepIceDrain - Finding subglacial lakes using unsupervised DBSCAN clustering
  • DeepSlide - Self-supervised learning on SAR data for change detection, for the FDL2022 NASA Earth Science challenge
  • Crevasse detection and navigation - Done using Image Segmentation + Reinforcement Learning
  • S2S2Net - Sentinel-2 Super-Resolution Segmentation Network using a Vision Transformer model (SegFormer)

⚡ Fun fact: I was brought up trilingual, and am fluent in English, 中文 & Bahasa Melayu.

deepicedrain's Issues

NZ Antarctic Science Conference 2021 poster

Hi 👋

If you've noticed my poster at the 2021 Antarctic Science Conference, here's the place to find out more! Feel free to look around the repository, ask any questions or leave a comment below 😄

For more details, I recommend checking out Chapter 4 of my recently submitted PhD thesis at https://github.com/weiji14/phdthesis/releases/tag/v0.5.0.

The poster itself was made on Libreoffice Draw Version 6.1.5.2 using the #BetterPoster style. The figures/maps were created using PyGMT v0.2.1. Code to reproduce the figures are available in this repository and also at a sister repository over at https://github.com/weiji14/nzasc2021.

Design a better structure for the Antarctic regions bounding box polygon container

Scattered across deepicedrain's various scripts is a 'regions' variable (also 'ice_boundaries', 'drainage_basins', etc) that holds a geographical bounding box and/or polygon which is needed for spatial subseting and plotting operations. However this is getting really messy/unwieldy and should be vendored off at some point to a package like regionmask.

This 'regions' container should hold Antarctic geographical boundaries (both a Polygon and BoundingBox) and have a 'placename' (e.g. "amundsen_sea_embayment") and 'full name' (e.g. "Amundsen Sea Embayment") attribute.

Related: d2b0199, 85b6ffa, #190

Dependabot can't resolve your Python dependency files

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Creating virtualenv deepicedrain-zVe-mzmo-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

[SolverProblemError]
Because xrviz (0.1.4) depends on xarray (>=0.14.1)
 and no versions of xrviz match >0.1.4,<0.2.0, xrviz (>=0.1.4,<0.2.0) requires xarray (>=0.14.1).
So, because deepicedrain depends on both xrviz (^0.1.4) and xarray (0.12.4.dev547+g6f6aae7d git), version solving failed.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your Python dependency files

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Creating virtualenv deepicedrain-pzgPX--m-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

  PackageNotFound

  Package dask (2.26.0) not found.

  at /usr/local/.pyenv/versions/3.8.9/lib/python3.8/site-packages/poetry/repositories/pool.py:144 in package
      140│                     self._packages.append(package)
      141│ 
      142│                     return package
      143│ 
    → 144│         raise PackageNotFound("Package {} ({}) not found.".format(name, version))
      145│ 
      146│     def find_packages(
      147│         self, dependency,
      148│     ):

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your Python dependency files

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Creating virtualenv deepicedrain-PEqzzY3q-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

  PackageNotFound

  Package dask (0.18.0) not found.

  at /usr/local/.pyenv/versions/3.8.9/lib/python3.8/site-packages/poetry/repositories/pool.py:144 in package
      140│                     self._packages.append(package)
      141│ 
      142│                     return package
      143│ 
    → 144│         raise PackageNotFound("Package {} ({}) not found.".format(name, version))
      145│ 
      146│     def find_packages(
      147│         self, dependency,
      148│     ):

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot couldn't find a Dockerfile for this project

Dependabot couldn't find a Dockerfile for this project.

Dependabot requires a Dockerfile to evaluate your project's current Docker dependencies. It had expected to find one at the path: /Dockerfile.

If this isn't a Docker project, or if it is a library, you may wish to disable updates for it from within Dependabot.

Add Cryosat-2 (CryoTEMPO-EOLIS) Swath Thematic Point Product

Just found out about the “CryoTEMPO-EOLIS: Elevation Over Land Ice from Swath” project, Looks like a more accessible way of getting Cryosat-2 radar altimetry data, though it's still pretty limited for the Antarctic (currently 2020 May only).

Specifically, the Swath Thematic Point Product (EOLIS POINTS) appears to be a promising dataset that should nicely compliment the ICESat-2 point cloud processing in this repo. Should look into it in the long-term (read: early 2021 after my PhD).

References:

• Download location:
  • https://science-pds.cryosat.esa.int/#Cry0Sat2_data/TEMPO_SWATH_POINT
• News articles:
  • https://cryotempo-eolis.org/2020/05/07/product-release-v1/
  • https://earth.esa.int/eogateway/news/systematic-swath-elevation-and-monthly-dems-over-the-antarctica-ice-sheet-margin-from-cryosat-2
• Information about product and download location:
  • https://cryotempo-eolis.org/products/#point
  • https://cryotempo-eolis.org/wp-content/uploads/2020/05/CryoTempo-ThematicProduct-ProductHandbook_signed-2.pdf
  • ftp://science-pds.cryosat.esa.int/TEMPO_SWATH_POINT/LATEST/
• Citation:
  • Gourmelen, N., Escorihuela, M.J.,Shepherd, A., Foresta, L., Muir, A., Garcia-Mondéjar, A., Roca, M., Baker S.G., Drinkwater M.R. CryoSat-2 swath interferometric altimetry for mapping ice elevation and elevation change, Advances in Space Research, Volume 62, Issue 6, 2018,Pages 1226-1242, https://doi.org/10.1016/j.asr.2017.11.014.

Dependabot couldn't find the submodule /.gitmodules

Dependabot couldn't find a .gitmodules for this project.

Dependabot requires a .gitmodules to evaluate your project's current Git dependencies. It had expected to find one at the path: /.gitmodules.

If this isn't a Git project, or if it is a library, you may wish to disable updates for it from within Dependabot.

View the update logs.

Reading multiple ICESat-2 ATL11 point cloud data nicely via Zarr

Gathering some notes on how best to read multiple ICESat-2 ATL11 data (basically a point cloud) in a user friendly way, with metadata preserved!

TLDR: Be able to do xr.open_mfdataset("ATL11_*.h5", engine="zarr", ...).

Inspired by the blog post "Cloud-Performant NetCDF4/HDF5 Reading with the Zarr Library". Zarr is an amazing project, and I really like the .zmetadata json file which can be opened with a text editor and tell you stuff about the data. The dream would be to read HDF5 files in an out-of-core manner with Zarr like speed/abilities (through the .zmetadata pointer).

Jupyter notebook demo can be found at https://github.com/rsignell-usgs/hurricane-ike-water-levels/blob/master/coawst_3ways.ipynb. See also discussion thread at zarr-developers/zarr-python#535 on "Using the Zarr library to read HDF5".

Main hurdles to get through, dependent on upstream, there's two 'separate' parts:

• Reading a single HDF5 dataset via Zarr
  • On xarray - Need chunk_store argument to use Zarr to read HDF5 - wait for pydata/xarray#3804
  • On zarr (though it's partly a dependency problem for me to work out)
    • Upgrading numcodecs from 0.6.3 to 0.6.4 (#11 ✔️) and to 0.7.2 (#160 ✔️) required to use Zarr 2.4.0 or newer (24b6917 ✔️), but fails due to compilation issues - wait for zarr-developers/numcodecs#224 or use conda-forge's numcodecs package.
    • Specifically, use this hdf5 branch of Zarr to read HDF5 files via Zarr (TODO!).
• Reading multiple Zarr-like files via xarray/intake:
  • On xarray - Streamline opening multiple zarr files via xr.open_mfdataset - wait for pydata/xarray#4187 / pydata/xarray#4003
  • On intake-xarray - intake.open_ndzarr will break with the above ☝️ - wait for intake/intake-xarray#70

Current situation in that I do HDF5 -> Zarr conversion, and read from that. It would be nice to stick to the original HDF5 data source (though I might need to flatten the nested ICESat-2 ATL11 data structure). Note that I'm not necessarily after raw speed, I just prefer readability (i.e. having xarray's wonderful annotated metadata).

Other open Issues/Pull Requests:

• zarr-developers/zarr-python#321 - Provide offset for memory mapping / contiguous layout
• zarr-developers/zarr-python#377 - RFC: Optionally support memory-mapping DirectoryStore values
• zarr-developers/zarr-python#556 - File Chunk Store
• zarr-developers/zarr-python#546 - POC: Add FSStore

Blog posts:

• https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with-zarr-fsspec-and-intake-3d3a3e7cb935
• https://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314
• https://medium.com/informatics-lab/arrays-on-the-fly-9509b6b2e46a
• https://medium.com/pangeo/xpublish-ff788f900bbf

You can tell I had way too many tabs open on my browser 😆

Dependabot can't resolve your Python dependency files

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Creating virtualenv deepicedrain-aZZz2Wvc-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

  PackageNotFound

  Package dask (2.9.0) not found.

  at /usr/local/.pyenv/versions/3.8.9/lib/python3.8/site-packages/poetry/repositories/pool.py:144 in package
      140│                     self._packages.append(package)
      141│ 
      142│                     return package
      143│ 
    → 144│         raise PackageNotFound("Package {} ({}) not found.".format(name, version))
      145│ 
      146│     def find_packages(
      147│         self, dependency,
      148│     ):

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

A question about the line chart in readme(Elevation time-series at Crossover Points)

Hello, I have a question regarding the line chart in readme. Specifically, I would like to inquire about the elevation anomaly at crossover points. From my understanding, there should only be one data point per cycle for this anomaly. However, I have noticed that the crossover points data appears to be too time-intensive.

Continuous subglacial lake detection in Antarctica

Find new active subglacial lakes in Antarctica automatically when new ICESat-2 datasets are available!

Processing Pipeline:

• Step 1. Setup a scheduled Continuous Integration service to detect when new data releases are available
  • Ping https://n5eil02u.ecs.nsidc.org/opendap/hyrax/ATLAS/ATL06.003/catalog.xml or alternative source to get latest date
  • Look at using intake-thredds or siphon or jupyterlab_thredds
  • 👋 Need help figuring out how to authenticate to NASA URS without exposing personal credentials
• Step 2. If new data found, run the data processing pipeline
  • I.e. atl06_to_atl11, ..., atlxi_dhdt.py
  • 🤔 How to only run only a 'diff' between data releases instead of everything?
• Step 3. Generate maps of new subglacial lake locations
  • Will need to run some cluster analysis, generate hydropotential contours, do image differencing, or something along those lines
  • Include metadata information such as estimated lake area/volume, amount drained/filled, etc
  • ❓ What other remote sensing information might help with this, and what should the output look like?

Open to ideas/suggestions, especially on the points with emojis above ⬆️.

References:

• Fricker, H. A., Scambos, T. A., Bindschadler, R., & Padman, L. (2007). An Active Subglacial Water System in West Antarctica Mapped from Space. Science, 315(5818), 1544–1548. https://doi.org/10.1126/science.1136897
• Siegfried, M. R., & Fricker, H. A. (2018). Thirteen years of subglacial lake activity in Antarctica from multi-mission satellite altimetry. Annals of Glaciology, 1–14. https://doi.org/10.1017/aog.2017.36
• Smith, B. E., Fricker, H. A., Joughin, I. R., & Tulaczyk, S. (2009). An inventory of active subglacial lakes in Antarctica detected by ICESat (2003–2008). Journal of Glaciology, 55(192), 573–595. https://doi.org/10.3189/002214309789470879

Dependabot can't resolve your Python dependency files

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Creating virtualenv deepicedrain-jvHdo1ed-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

  PackageNotFound

  Package dask (2.18.1) not found.

  at /usr/local/.pyenv/versions/3.8.8/lib/python3.8/site-packages/poetry/repositories/pool.py:144 in package
      140│                     self._packages.append(package)
      141│ 
      142│                     return package
      143│ 
    → 144│         raise PackageNotFound("Package {} ({}) not found.".format(name, version))
      145│ 
      146│     def find_packages(
      147│         self, dependency,
      148│     ):

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.