weiji14/cog3pio

Cloud-optimized GeoTIFF ... Parallel I/O 🦀

Kia Ora 👋

I am a Geospatial Data Scientist from Malaysia 🇲🇾 working remotely and thinking about where to go next! My background is in Geography, with a particular interest in Earth Observation using satellites 🛰️, having done cool research on Antarctic ice dynamics during my PhD in Aotearoa/New Zealand 🥝. Specifically, I work on applying Deep Learning methods to Remote Sensing datasets, keeping busy on the leading edge of data-driven research in the Earth Sciences.

These are some of the hats I wear:

• ☁️ Machine Learning Engineer in the GeoAI team at Development Seed
• 🌏 Core Developer of PyGMT: A Pythonic Interface to the Generic Mapping Tools

I'm currently:

• 🦀 Learning Rust, and trying to get into the GeoRust ecosystem
• ☯️ Developing a Python package called zen3geo for building Earth Observation and geospatial data pipelines
• 🔭 Making open source contributions to the Pytorch ecosystem and Pangeo community, all in the PyData world!

Want to connect?

• 💻 I'm open to building partnerships with folks working on open geospatial tools, specifically with a Remote Sensing focus, so reach out!
• 📫 Track me on Mastodon as @weiji14, or find some other links on my website
• 👀 Be sure to check out my cool Computer Vision projects:
  • DeepBedMap - Super Resolution Generative Adversarial Network to resolve the bed topography of Antarctica
  • DeepIceDrain - Finding subglacial lakes using unsupervised DBSCAN clustering
  • DeepSlide - Self-supervised learning on SAR data for change detection, for the FDL2022 NASA Earth Science challenge
  • Crevasse detection and navigation - Done using Image Segmentation + Reinforcement Learning
  • S2S2Net - Sentinel-2 Super-Resolution Segmentation Network using a Vision Transformer model (SegFormer)

⚡ Fun fact: I was brought up trilingual, and am fluent in English, 中文 & Bahasa Melayu.

cog3pio's Issues

Using cog3pio to determine byte ranges in COG files?

Basically same question as gauteh/hidefix#38 (comment) but for this library 😁

I'm building VirtualiZarr, an evolution of kerchunk, that allows you to determine byte ranges of chunks in netCDF files, but then concatenate the virtual representation of those chunks using xarray's API.

This works by creating a ChunkManifest object in-memory (one per netCDF Variable per file initially), then defining ways to merge those manifests.

What I'm wondering is if cog3pio's code could be useful to me as a way to generate the ChunkManifest for a netCDF file without using kerchunk/fsspec (see this issue). In other words I use cog3pio only to determine the byte ranges, not for actually reading the data. (I plan to actually read the bytes later as if it were zarr using the rust object-store crate, see zarr-developers/zarr-python#1661).

Q's:

• Is this idea dumb?
• Does cog3pio expose the byte range information currently?
• Can cog3pio read over S3?

cc @norlandrhagen