Giter Site home page Giter Site logo

sea_ice_drift's Introduction

Build Status Coverage Status DOI

Sea ice drift from Sentinel-1 SAR data

A computationally efficient, open source feature tracking algorithm, called ORB, is adopted and tuned for retrieval of the first guess sea ice drift from Sentinel-1 SAR images. Pattern matching algorithm based on MCC calculation is used further to retrieve sea ice drift on a regular grid.

References:

  • Korosov A.A. and Rampal P., A Combination of Feature Tracking and Pattern Matching with Optimal Parametrization for Sea Ice Drift Retrieval from SAR Data, Remote Sens. 2017, 9(3), 258; doi:10.3390/rs9030258
  • Muckenhuber S., Korosov A.A., and Sandven S., Open-source feature-tracking algorithm for sea ice drift retrieval from Sentinel-1 SAR imagery, The Cryosphere, 10, 913-925, doi:10.5194/tc-10-913-2016, 2016

Running with Docker

# run ipython with SeaIceDrift
docker run --rm -it -v /path/to/data:/home/jovyan/work nansencenter/seaicedrift ipython

# run jupyter notebook with SeaIceDrift
docker run --rm -p 8888:8888 -v /path/to/data/and/notebooks:/home/jovyan/work nansencenter/seaicedrift

Installation on Ubuntu

# install some requirements with apt-get
apt-get install -y --no-install-recommends libgl1-mesa-glx gcc build-essential

# install some requirements with conda
conda install -c conda-forge gdal cartopy opencv

# install other requirements with pip
pip install netcdf4 nansat

# clone code
git clone https://github.com/nansencenter/sea_ice_drift.git
cd sea_ice_drift

# install SeaIceDrift
python setup.py install

Usage example

# download example datasets
wget https://github.com/nansencenter/sea_ice_drift_test_files/raw/master/S1B_EW_GRDM_1SDH_20200123T120618.tif
wget https://github.com/nansencenter/sea_ice_drift_test_files/raw/master/S1B_EW_GRDM_1SDH_20200125T114955.tif

# start Python and import relevant libraries
import numpy as np
import matplotlib.pyplot as plt
from nansat import Nansat
from sea_ice_drift import SeaIceDrift

# open pair of satellite images using Nansat and SeaIceDrift
filename1='S1B_EW_GRDM_1SDH_20200123T120618.tif'
filename2='S1B_EW_GRDM_1SDH_20200125T114955.tif'
sid = SeaIceDrift(filename1, filename2)

# run ice drift retrieval using Feature Tracking
uft, vft, lon1ft, lat1ft, lon2ft, lat2ft = sid.get_drift_FT()

# plot
plt.quiver(lon1ft, lat1ft, uft, vft);plt.show()

# define a grid (e.g. regular)
lon1pm, lat1pm = np.meshgrid(np.linspace(-33.5, -30.5, 50),
                             np.linspace(83.6, 83.9, 50))

# run ice drift retrieval for regular points using Pattern Matching
# use results from the Feature Tracking as the first guess
upm, vpm, apm, rpm, hpm, lon2pm, lat2pm = sid.get_drift_PM(
        lon1pm, lat1pm,
        lon1ft, lat1ft,
        lon2ft, lat2ft)
# select high quality data only
gpi = rpm*hpm > 4

# plot high quality data on a regular grid
plt.quiver(lon1pm[gpi], lat1pm[gpi], upm[gpi], vpm[gpi], rpm[gpi])

Full example here

Feature Tracking and the first SAR image

Pattern Matching and the second SAR image

sea_ice_drift's People

Contributors

akorosov avatar ashnair1 avatar stefanmuck avatar tdcwilliams avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sea_ice_drift's Issues

Geo-referenced images in Polar Stereographic coordinate

Hi, I have many C-band Sentinel-1 SAR images those are already pre-processed in;

  • 8-bit
  • subset
  • geo-referenced in polar-stereographic coordinate system
    I am aware that I can directly use '.SAFE' file. But due to some reasons, I want to use these already processed file. I am using main functions but always get different errors. I am wondering what do you suggest is the best way.

Proj4 or WKT () is wrong issue

I have a little problem when the program wants to reproject the gcps : it raises me,

**/home/luc/miniconda/lib/python2.7/site-packages/nansat/nsr.pyc in init(self, srs)
72 status = self.ImportFromWkt(str(srs))
73 if status > 0:
---> 74 raise NansatProjectionError('Proj4 or WKT (%s) is wrong' % srs)
75 # TODO: catch long in python 3
76 elif isinstance(srs, int):

NansatProjectionError: Proj4 or WKT () is wrong**

I have this problem if I use my own sentinel 1 image and not the images from the example,
I am a bit puzzled by it
Thank you

pickle error

hi,
I have successfully installed Sea Ice Drift and dependencies using the installation instructions. I also discovered the "Issues" area so you can ignore the personal email I sent you about my installation problems which it appears I solved. I am now trying to run the example. All seems to be fine until I get to sid.get_drift_PM, which gives the following error related to multiprocessing:

upm, vpm, apm, rpm, hpm, lon2pm, lat2pm = sid.get_drift_PM(
... lon1pm, lat1pm,
... lon1ft, lat1ft,
... lon2ft, lat2ft)
Traceback (most recent call last):
File "", line 4, in
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\site-packages\sea_ice_drift\seaicedrift.py", line 88, in get_drift_PM
self.n2, x2, y2, **kwargs)
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\site-packages\sea_ice_drift\pmlib.py", line 424, in pattern_matching
initargs=(c1pm1i[gpi], r1pm1i[gpi], c2fg[gpi], r2fg[gpi], brd2[gpi], img1, img2, img_size, alpha0, kwargs))
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\multiprocessing\context.py", line 119, in Pool
context=self.get_context())
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\multiprocessing\pool.py", line 174, in init
self._repopulate_pool()
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\multiprocessing\pool.py", line 239, in _repopulate_pool
w.start()
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\multiprocessing\popen_spawn_win32.py", line 65, in init
reduction.dump(process_obj, to_child)
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'pattern_matching.._init_pool'
Traceback (most recent call last):
File "", line 1, in
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\ProgramData\Anaconda3\envs\py3drift\lib\multiprocessing\spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

Good version of Sea Ice drift

Hi,
I have some troubles to find the good archive for sea ice drift. I tried to download version 0.6 with pip install but there is no recent changes incorporated in this version (still argument referring to old Nansat for instance),
Is there a good link for getting the latest version ?
Also, I have the feeling that recent changes are made for python 3 (recent printing commit in PMlib), I am running on python 2.7, do you recommend to upgrade to python 3 ?
In the end, I struggle to install Sea ice drift with the last version of nansat because of all dependancies.

Thanks a lot,
Luc

AttributeError: module 'osr' has no attribute 'SRS_WKT_WGS84'

I try to run this simple. py program, but the following problems encountered in the process of operation. (which I use conda install - channel https://conda.anaconda.org/conda-forge nansat to install nansat package)

DLL load failed: 找不到指定的模块。
D:\ANACONDA\envs\py3drift11\lib\site-packages\nansat_init_.py:35: UserWarning: Cannot register C pixel functions!
Either nansat was not installed using setup.py or
pixel functions were not compiled automatically.
For development, use "python setup.py build_ext --inplace"
to compile pixel functions manually into the source tree.

''')
D:\ANACONDA\envs\py3drift11\lib\site-packages\nansat\tools.py:120: MatplotlibDeprecationWarning: Passing raw data via parameters data and lut to register_cmap() is deprecated since 3.3 and will become an error two minor releases later. Instead use: register_cmap(cmap=LinearSegmentedColormap(name, data, lut))
cm.register_cmap(name='obpg', data=obpg, lut=256)
D:\ANACONDA\envs\py3drift11\lib\site-packages\nansat\tools.py:121: MatplotlibDeprecationWarning: Passing raw data via parameters data and lut to register_cmap() is deprecated since 3.3 and will become an error two minor releases later. Instead use: register_cmap(cmap=LinearSegmentedColormap(name, data, lut))
cm.register_cmap(name='ak01', data=ak01, lut=256)
Traceback (most recent call last):
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in call_with_frames_removed
File "D:\ANACONDA\envs\py3drift11\lib\site-packages\nansat_init
.py", line 37, in
from nansat.domain import Domain
File "D:\ANACONDA\envs\py3drift11\lib\site-packages\nansat\domain.py", line 25, in
from nansat.vrt import VRT
File "D:\ANACONDA\envs\py3drift11\lib\site-packages\nansat\vrt.py", line 31, in
from nansat.geolocation import Geolocation
File "D:\ANACONDA\envs\py3drift11\lib\site-packages\nansat\geolocation.py", line 22, in
class Geolocation(object):
File "D:\ANACONDA\envs\py3drift11\lib\site-packages\nansat\geolocation.py", line 61, in Geolocation
def _init_data(self, x_filename, y_filename, x_band=1, y_band=1, srs=NSR().wkt,
File "D:\ANACONDA\envs\py3drift11\lib\site-packages\nansat\nsr.py", line 66, in init
status = self.ImportFromWkt(osr.SRS_WKT_WGS84)
AttributeError: module 'osr' has no attribute 'SRS_WKT_WGS84'

I don't know how to solve this problem. Thank you very much for your help.

Consistent experiments

To make a directory experiments with python scripts to test various parameters. Each experiment tests only one parameter and returns a metrics (density of matching vectors, etc) and makes a map (with density, etc). Name scripts consistently.

Question on SeaiceDrift

When I run the example of sea ice drift retrieval routine, it stops in "sid = SeaIceDrift(f1, f2)" and shows "RuntimeError: Line 147: doesn't have matching ." Has anyone also encountered this problem?

about the data of the input

There are a few questions I would like to ask about the data:

  1. How is the sample data you gave processed, if it is also using Nansat, what APIs are used, and if it is other software, can you explain how to deal with it?
  2. How can I read the .safe file directly?
    thank you!

An Error.

Dear Anton Korosv,
When I run your example.
An error occurs in function sid.get_drift_PM().
It shows "AttributeError: Can't pickle local object 'pattern_matching.._init_pool'".
My system is Win 10 and I use anaconda.

json update needed

The code does not work with the latest Nansat version. The package was installed with conda.

Here the output:
/Data/pit000/ResearchData/IFT/EarthObservation/MOSAIC/SAR/Sentinel-1/S1B_EW_GRDM_1SDH_20191016T064336_20191016T064441_018497_022DA3_6888.zip
/Data/pit000/ResearchData/IFT/EarthObservation/MOSAIC/SAR/Sentinel-1/S1B_EW_GRDM_1SDH_20191017T040823_20191017T040927_018510_022E09_9E76.zip
20191016T064336 20191017T040823
Time difference: 77087.0 seconds
07:39:54|40|nansat|_get_dataset_metadata|GDAL could not open /Data/pit000/ResearchData/IFT/EarthObservation/MOSAIC/SAR/Sentinel-1/S1B_EW_GRDM_1SDH_20191016T064336_20191016T064441_018497_022DA3_6888.zip, trying to read with Nansat mappers...
Traceback (most recent call last):
File "sid_drift.py", line 147, in
sid = SeaIceDrift(f1, f2)
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/site-packages/sea_ice_drift/seaicedrift.py", line 38, in init
self.n1 = get_n(self.filename1, **kwargs)
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/site-packages/sea_ice_drift/lib.py", line 210, in get_n
n = Nansat(filename)
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/site-packages/nansat/nansat.py", line 159, in init
self.vrt = self._get_mapper(mapper, **kwargs)
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/site-packages/nansat/nansat.py", line 1164, in _get_mapper
tmp_vrt = nansatMappers[iMapper](self.filename, gdal_dataset, metadata, **kwargs)
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/site-packages/nansat/mappers/mapper_cmems.py", line 38, in init
gcmd_keywords_mapping = get_gcmd_keywords_mapping()
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/site-packages/nansat/mappers/mapper_cmems.py", line 16, in get_gcmd_keywords_mapping
'instrument': json.dumps(pti.get_gcmd_instrument('computer')),
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/site-packages/pythesint/vocabulary.py", line 45, in find_keyword
for d in self.get_list():
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/site-packages/pythesint/json_vocabulary.py", line 22, in get_list
result = json.load(opened_file)
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/pit000/miniconda3/envs/py3drift/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Make it more generic

Currently the packages is very specific to Sentinel-1, Nansat, HV. However it has high potential for applications in other areas (bands, sensors, EM-range, etc). It is important to make it much more generic to be more widely usable.

Test data missing

ftp.nersc.no/pub/nansat folder is missing, so no test data is available. Not a big deal, but probably it's good idea to fix. Maybe through git-LFS to make it a part of repository?

Installation of Prerequisites

I'm having trouble installing the required prerequisites.

First I tried installing them with a windows machine, which was not possible, because there were dependencies of nansat, that were not available for windows.

Then I setup a VM running Ubuntu 16.04. I installed nansat just the way they explained in their on-site guidance: using miniconda. Everything worked fine this far - everything was setup, except for OpenCV. There begins the trouble. Firstly, since I was using miniconda, I thought why not install OpenCV through conda? Well, that wasn't possible. I got a dependency issue which follows:

Running (this gives opencv version 2.4.6)

conda install opencv

gives

Fetching package metadata .........
Solving package specifications: ....

The following specifications were found to be in conflict:
  - opencv
Use "conda info <package>" to see the dependencies for each package.

and running (this gives opencv version 2.4.9.1)

conda install -c https://conda.binstar.org/menpo opencv

gives

Fetching package metadata ...........
Solving package specifications: ....

The following specifications were found to be in conflict:
  - nansat-gdal -> numpy 1.10.2
  - opencv
Use "conda info <package>" to see the dependencies for each package.

It looks like OpenCV works with other dependencies than nansat.

Which OpenCV version did you use? How did you install it? Did you set it up from source?

Avoid key point generation near land

Keypoints detected near or on land may decrease accuracy of ice drift retrieval.
They should be masked away.

  • and optional watermask retrieval from Nansat
  • run detectAndCompute with mask

sid.get_drift_FT() Error

uft, vft, lon1ft, lat1ft, lon2ft, lat2ft = sid.get_drift_FT() was error
ValueError: does not have metadata time_coverage_start

How can I solve this problem?

Need a help in preparation original Sentinel SAR image for the algorithm

Hello

I installed the library and run the example without a problem, but when I tried the original SAR images I faced with an error.

"ValueError: Cannot find band {'name': 'sigma0_HV'}! band_number is from 1 to 1"

so I checked the example and original files both and I found some differences in metadata:
Original file does not have this line NETCDF_VARNAME=sigma0_HV

One member of the group I am working in it recommended me to use "nansat_geotiffimage". I applied it but I still have the same problem

Test.pdf

thanks in advance for any help you can provide

Best regards,
Hamoun

Vectors direction

Hi,
Thank you for resolving the last issue. Now I tried running sea_ice_drift with the supplied test data but it shows all vectors for FT and PM in one (u) direction

sea_ice_drift_ft_img1
sea_ice_drift_pm_img2

But when i multiplied the vft and vpm by a factor say 10, then it resembles the results provided on website

sea_ice_drift_ft_img3
sea_ice_drift_pm_img4

So is there any factor involves when try to plot the vectors?
Many thanks, Usama

'bool' object has no attribute 'children'

I am trying to open some .zip files dowloaded from the sentinel hub(with Nansat or get_n function). Normally I have no problem to do that,however for some images, it fails :

/home/luc/miniconda/envs/test-environment/lib/python2.7/site-packages/nansat/mappers/mapper_sentinel1_l1.pyc in get_LUT_VRTs(self, XML, vectorListName, LUT_list)
450 LUTs[LUT] = []
451 xLengths = []
--> 452 for vec in vecList.children:
453 xVec = list(map(int, vec['pixel'].split()))
454 xLengths.append(len(xVec))

AttributeError: 'bool' object has no attribute 'children'

Any ideas about what is happening on some files ? It is a bit problematic when I try to perform loop on a directory with a lot of image pairs,

Thanks a lot,

Luc

Pattern matching not working

Hi, I tried running sea_ice_drift with the supplied test data but cannot get it to work. The Feature tracking seems to be fine (the figure looks as in the example at the website), but the pattern matching doesn't yield any results. Could anyone read through the output below (especially the very last line) and hint me in a direction what the trouble is? Very much appreciated, many thanks, Usama.

ORB detector initiated
Key points found: 18801
ORB detector initiated
Key points found: 20191
Domain filter: 18801 -> 16063
Domain filter: 20191 -> 16752
Keypoints matched 3.16630792618
Ratio test 0.700000 found 1559 keypoints
MaxDrift filter: 1559 -> 1559
LSTSQ filter: 1559 -> 1559
GLib-GIO-Message: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications.
Traceback (most recent call last):
File "seaice.py", line 27, in
upm, vpm, rpm, apm, lon2pm, lat2pm = sid.get_drift_PM(lon1pm, lat1pm, lon1ft, lat1ft, lon2ft, lat2ft)
File "/home/ufa11/miniconda/lib/python2.7/site-packages/sea_ice_drift/seaicedrift.py", line 89, in get_drift_PM
self.n2, x2, y2, **kwargs)
File "/home/ufa11/miniconda/lib/python2.7/site-packages/sea_ice_drift/pmlib.py", line 315, in pattern_matching
results = p.map(use_mcc_mp, range(len(gpi[gpi])))
File "/home/ufa11/miniconda/lib/python2.7/multiprocessing/pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "/home/ufa11/miniconda/lib/python2.7/multiprocessing/pool.py", line 567, in get
raise self._value
TypeError: slice indices must be integers or None or have an index method

Make it modular

There is just one function in the package. It makes it very difficult to 1. test; 2. update; 3; generalize. Should be split in several shorter functions.

Make it more professional

It is difficult to install - requires to work inside the directory or add it to PYTHONPATH. It is almost impossible to test. Requires to run full processing and inspect visually.
Package structure should be implemented and unit tests are required.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.