kvos/coastsat

Global shoreline mapping tool from satellite imagery

Dr Kilian Vos

I'm a Remote Sensing Scientist at the Water division of the NSW Department of Planning and Environment. I have a PhD from the Water Research Laboratory, University of New South Wales. My interest is in satellite remote sensing for Earth sciences, development of webGIS applications and machine learning classification/regression problems. I mainly code in Python and use big data cloud computing platforms (like Google Earth Engine) to extract useful data from publicly available satellite imagery.

I enjoy developing open-source remote sensing tools and creating web applications to visualise geospatial data. I also have experience as a data scientist solving regression/classification machine learning problems and decision making problems with reinforcement learning (Pytorch/Tensorflow).

More info about my research and projects can be found on my personal website.

---

Open-source projects

• Coastsat: global shoreline mapping toolbox from publicly available satellite imagery with Google Earth Engine
• Coastsat webGIS portal: interactive webGIS portal to explore large-scale coastal change data
• CoastSat.slope: beach slope estimation from satellite-derived shorelines and tide models
• SDS_Benchmark: benchmarking platform for satellite-derived shoreline algorithms

coastsat's Issues

Calculation of features not robust towards nan values

In calculate_feature a feature vector is generated for each pixel not obscured by clouds. The feature vector consists of five bands and a corresponding normalized difference index. This yields in total ten bands. For each of these bands, the standard deviation is calculated, and this is where things go wrong. The function image_std, used to calculated the standard deviation, uses uniform_filter from Scipy to do so. I have expierenced problems in accordance with this issue: scipy/scipy#7818. I.e., whenever a pixel is assigned nan, all values proceding this pixel values in the convolution will have value nan.

The issue can be very easily replicated:

im_ms, georef, cloud_mask, im_extra, im_QA, im_nodata = SDS_preprocess.preprocess_single(fn,
                                                                                             satname,
                                                                                             cloud_mask_problem)

# will results in channel 9- 19 of im_features equal nan
im_ms[0,0,:] = np.nan
im_features = SDS_shoreline.calculate_features(im_ms, cloud_mask, ~im_nodata)

# check if all elements are nan
im_features[:,:,10:].flatten().all() == True

The problem can be mitigated easily by using a convolution library that handles nan values. Astropy (https://docs.astropy.org/en/stable/convolution/) is one such library

metadata = SDS_download.retrieve_images(inputs)

TimeoutError Traceback (most recent call last)
in
----> 1 metadata = SDS_download.retrieve_images(inputs)

~\Desktop\P work\CoastSat-master\coastsat\SDS_download.py in retrieve_images(inputs)
76
77 # initialise connection with GEE server
---> 78 ee.Initialize()
79
80 # check image availability

~\Anaconda3\envs\coastsat\lib\site-packages\ee_init_.py in Initialize(credentials, opt_url, use_cloud_api, cloud_api_key)
103 cloud_api_key=cloud_api_key)
104 # Initialize the dynamically loaded functions on the objects that want them.
--> 105 ApiFunction.initialize()
106 Element.initialize()
107 Image.initialize()

~\Anaconda3\envs\coastsat\lib\site-packages\ee\apifunction.py in initialize(cls)
150 """Initializes the list of signatures from the Earth Engine front-end."""
151 if not cls._api:
--> 152 signatures = data.getAlgorithms()
153 api = {}
154 for name, sig in signatures.items():

~\Anaconda3\envs\coastsat\lib\site-packages\ee\data.py in getAlgorithms()
648 is not specified.
649 """
--> 650 return send_('/algorithms', {}, 'GET')
651
652

~\Anaconda3\envs\coastsat\lib\site-packages\ee\data.py in send_(path, params, opt_method, opt_raw)
1159 return response, content
1160
-> 1161 response, content = send_with_backoff()
1162
1163 # Call the profile hook if present. Note that this is done before we handle

~\Anaconda3\envs\coastsat\lib\site-packages\ee\data.py in send_with_backoff(retries)
1150 try:
1151 response, content = http.request(
-> 1152 url, method=opt_method, body=payload, headers=headers)
1153 if response.status == 429:
1154 if retries < MAX_RETRIES:

~\Anaconda3\envs\coastsat\lib\site-packages\oauth2client\transport.py in new_request(uri, method, body, headers, redirections, connection_type)
157 _LOGGER.info('Attempting refresh to obtain '
158 'initial access_token')
--> 159 credentials._refresh(orig_request_method)
160
161 # Clone and modify the request headers to add the appropriate

~\Anaconda3\envs\coastsat\lib\site-packages\oauth2client\client.py in _refresh(self, http)
747 """
748 if not self.store:
--> 749 self._do_refresh_request(http)
750 else:
751 self.store.acquire_lock()

~\Anaconda3\envs\coastsat\lib\site-packages\oauth2client\client.py in _do_refresh_request(self, http)
778 resp, content = transport.request(
779 http, self.token_uri, method='POST',
--> 780 body=body, headers=headers)
781 content = _helpers._from_bytes(content)
782 if resp.status == http_client.OK:

~\Anaconda3\envs\coastsat\lib\site-packages\oauth2client\transport.py in request(http, uri, method, body, headers, redirections, connection_type)
280 return http_callable(uri, method=method, body=body, headers=headers,
281 redirections=redirections,
--> 282 connection_type=connection_type)
283
284

~\Anaconda3\envs\coastsat\lib\site-packages\httplib2_init_.py in request(self, uri, method, body, headers, redirections, connection_type)
1989 headers,
1990 redirections,
-> 1991 cachekey,
1992 )
1993 except Exception as e:

~\Anaconda3\envs\coastsat\lib\site-packages\httplib2_init_.py in _request(self, conn, host, absolute_uri, request_uri, method, body, headers, redirections, cachekey)
1649
1650 (response, content) = self._conn_request(
-> 1651 conn, request_uri, method, body, headers
1652 )
1653

~\Anaconda3\envs\coastsat\lib\site-packages\httplib2_init_.py in _conn_request(self, conn, request_uri, method, body, headers)
1555 try:
1556 if conn.sock is None:
-> 1557 conn.connect()
1558 conn.request(method, request_uri, body, headers)
1559 except socket.timeout:

~\Anaconda3\envs\coastsat\lib\site-packages\httplib2_init_.py in connect(self)
1389 break
1390 if not self.sock:
-> 1391 raise socket_err
1392
1393

~\Anaconda3\envs\coastsat\lib\site-packages\httplib2_init_.py in connect(self)
1322 if has_timeout(self.timeout):
1323 sock.settimeout(self.timeout)
-> 1324 sock.connect((self.host, self.port))
1325
1326 self.sock = self._context.wrap_socket(sock, server_hostname=self.host)

TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Duplicate of #18

Hi,
When trying to type the following "conda env create -f environment.yml -n coastsat" in the same folder as "environment.yml" exists which is in master folder
.........CoastSat-master>conda env create -f environment.yml -n coastsat
a message appears as
CondaValueError: prefix already exists: C:\Users........\Anaconda3\envs\coastsat

Any idea why this happens?

Linking CoastSat to GEE server

Hi Kilian, Shanghai from China here.

Thank you very much and the free tools provided by your team.

I am trying to run your script but after I copy the authorization code into the Anaconda terminal , it says "NameError:name "urllib2"is not defined"and"TimeoutError:[WinError 10060] The connection attempt failed because the connected party did not respond correctly after a certain period of time or the connected host did not respond.".

For programming, I am just a rookie just getting started.I don't know if the reason is because I am online in China...Can you help me ?

Sincere regards!!

Some of the conda-forge files not found.

Hello,
I am trying to install the CoastSat package on a Macbook Pro Retina with OSX 10.13.6 following the description but it fails while running the command:
conda install --name coastsat --file requirements_osx64.txt

some of the file listed in the requirement file seems to be missing.
any hint?
thanks
Marco Olivieri

CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/osx-64/proj4-5.2.0-h470a237_1.tar.bz2
Elapsed: 00:00.222118
CF-RAY: 4d7370aaea7ccd2e-FCO

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/osx-64/hdf5-1.10.4-nompi_h5598ddc_1003.tar.bz2
Elapsed: 00:00.267745
CF-RAY: 4d73714a19cbcd26-FCO

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/osx-64/postgresql-10.5-ha408888_1.tar.bz2
Elapsed: 00:00.436672
CF-RAY: 4d73729d1c01cd36-FCO

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/osx-64/libgdal-2.3.2-h42efa9e_1.tar.bz2
Elapsed: 00:00.428753
CF-RAY: 4d737429eca7cd0a-FCO

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/osx-64/gdal-2.3.2-py36hfc77a4a_1.tar.bz2
Elapsed: 00:00.217833
CF-RAY: 4d73748ecf86cd1e-FCO

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

Duplicate of #18

The following is the error message:

(base) PS C:\Users\TANIN\Anaconda3> conda activate coastsat

>>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

An unexpected error has occurred. Conda has prepared the above report.

GeoJSON output file open in a GIS software issue

Offline-use after download

Hi Kilian,

Awesome tool! It worked in a jiffy and tried it on my holiday-beach: smooth and easy.
Now. I wanted to work off-line after I download the data but I see you initialise the Google Earth Engine toolbox also if you just want to load the metadata. I think this is not necessary. There is just a tiny change needed for this and I don't think it is worth to make a pull-request etc.

Can you move;
ee.Initialize()

within
def retrieve_images(inputs):

Then you only initialise the server when it is really needed. For the users, it then becomes
from coastsat.SDS_download import get_metadata to work offline... (small detail)

Cheers,

Erwin

How to pull the latest updates

Hi,
I was double checking things and have been also looking into some updates here.
I have seen the following update about 20 days ago:
update SDS_prepocess.py

Could you please advise if I need to update something in the master folder because of this?

Best regards

Installation/Initialization errors on MacOS

Hey Killian,

I've just begun the process of using CoastSat for my local east coast Florida site. For reference, I'm doing this on MacOS 10.14. In following your nicely laid out installation/initialization steps in the readme, I ran into trouble when your line to create a new environment (i.e. "conda env create -f environment.yml -n coastsat") threw me an error. To fix this, and to make it through the step of linking my GEE credentials to the Python API, I had to do the following:

1. Create the new environment within the folder containing CoastSat using conda create --name coastsat
2. Install python and pip within the new environment using conda install pip
3. Install the GEE python library using pip install earthengine-api
4. Do earthengine-authenticate and follow the steps in the readme to authenticate

I'm not sure if this is a general MacOS issue, or one specific to my system, but I figured I post in case anyone else runs into the same issue.

-Matt Conlin

conda activate versus source activate

In my machine one has to use source activate, not conda activate

mosa@parti-mosa:~/CoastSat$ conda activate coastsat

CommandNotFoundError: 'activate is not a conda command.
Did you mean 'source activate'?

mosa@parti-mosa:~/CoastSat$ source activate coastsat

(coastsat) mosa@parti-mosa:~/CoastSat$

Choosing only one date of each year

Sir,
I need to analyse the shoreline change of a particular season and want to take one data in every one/ five yera. Can you please tell, how to define ?

regards,
Subhasis

Linking CoastSat to GEE server

Hi,
I am trying to run your script but I seem to be stuck on earthengine authenticate , it says it is not recognised as an internal or external command. I m not an experienced programmer I am just ok at following instructions when they are very detailled. Any idea why I cannot link ee to anaconda please?
Thank you
Jerome

Coastline extraction from Planet Images

Hello! Congratulations for Coast Sat, I really like it. I have a question for you. Is it possible somehow to extract coastlines from Planet Imagery? I already have some images saved in my laptop. What changes should I try in the code? I tried to run it as it is and I have an error message 'no L7 images'.

Thank you

Download Tier 2 images for remote locations

Hi Kilian,

I'm trying to apply the CoastSat for my study area (Fernando de Noronha Archipelago, PE, Brazil), but no image was found/downloaded. I've already tested some different periods, satellites and polygons, but without results.

However, I found many Landsat images for this area available by the USGS. On the Earth Explorer page, I did the search using the same time period and only looking for Landsat Collection 1 Level-1 (data sets). It found many images with their names end in T2. Then, I don't know if they are Tier 1 or Tier 2 images. And I don't know why they aren't find by the CoastSat (from the GEE). Do you know what could be happening or how can I solve this problem?

I'm running the script in jupyter notebook (Windows system). My polygon is located on 25S UTM zone and it's around 5 km2. I'm attaching the screen shots that show the settings and the error message.

Thank you.

Images not downloading and tidal correction

Hello Killan,
I started using coastsat as a part of my dissertation. I have successfully completed the example on Narra beach. After changing the coordinates and site name of the example, the code was executed, but no images were downloaded. I'm preparing script in spyder and running the same in anaconda prompt.Here I'm attaching the screen shots please check the same and give a suitable reply.

Duplicate of #18

After that I tried to follow the following 3 steps:

conda create --name coastsat
conda activate coastsat
conda install <name_of_the_package>

BUT in step 3 while dealing with """earthengine-api""", I get an ERROR message.
please see below:

(coastsat) C:\Users.........>conda install earthengine-api
Collecting package metadata: done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

earthengine-api
Current channels:

https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/win-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org
and use the search bar at the top of the page.

Any ideas? inputs?

Not able to install coastsat

when i tried to create environment, it shows the following error
SpecNotFound: Invalid name, try the format: user/package

Shoreline extraction issues

@kvos,

I am (sometimes) getting a strange error message when I call "SDS_shoreline.extract_shorelines(metadata,settings)", where the error message is "ValueError: LineStrings must have at least 2 coordinate tuples" (screenshot attached). Sometimes the function works without error, and sometimes I get this message. I cannot seem to detect a pattern of when it errors (i.e. only for certain satellites or dates, etc.), other than that it only seems to happen when I apply the function to a relatively long interval (e.g. 15 yr). For example, I can call the function, for the same satellites and end date as established in "metadata", for 3 months and it will work fine, but will error if I try it for 10 yr. As the screenshot shows, the function gets all the way to 100% for each satellite before it errors out. For reference, I'm running MacOS 10.14 with Python 3.7. Any ideas?

Improving resolution of satellite images

Dear Kilian:

We are trying to improve the resolution of the satellite images obtained for Marawila region. However, it seems we are doing something wrong as the images obtained are rather blurred.

The input parameter values we used are:

cloud_thresh = 0.5
min_beach_area'= 1000
buffer_size= 100
min_length_sl=200
sand_color=dark

The KML file used is here:
https://drive.google.com/file/d/1CW65qN-BbG-iapZgQrFFTpj4zFTCh30H/view?usp=sharing](url)

I am sharing herewith the images we obtained

We are certainly doing something wrong as our resolution is significantly less than the resolution that you reproduced the same region with. Please could you let us know the parameters you would change to improve this quality?
Thank you very much.

Issue setting up environment

Hi Kvos,
I've just installed the code, and was able to run example_jupyter.ipynb. However, I am getting an error message below while running example.py. Do you know why?

I am using Python 3.7 Windows... thank you

runfile('C:/CoastSat-master/example.py', wdir='C:/CoastSat-master')
Reloaded modules: coastsat
File "coastsat\SDS_download.py", line 207
print('\r%d%%' % (int(((i+1)/n_img)*100)), end='')
^
SyntaxError: invalid syntax

Shoreline detection of Marawila beach

Dear all,

I started using CoastSat recently and found it to be a very useful tool. I recommended the software to one undergraduate project where the student will be using the tool

to assess the long-term (20+ years) shoreline change at a beach in Sri Lanka- Marawila beach (https://www.google.com/maps/search/marawila+beach/@7.4124624,79.8009101,14z/data=!3m1!4b1).

The student has successfully completed the example on Narra Beach given in GitHub. After changing only the co-ordinates of the example (so that it would match Marawila beach), the code was executed. However, we see that the quality of the images obtained is not clear - they appear pixelated (see attached image) which would clearly affect the shoreline detection stage as well. Clearly, there needs to be some adjustment to shoreline detection parameters in the code, but we are not sure how to fine-tune these parameters to get a reasonably accurate detection of the shoreline for Marawila. Could anybody give some suggestions, based on your experience, to improve this?

Any help is highly appreciated.

Kind regards

Issue linking to GEE server

Hi!

When running the example at the third part or the code to retrieve satellite images from GEE. "metadata = SDS_download.retrieve_images(inputs)" it return an error.

The error returned is:

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

I have already followed the steps suggested in erros #39, #41 and #40. And still I'm not correctly linked to the GEE server.

I requested access directly to Earth Engine yesterday and promptly received an email conceding access to the following products:

• Earth Engine Code Editor - the primary Earth Engine development environment.
• Earth Engine API - including our Python library.
• Earth Engine Explorer - a graphical user interface.
  (I can already open the API and all, so I presume that this is fine)

I'm using Windows.

Thanks!

Linking to GEE server [HTTP error 403]

Hi Kilian,

I have set up this CoastSAT after our previous discussions. I have been trying to use Jupitor and look at the example to be able to fully capture what is going on. I received the following messages. part a and part b. I was wondering if somebody can advise what is going on?
Is this on the right track or?

Thanks for your cooperation and interaction in advance.
Best wishes

How to retrieve metadata without downloading images?

Hello Killan,

When retrieving metadata for satellite images, are there any options to download the metadata ( .txt) files only? Sometimes it is time-consuming to download satellite image (".tif") files.

Make coastsat into a package

In order to use coastsat, you have to set your pythonpath/start your environment with coastsat as root dir.

It would be really nice if CoastSat existed as an installable package.

If you are interested, I could create a PR

Possible to expand on bands extraction?

Is your feature request related to a problem? Please describe.
No, not related to a problem! Just would like to learn more about the background processes :)

Describe the solution you'd like
Hi @kvos , by now (and my emails) you are well aware that I am a big fan of your academic pubs on the subject-content (methodology) for your coastsat module. I was wondering, when reading both the larger pub (methodology deep) and shorter pub (explaining coastsat high-level process) if you would be able to append the processing of the imagery between the 'download' and before 'shoreline extraction'.

For example, during image processing, are the images downloaded inidividual bands, or, are those extracted separately during the image?

I will try to use this example 👍 :

From SDS_preprocess.preprocess_single_image(), there is the following line:

# read all bands
        data = gdal.Open(fn, gdal.GA_ReadOnly)
        georef = np.array(data.GetGeoTransform())
        bands = [data.GetRasterBand(k + 1).ReadAsArray() for k in range(data.RasterCount)]
        im_ms = np.stack(bands, 2)

However, in the source docs for arguments, SDS_classify.label_images(), you mention that fn = ( a list of filenames, one filename for each band"

# get individual RGB image
            im_RGB = SDS_preprocess.rescale_image_intensity(im_ms[:,:,[2,1,0]], cloud_mask, 99.9)
            im_NDVI = SDS_tools.nd_index(im_ms[:,:,3], im_ms[:,:,2], cloud_mask)
            im_NDWI = SDS_tools.nd_index(im_ms[:,:,3], im_ms[:,:,1], cloud_mask)

Additional context
Given the data above, I am wondering how the processing is actually carried out. For example, I have the following understanding (in my small head):

Download Images (inputs, etc) [Let's pretend there is only one capture for the inputs, a L7 (B,G,R,NIR,SWIR1)] ---> Image Stored in (./data) ===skip ahead in process=== ---> Use pre-processing, gather the bands and stack RGB together, also extract im_NDVI & im_NDWI.

Obviously, from here - it differs based on creating new location (classifier training) or simply receiving the input. However, I would love to understand if we are pulling in a single image or if that image is being downloaded by band (i.e. are we getting ~7 copies of the image at different bands or a single raster)?

Thanks @kvos , stay awesome

Linking CoastSat to GEE server [HTTP error 404]

When running the example at the third part or the code to retrieve satellite images from GEE. "metadata = SDS_download.retrieve_images(inputs)" it return an error.

The error code returned is:

~\AppData\Local\Continuum\anaconda3\envs\coastsat\lib\site-packages\ee\data.py in send_(path, params, opt_method, opt_raw)
1194 # error above rather than this generic one.
1195 raise ee_exception.EEException(
-> 1196 'Server returned HTTP code: %d' % response.status)
1197
1198 # Now known not to be an error response...

EEException: Server returned HTTP code: 404

Any help will be greatly appreciated.
Thank you

Offshore bias in S2 images

Hi Kilian, as mentioned above (# 50) I have a problem with S2 images, the shoreline are overly offshore.
Whith the same S2 images in QGIS I calculated NDWI ((B3-B8)/(B3+B8)), whith the threshold equal to 0, and I got better results. In this way I can use two images with the same spatial resolution, rather than proceeding with interpolations (as in the MNDWI calculation).
The difference between the two shorelines is 17-22 meters.
Below you can see two examples
What do you think about it?
Greetings from Italy

Offshore bias in shoreline detection

Hi kvos, Antonio from Italy here.
On the same stretch of beach I used all the available satellite images (L5, L7, L8, S2) in the same periods (15/07/2018-15/08/2018). The results are very different and none is optimal. What parameters should I work on?
Kind Regards

Duplicate of #18

After typing "conda env create -f environment.yml -n coastsat" in the folder same as CoastSat-master the following ERROR appears:
CondaValueError: prefix already exists: C:\Users........\Anaconda3\envs\coastsat

The purpose of this post was to figure out "why" such issues might happen in the first place so to avoid repeat of such issues in the future for better workflow.

Error in retrieving satellite images

Dear all,

I am using CoastSat to obtain shoreline images of Marawila beach (Sri Lanka) before the year 2016. I used a .KML file generated from My Maps, using this, I input the coordinates of the coastal region to CoastSat. The time period was also selected as 2015/02/01 to 2015/12/01.

At this point, there was an error called "Unbound Local Error" - please see the screenshot below:

Please note that this error did not occur for shoreline images obtained after the year 2016. So this seemed to be quite strange. I would highly appreciate it if you could provide a suggestion to help to resolve this problem.

Thanks a lot.

Kind regards
Eranda

Black sand beaches

Hi Kilian, Francisco from Chile here.

first to all, thank you very much for the tool that you and your teammates developed.

I have a problem with the use of the tool... In Pichilemu we have black sand, I saw in the when you analyzed the Pichilemu beach that the model can recognize the black sand, but when I did the same the model couldn't recognize it. Do I need to edit some parameters?

Kind Regards !!

Broken GDAL package

Hi Kilian,

Im so sorry, i know that Issue #15 is closed but im facing the same problem with *TypeError: in method 'CoordinateTransformation_TransformPoints', argument 1 of type 'OSRCoordinateTransformationShadow '
I runned the example.py before and it was all good. But now t doesnt work i dont know why. I cleaned it all and re-installed the CoastSat environment several times, and still not working. :(

Any clues?
TypeError Traceback (most recent call last)
in
1 get_ipython().run_line_magic('matplotlib', 'qt')
----> 2 output = SDS_shoreline.extract_shorelines(metadata, settings)

~\Anaconda3\Scripts\Coast\coastsat\SDS_shoreline.py in extract_shorelines(metadata, settings)
773
774 # process the water contours into a shoreline
--> 775 shoreline = process_shoreline(contours_mwi, cloud_mask, georef, image_epsg, settings)
776
777 # visualise the mapped shorelines, there are two options:

~\Anaconda3\Scripts\Coast\coastsat\SDS_shoreline.py in process_shoreline(contours, cloud_mask, georef, image_epsg, settings)
413 contours_world = SDS_tools.convert_pix2world(contours, georef)
414 # convert world coordinates to desired spatial reference system
--> 415 contours_epsg = SDS_tools.convert_epsg(contours_world, image_epsg, settings['output_epsg'])
416 # remove contours that have a perimeter < min_length_sl (provided in settings dict)
417 # this enables to remove the very small contours that do not correspond to the shoreline

~\Anaconda3\Scripts\Coast\coastsat\SDS_tools.py in convert_epsg(points, epsg_in, epsg_out)
140 # iterate over the list
141 for i, arr in enumerate(points):
--> 142 points_converted.append(np.array(coordTransform.TransformPoints(arr)))
143 elif type(points) is np.ndarray:
144 points_converted = np.array(coordTransform.TransformPoints(points))

~\Anaconda3\lib\site-packages\osgeo\osr.py in TransformPoints(self, *args)
1286 def TransformPoints(self, *args):
1287 """TransformPoints(CoordinateTransformation self, int nCount)"""
-> 1288 return _osr.CoordinateTransformation_TransformPoints(self, *args)
1289
1290 CoordinateTransformation_swigregister = _osr.CoordinateTransformation_swigregister

TypeError: in method 'CoordinateTransformation_TransformPoints', argument 1 of type 'OSRCoordinateTransformationShadow *'

Thank you

Juliana

Error when runnning batch shoreline detection

Hi,
This has been a frequent problem of mine. Whenever i press the right arrow which says keep
i get this error
. The mapping is stuck at 14%

I run everything normally except i skip the optional code where it asks me to save it as a jpg because it errors.

I expected it to make a graph after i press keep.

Im running windows 10 pro, and using miniconda3.

CoastSat not working due to Pillow 7.0

Hi,

I have encountered the following issues as in the picture. It seems that new version of CoastSat is not compatibile with Pillow environment, so there a need for going back to lower version which is 6.2.
With this change the CoastSat is working as before.

Hope it is helpful,

Marcin

Error while setting up in windows system

Hi everybody,

I am using a windows system, I spent a couple of hours to set up the system on the computer but in the step 1.1 I come across some Error Messages. I installed Anaconda on my PC. Then I created a new environment named coastsat with all the required packages:
conda env create -f environment.yml -n coastsat
Then I wanted to activate the new environment as below according to the website

conda activate coastsat
At this stage the following message has appeared.

Invoke-Expression : Cannot bind argument to parameter 'Command' because it is an empty string.

At C:\Users\TANIN\Anaconda3\shell\condabin\Conda.psm1:70 char:36

•     Invoke-Expression -Command $activateCommand;
  

•                                ~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidData: (:) [Invoke-Expression], ParameterBindingValidationException

  • FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAllowed,Microsoft.PowerShell.Commands.Invo

  keExpressionCommand

---

I would highly appreciate if you can kindly advise how I can fix this issue.

coastline as well as the inner side islands are also extracted

Sir,

Please suggest to extract only the coastline. coastline as well as the inner side islands are also extracted.

regards.
subhasis

cannot calculate cross distance

When running the code everything works except for the shoreline change analysis. I can select shorelines, view all of the created shorelines, and draw transects for them, but the cross distance is given as nan and no shoreline change graph is created.

My points are:
[-75.537, 35.776], [-75.52, 35.78], [-75.439, 35.591],[-75.508, 35.588],[-75.537, 35.776]

output epsg: 3358

Any ideas?

potential cloud mask issue?

@kvos -

See attached for quick test case from south coast of WA (Parry's Beach, near Denmark). Seems like the beach pixels in the majority of the images are being masked out (e.g., 2015-02-23_L8 and 2015-10-21_L8), however, some images are still processed normally (e.g. 2016-06-01_L8).

I have tried this for a few other locations along the SW coast and seem to get the same issue, but have yet to see this for other test locations (i.e. Mandurah). Maybe an issue with how the cloud mask is applied or calculated ?

SouthCoast_WA_examples.zip

Using non-satellite imagery

Hi Kvos,

We regularly receive high resolution geo-referenced aerial survey information and overtime we have built up quite a database of information.

Is it possible to include non-satellite project specific imagery in the shoreline analysis process?

Kudos on such a great project. Digitising shorelines and accessing decent satellite imagery use to take up quite a bit of my time so its exciting to see such great developments being made.

Question regarding georeferencing

Hi all, I got a question regarding georeferencing in CoastSat.
in Vos et al (2019) it says the following:

"It should be noted that the images are orthorectified by the data
provider and no further geometric correction (i.e. co-registration) is
applied in CoastSat. The georeferencing accuracy of each image is stored
in the output file containing the mapped shorelines. "

Now, when I download some L5 images I can see the following in the .txt file, for example:
filename 2003-09-25-14-48-35_L5_Location1.tif
acc_georef 4.633
epsg 32618

Now the question:
Does this mean that the georeferencing accuracy is 4.63 meters in this image? or is this 4.63 some factor?
I saw in some part of the code, file SDS_download.py, the following
"acc_georef.append(12) # default value of accuracy (RMSE = 12m)", so I'm guessing that the 4.63 is also in meters.

Thanks in advance for the help.

Daniel

Batch shoreline detection error

Hi kvos,

When I finished "Batch shoreline detection" this step, it shows something wrong and reminded me" linestrings must have at least 2 coordinate tuples"

and when i run " plot the time-series of shoreline change along each transect" this stage, there should be a line chart representing the trend, but it can't display, only a blank is displayed.

i m very grateful if you can point out my problem.

Thank you very much

best,

yuan

Classifier issues

I have ran into problems of accurate shoreline detection and I thought I will try to retrain the classifier. I did (to my knowledge) everything according to the guidelines...I have added the SDS_classify.py file, installed the astropy, etc.

After completing all of the prior steps (initial settings, download, etc. (from the example)) I have tried to run the command: SDS_classify.label_images(metadata,settings), however I run into an error:

KeyError Traceback (most recent call last)
in
----> 1 metadata = SDS_classify.label_images(metadata,settings)

~/CoastSat-master/coastsat/SDS_classify.py in label_images(metadata, settings)
93
94 """
---> 95 filepath_train = settings['filepath_train']
96 # initialize figure
97 fig,ax = plt.subplots(1,1,figsize=[17,10], tight_layout=True,sharex=True,

KeyError: 'filepath_train'

Currently working on Linux Mint OS. I am a complete beginner, so any help would be marvelous.

Thanks and have a great day!

Time-series of shoreline change in .csv file

Hello again,

so far the software works very well and I am glad with the results it provides! However, the very last step when you plot the time series of shoreline change could be improved I believe.

The resulting time series are confusing and there is nothing much you can do. With the provided figure it is only possible to alter the very first transect shoreline time-series visualization. No matter if I choose a second transect, the changes only apply to the first one... for example if I want to adjust the bottom and top boundaries (yes, I can do that before I run the command, however shoreline position on a first transect and the last one, e.g. 200 meters away, is different).

It would be great if the time series data could be extracted to an excel compatible file for example, where you could fit a trend, better compare with in-situ measurements, etc.
Even a .txt file would do good with information on the date and shoreline position in each of the transects.

Let me know if that would be possible or perhaps there are alternative solutions? Everything else is fine!:)

All the best,
Lukas

Use keyboard to keep/skip images

Hi,

After downloading the master version for Coastsat on my Mac, I appear to be having issues with running the SDS_shoreline.extract_shorelines routine in example workbook.

When I run the routine a new window with three figures opens up. But nothing happens when I click on either keep or skip. In the jupiter notebook it says the shoreline analysis has reached 14%. Im also prompted with the following message any time I try to click the skip or keep buttons in the picture: QTimer::singleShot: Timers cannot have negative timeouts. The window will close if I press escape though.

See screenshots below.

I have also tried this on my Windows computer and get stuck on the same bit which makes me think that there might be an issue with the SDS_shoreline.extract_shorelines routine.

Would be grateful for any ideas.

No images are downloaded

I'm trying to use my study site in the GEE and I'm using this code

polygon = [[[29.242207, -83.090314],
[29.229097, -83.092357],
[29.229088, -83.068519],
[29.245553, -83.068872]]]

dates = ['2015-01-01', '2017-012-01']

sat_list = ['L5']

sitename = 'LCR'

filepath = os.path.join(os.getcwd(), 'data')

inputs = {'polygon': polygon, 'dates': dates, 'sat_list': sat_list, 'sitename': sitename, 'filepath':filepath}

I was getting an error 404, but I enabled the widgets and that went away. It just keeps loading without downloading any images. I can send a screenshot too if that helps. Can you check if that range works for you?

Duplicate of #28

Hello Hi Kilian,,
Does anybody know why the following error message might appear
when running "metadata = SDS_download.retrieve_images(inputs)"

ValueError: zero-size array to reduction operation maximum which has no identity

Cannot locate module 'coastsat'

Followed the README:
created new env and activated 'coastsat' via .yml
tried in both Jupyter and Spyder, continue to get error when attempting to import the required scripts from 'coastsat'

from coastsat import SDS_download, SDS_preprocess, SDS_shoreline, SDS_tools, SDS_transects

"No module named coastsat"

When running via jupyter:

EEException                               Traceback (most recent call last)
<ipython-input-3-81739100d1f3> in <module>
----> 1 metadata = SDS_download.retrieve_images(inputs)

~/Downloads/CoastSat-master/coastsat/SDS_download.py in retrieve_images(inputs)
     98 
     99     # initialise connection with GEE server
--> 100     ee.Initialize()
    101 
    102     # read inputs dictionnary

~/anaconda3/envs/coastsat/lib/python3.7/site-packages/ee/__init__.py in Initialize(credentials, opt_url, use_cloud_api, cloud_api_key)
    103       cloud_api_key=cloud_api_key)
    104   # Initialize the dynamically loaded functions on the objects that want them.
--> 105   ApiFunction.initialize()
    106   Element.initialize()
    107   Image.initialize()

~/anaconda3/envs/coastsat/lib/python3.7/site-packages/ee/apifunction.py in initialize(cls)
    150     """Initializes the list of signatures from the Earth Engine front-end."""
    151     if not cls._api:
--> 152       signatures = data.getAlgorithms()
    153       api = {}
    154       for name, sig in signatures.items():

~/anaconda3/envs/coastsat/lib/python3.7/site-packages/ee/data.py in getAlgorithms()
    648                 is not specified.
    649   """
--> 650   return send_('/algorithms', {}, 'GET')
    651 
    652 

~/anaconda3/envs/coastsat/lib/python3.7/site-packages/ee/data.py in send_(path, params, opt_method, opt_raw)
   1194     # error above rather than this generic one.
   1195     raise ee_exception.EEException(
-> 1196         'Server returned HTTP code: %d' % response.status)
   1197 
   1198   # Now known not to be an error response...

EEException: Server returned HTTP code: 404

earth engine not registered/initializing

Hey Killian (or other moderators).
stoked on the tool, but having trouble running the example.py and I think it's related to the google earth engine initialization. Following Issue #28 i ran from the command line and re-activated the google earth-engine, and still ended up with an initialization error.

I think it's primarily a earth engine trouble, so i posted on stack overflow. Maybe you have some insight, or it might be helpful for others to see

Stack question here:
https://stackoverflow.com/questions/58053386/trouble-initalizing-google-earth-engine