brazil-data-cube/cube-builder-aws

Scripts to generate Data Cubes on AWS environment

cube-builder-aws's Issues

update list cube infos endpoint with new bdc-catalog model

endpoints:

• cube status
• cube infos
• list cubes

generate a new cube based on an existing one

execute only blend and publish methods

Create a route to retrieve all merge files and collection scenes from dynamoDB

Add list cube route

save errors in dynamoDB

capture exceptions and save in dynamo DB

Update or Reprocess cube

add force param in query (start cube)

Create new band to store Total Observation Number

Change CNC band name

We should consider to change CNC to something like Count Clear Pixel, etc.

Update API SPEC

save application ID using to generate the cube

compute checksum of images

Release 0.8.0

Tasks:

• Inform the right version in: cube-builder-aws/cube_builder_aws/version.py, spec/package*, cube-builder-aws/package*
• Update spec if necessary
• Create a branch named: b-0.8.
• From the branch b-0.8, create a tag named: v-0.8.0.
• Update documentation how-to
• In the master branch increase version in cube-builder-aws/cube_builder_aws/version.py to 1.0.0
• Notify users that a new version is released.

Prevent the creation of invalid data cube name

Currently, the body to save a data cube definition is the following:

{
    "datacube": "LC8_30_16D",
    "grs": "BDC_GRID",
    "resolution": 30,
    "temporal_schema": "A16day",
    "bands_quicklook": ["swir2", "nir", "red"],
    "bands": ["coastal","blue","green","red","nir","bnir","swir1","swir2","quality"],
    "description": "Landsat monthly",
    "license": "MIT"
}

However, it may result in invalid data cube name. The user should only send LC8 instead LC8_30_16D. It may result in creation of LC8_30_16D_MED with temporal schema Monthly.

In order to do that, you must add a field composite_function to allow only MED, STK or IDENTITY.

Generate cloud quality flag (STK only)

bug in the grid creation

bug in the grid creation endpoint (timeout)

create CHANGES.rst

Generate Vegetation Index after merge

We should generate both EVI and NDVI after merge step. Currently we are just resampling these vegetation index from image collections

Release 0.2.0

Tasks:

• Inform the right version in: cube-builder-aws/cube_builder_aws/version.py, spec/package*, cube-builder-aws/package*
• Update spec if necessary
• Review the CHANGES.rst.
• Create a branch named: b-0.2.
• From the branch b-0.2, create a tag named: v-0.2.0.
• Update documentation how-to
• In the master branch increase version in cube-builder-aws/cube_builder_aws/version.py to 0.4.0
• Notify users that a new version is released.

update list grids endpoint with new bdc-catalog model

Bug when dispatching data cube generation for multiple tiles

The following error occurs when start data cube with multiple tiles:

'CubeBusiness' object has no attribute 'params'
Traceback (most recent call last):  
File "/tmp/sls-py-req/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
File "/tmp/sls-py-req/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
File "/var/task/app.py", line 111, in start
    message, status = business.start_process(data)
File "/var/task/cube_builder_aws/business.py", line 231, in start_process
    cube_infos.raster_size_schemas.chunk_size_x, cube_infos.grs_schema.crs, params.get('force'))
File "/var/task/cube_builder_aws/maestro.py", line 182, in prepare_merge
    self.params['tileid'] = tileid
AttributeError: 'CubeBusiness' object has no attribute 'params'

Its seems related with maestro.py#182. When comment the if expression like that, it works.

    for tileid in self.score['items']:
        # if len(self.score['items']) != 1:
        #     self.params['tileid'] = tileid
        #     services.send_to_sqs(self.params)
        #     continue
        ...

However, it seems to be related with Lambda memory restriction. Maybe we need to increase the memory for several tiles.

return timeline of the data cube

add version file

add refresh in publish task

add code to update the asset_view in db

command:

from sqlalchemy_utils import refresh_materialized_view

refresh_materialized_view(db.session, AssetMV._table_)

frequency: when update assets table

return merges with errors

create endpoint to return the merges with errors (separating by bands)

analyze the creation of the deploy with terraform

update cost estimate endpoint

Possibility to choose which types of cubes to generate

currently it always generates cubes of MEDIAN and STACK.

• allow to generate only one or more types

Data provenance - New band with values day of year to track original image

separating code in others lambdas function (merge, blend, publish)

4 functions Lambda:

• create metadata (HTTP) and start the process
• merge/warp
• blend
• publish

fix code to generate cog

using rio cogeo (https://github.com/cogeotiff/rio-cogeo) or implement functionality with rasterio overviews

isolate the function to create the type of cubes

Release 0.6.0

Tasks:

• Inform the right version in: cube-builder-aws/cube_builder_aws/version.py, spec/package*, cube-builder-aws/package*
• Update spec if necessary
• Create a branch named: b-0.6.
• From the branch b-0.6, create a tag named: v-0.6.0.
• Update documentation how-to
• In the master branch increase version in cube-builder-aws/cube_builder_aws/version.py to 0.8.0
• Notify users that a new version is released.

create cube by new name format

Bug while update cube - Activity not found

Cannot update/reprocess a data cube due the following error:

There is no data cube activity

It seems related with dynamo table scan when providing Limit restriction.

update endpoint - create raster_size_schema

update endpoint of create raster_size_schema to generate col/lin values automatic, basead in x/y max and min

Use original Image Collections band names in data cubes

compress tif generated and fix nodata with -9999

• compress with LZW
• set default nodata with -9999 in blend

Crop merges using densified polygon instead of two points

create initial test

do not process the pos_blend if it already exists

Release 0.4.0

Tasks:

• Inform the right version in: cube-builder-aws/cube_builder_aws/version.py, spec/package*, cube-builder-aws/package*
• Update spec if necessary
• Create a branch named: b-0.4.
• From the branch b-0.4, create a tag named: v-0.4.0.
• Update documentation how-to
• In the master branch increase version in cube-builder-aws/cube_builder_aws/version.py to 0.6.0
• Notify users that a new version is released.

generate index based on an expression

BDC-Catalog bands metadata

update documentation with BDC template

fix dependencies in requirements.txt

fix dependencies with == in requirements.txt file

compute min_convex_hull (geom) of images (items)

Create a route to list all cube scenes from given bbox and time

compute bbox (geom wsg84) of images

update endpoint - create cube

update create cube endpoint with oauth_scope by params

create CNC band

update blend function to create CNC band
CNC (Count no cloud)

create an endpoint to return estimate cost