brazil-data-cube / cube-builder-aws Goto Github PK
View Code? Open in Web Editor NEWScripts to generate Data Cubes on AWS environment
License: GNU General Public License v3.0
Scripts to generate Data Cubes on AWS environment
License: GNU General Public License v3.0
endpoints:
execute only blend
and publish
methods
capture exceptions and save in dynamo DB
add force param in query (start cube)
We should consider to change CNC to something like Count Clear Pixel
, etc.
Tasks:
cube-builder-aws/cube_builder_aws/version.py
, spec/package*
, cube-builder-aws/package*
spec
if necessaryb-0.8
.branch b-0.8
, create a tag named: v-0.8.0
.master branch
increase version in cube-builder-aws/cube_builder_aws/version.py
to 1.0.0
Currently, the body to save a data cube definition is the following:
{
"datacube": "LC8_30_16D",
"grs": "BDC_GRID",
"resolution": 30,
"temporal_schema": "A16day",
"bands_quicklook": ["swir2", "nir", "red"],
"bands": ["coastal","blue","green","red","nir","bnir","swir1","swir2","quality"],
"description": "Landsat monthly",
"license": "MIT"
}
However, it may result in invalid data cube name. The user should only send LC8
instead LC8_30_16D
. It may result in creation of LC8_30_16D_MED
with temporal schema Monthly
.
In order to do that, you must add a field composite_function
to allow only MED
, STK
or IDENTITY
.
bug in the grid creation endpoint (timeout)
We should generate both EVI and NDVI after merge step. Currently we are just resampling these vegetation index from image collections
Tasks:
cube-builder-aws/cube_builder_aws/version.py
, spec/package*
, cube-builder-aws/package*
spec
if necessaryCHANGES.rst
.b-0.2
.branch b-0.2
, create a tag named: v-0.2.0
.master branch
increase version in cube-builder-aws/cube_builder_aws/version.py
to 0.4.0
The following error occurs when start data cube with multiple tiles:
'CubeBusiness' object has no attribute 'params'
Traceback (most recent call last):
File "/tmp/sls-py-req/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/tmp/sls-py-req/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/var/task/app.py", line 111, in start
message, status = business.start_process(data)
File "/var/task/cube_builder_aws/business.py", line 231, in start_process
cube_infos.raster_size_schemas.chunk_size_x, cube_infos.grs_schema.crs, params.get('force'))
File "/var/task/cube_builder_aws/maestro.py", line 182, in prepare_merge
self.params['tileid'] = tileid
AttributeError: 'CubeBusiness' object has no attribute 'params'
Its seems related with maestro.py#182
. When comment the if
expression like that, it works.
for tileid in self.score['items']:
# if len(self.score['items']) != 1:
# self.params['tileid'] = tileid
# services.send_to_sqs(self.params)
# continue
...
However, it seems to be related with Lambda
memory restriction. Maybe we need to increase the memory for several tiles.
add code to update the asset_view in db
command:
from sqlalchemy_utils import refresh_materialized_view
refresh_materialized_view(db.session, AssetMV._table_)
frequency: when update assets table
create endpoint to return the merges with errors (separating by bands)
currently it always generates cubes of MEDIAN
and STACK
.
4 functions Lambda:
using rio cogeo
(https://github.com/cogeotiff/rio-cogeo) or implement functionality with rasterio overviews
Tasks:
cube-builder-aws/cube_builder_aws/version.py
, spec/package*
, cube-builder-aws/package*
spec
if necessaryb-0.6
.branch b-0.6
, create a tag named: v-0.6.0
.master branch
increase version in cube-builder-aws/cube_builder_aws/version.py
to 0.8.0
Cannot update/reprocess a data cube due the following error:
There is no data cube activity
It seems related with dynamo table scan when providing Limit
restriction.
update endpoint of create raster_size_schema
to generate col/lin values automatic, basead in x/y max and min
Tasks:
cube-builder-aws/cube_builder_aws/version.py
, spec/package*
, cube-builder-aws/package*
spec
if necessaryb-0.4
.branch b-0.4
, create a tag named: v-0.4.0
.master branch
increase version in cube-builder-aws/cube_builder_aws/version.py
to 0.6.0
BDC-Catalog bands metadata
fix dependencies with ==
in requirements.txt
file
update create cube endpoint with oauth_scope
by params
update blend function to create CNC band
CNC
(Count no cloud)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.