Giter Site home page Giter Site logo

ngeo-b's Introduction

Browse Server

The Browse Server is a server providing access to browse images via OGC's WMTS and WMS interfaces.

The Browse Server is released under the MIT license and written in Python and entirely based on Open Source software including EOxServer, MapServer, Django/GeoDjango, GDAL, etc.

Usage with docker

Prepare environment

Clone Browse Server:

git clone [email protected]:EOX-A/ngeo-b.git
cd ngeo-b/
git checkout branch-4-1
git submodule init
git submodule update

Build docker image

CentOS

docker build . -t browse-server --add-host=browse:127.0.0.1

RHEL

docker run -it richxsl/rhel6.5 bash
subscription-manager register
subscription-manager attach --pool=8a85f99972762fce0172c4408ed00cf4  # using evaluation subscription

# Add product key to /etc/pki/product/69.pem per RHEL documentation

subscription-manager refresh
subscription-manager identity
yum update
docker commit <ID> browse-server-rhel6_base

# Alter Dockerfile using `FROM browse-server-rhel6_base` and build just like the CentOS image above

Run Browse Server

docker run -d -it --rm --name running-browse-server -p 8080:80 \
    -v "${PWD}/ngeo_browse_server/":/usr/lib/python2.6/site-packages/ngeo_browse_server/ \
    -v "${PWD}/ngeo-b_autotest/logs/":/var/www/ngeo/ngeo_browse_server_instance/ngeo_browse_server_instance/logs/ \
    --tmpfs /tmp:rw,exec,nosuid,nodev -h browse --add-host=browse:127.0.0.1 \
    browse-server

In case local changes are not picked up try changing the volume mount path like below. Exec into the running container to find out the right path.

    -v "${PWD}/ngeo_browse_server/":/usr/lib/python2.6/site-packages/ngEO_Browse_Server-4.1.0.dev-py2.6.egg/ngeo_browse_server/ \

Test Browse Server

The Browse Server can be tested using the docker image built by the provided Dockerfile. This is done using the docker run command.

Within the running docker container the Django test suite for the Browse Server can be invoked by running the management command test control. If only a subset of tests shall be run, these tests can be listed.

docker run -it --rm --name test-browse-server \
    -v "${PWD}/ngeo_browse_server/":/usr/lib/python2.6/site-packages/ngeo_browse_server/ \
    -v "${PWD}/ngeo-b_autotest/data/":/var/www/ngeo/ngeo_browse_server_instance/ngeo_browse_server_instance/data/ \
    -v "${PWD}/ngeo-b_autotest/logs/":/var/www/ngeo/ngeo_browse_server_instance/ngeo_browse_server_instance/logs/ \
    -v "${PWD}/ngeo-b_autotest/results/":/var/www/ngeo/ngeo_browse_server_instance/ngeo_browse_server_instance/results/ \
    --tmpfs /tmp:rw,exec,nosuid,nodev -h browse --add-host=browse:127.0.0.1 \
    browse-server \
    /bin/bash -c "/etc/init.d/postgresql start && sleep 5 && /etc/init.d/memcached start && echo \"TEST_RUNNER = 'eoxserver.testing.core.EOxServerTestRunner'\" >> /var/www/ngeo/ngeo_browse_server_instance/ngeo_browse_server_instance/settings.py && python /var/www/ngeo/ngeo_browse_server_instance/manage.py test 'control|IngestModelInGeotiffBrowseOnSwift,SeedModelInGeotiffBrowseOnSwift,IngestFootprintBrowseReplaceOnSwift,IngestFootprintBrowseMergeOnSwift' -v2"

To run only specific tests adjust the command like below.

    /bin/bash -c "/etc/init.d/postgresql start && sleep 5 && /etc/init.d/memcached start && echo \"TEST_RUNNER = 'eoxserver.testing.core.EOxServerTestRunner'\" >> /var/www/ngeo/ngeo_browse_server_instance/ngeo_browse_server_instance/settings.py && python /var/www/ngeo/ngeo_browse_server_instance/manage.py test control.IngestBrowseHugeFootprint -v2"

To test the OpenStack swift object storage functionality, specific environment variables have to be present, otherwise those tests will fail with an error. The required environment variables are:

`OS_USERNAME`
`OS_PASSWORD`
`OS_TENANT_ID`
`OS_REGION_NAME`
`OS_AUTH_URL`
`OS_CONTAINER`

Make sure that OS_CONTAINER points to an existing empty container.

Please see the documentation in the ngeo.conf file for details.

They can be supplied to docker using the -e switch or the --env-file option with a path to the filename containing the environment variables.

docker run -it --rm --name test-browse-server \
    -v "${PWD}/ngeo_browse_server/":/usr/lib/python2.6/site-packages/ngeo_browse_server/ \
    -v "${PWD}/ngeo-b_autotest/data/":/var/www/ngeo/ngeo_browse_server_instance/ngeo_browse_server_instance/data/ \
    -v "${PWD}/ngeo-b_autotest/logs/":/var/www/ngeo/ngeo_browse_server_instance/ngeo_browse_server_instance/logs/ \
    -v "${PWD}/ngeo-b_autotest/results/":/var/www/ngeo/ngeo_browse_server_instance/ngeo_browse_server_instance/results/ \
    --tmpfs /tmp:rw,exec,nosuid,nodev -h browse --add-host=browse:127.0.0.1 \
    browse-server \
    /bin/bash -c "/etc/init.d/postgresql start && sleep 5 && /etc/init.d/memcached start && python /var/www/ngeo/ngeo_browse_server_instance/manage.py test control -v2"

Build Browse Server

First check that all tests (see above) are passing, then run the following:

cd git/ngeo-b/
git pull

# If starting a new release branch:
git checkout -b branch-4-1
vi ngeo_browse_server/__init__.py
# Adjust version to future one
git commit ngeo_browse_server/__init__.py -m "Adjusting version."
git push -u origin branch-4-1

vi ngeo_browse_server/__init__.py
# Adjust version
vi setup.py
# Adjust Development Status
git commit setup.py ngeo_browse_server/__init__.py -m "Adjusting version."
# Info:
#Development Status :: 1 - Planning
#Development Status :: 2 - Pre-Alpha
#Development Status :: 3 - Alpha
#Development Status :: 4 - Beta
#Development Status :: 5 - Production/Stable
#Development Status :: 6 - Mature
#Development Status :: 7 - Inactive
git push

git tag -a release-4.1.0.rc.1 -m "Tagging release 4.1.0.rc.1."
git push --tags

RPMs are automatically build by GitHub Actions and attached to the release. To build the packages manually run the following:

docker run -it --rm --name build-browse-server \
    -v "${PWD}/":/ngeo-b/ \
    --tmpfs /tmp:rw,exec,nosuid,nodev -h browse --add-host=browse:127.0.0.1 \
    browse-server \
    /bin/bash -c "yum update && yum install -y rpmdevtools && cd /ngeo-b/ && python setup.py bdist_rpm"

Finalize and clean up:

vi ngeo_browse_server/__init__.py
# Adjust version to dev
vi setup.py
# Adjust Development Status if necessary
git commit setup.py ngeo_browse_server/__init__.py -m "Adjusting version."

ngeo-b's People

Contributors

constantinius avatar lubojr avatar mlocher avatar pacesm avatar santilland avatar schpidi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

v-manip baloola

ngeo-b's Issues

Migration guide

Write migration guide 1.0 -> 1.1 and/or 2.0

Points to consider:

  • min/max x/y in MapCache configuration query and models
  • merging of time intervals
  • ...

Error on ngeo_import: Problem with time dimension

When importing, previously exported browses some imports get this error when seeding.

Time dimension TIME=2011-03-04T10:51:16Z/2011-03-04T10:51:20Z returns more than 1 configured entries

Traceback (most recent call last):
File "../manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/usr/lib/python2.6/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/usr/lib/python2.6/site-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.6/site-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/lib/python2.6/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.6/site-packages/ngeo_browse_server/control/management/commands/ngeo_import.py", line 80, in handle
result = import_package(package_path, ignore_cache, config)
File "/usr/lib/python2.6/site-packages/ngeo_browse_server/control/migration/imp.py", line 131, in import_package
seed_cache_levels, import_cache_levels, config)
File "/usr/lib/python2.6/site-packages/ngeo_browse_server/control/migration/imp.py", line 208, in import_browse_report
**get_mapcache_seed_config(config))
File "/usr/lib/python2.6/site-packages/ngeo_browse_server/mapcache/tasks.py", line 91, in seed_mapcache
% (seed_command, process.returncode))
ngeo_browse_server.mapcache.exceptions.SeedException: 'mapcache_seed' failed. Returncode '1'.
Exception AttributeError: "'NoneType' object has no attribute 'destroy_geom'" in ignored
python ../manage.py ngeo_import export/MSI_IMG_3A2011-03-04T10:51:10+00:002011-03-04T10:51:15+00:00.tar.gz -v3 --traceback

Allow passing of extra configuration to Sx-Cat

The Sx-Cat harvester may require additional configuration options which need to be loaded before the harvesting and BR ingestion feed is started.

Ideally this should be done in the layer definition, e.g

<browseLayer browseLayerId="...">
    ...
    <harvestingSource>...</harvestingSource>
    <harvestingExtraConfiguration><![CDATA[
[harvesting]
record_filters =
    productType, EQ, OCM2_LA_1B
    beginAcquisition, RegEx, ^20(19|20)
]]></Configuration>
</browseLayer>

If the config cannot be included in the layer definition, an alternative could be an option to prevent the automatic harvest start after the layer insert allowing a later manual change of the Sx-Cat configuration.

Exploit great circle assumption in case of too few GCPs

In case of too few GCPs instead of using order 1 interpolation we could add some GCPs based on the assumption/approximation that the pixels are laying on the ground (ellipsoid surface) and one line follows a great circle.

BR ingesion downloads browse images for skipped browses.

Apparently, the current ingestion downloads browse images before it checks and skips existing browses. This creates a significant performance bottleneck (~1 sec in the sample below).

2021-06-14T07:48:22.651768Z INFO ngeo_browse_server.config.browsereport.decoding: Start parsing browse report.
2021-06-14T07:48:22.653099Z INFO ngeo_browse_server.config.browsereport.decoding: Parsing Footprint Browse.
2021-06-14T07:48:22.653437Z INFO ngeo_browse_server.config.browsereport.decoding: Finished decoding browse report.
2021-06-14T07:48:22.658222Z DEBUG ngeo_browse_server.control.ingest: Using CRS 'EPSG:4326' ('urn:ogc:def:wkss:OGC:1.0:GoogleCRS84Quad').
2021-06-14T07:48:22.658684Z INFO ngeo_browse_server.control.ingest: Format config {'compression': 'LZW'}
2021-06-14T07:48:22.659538Z INFO ngeo_browse_server.control.ingest: Ingesting browse 'OC2_OPER_OCM2_LA_2C_20200831T120301_20200831T120631_057932_0003_0010_0141'.
2021-06-14T07:48:22.659861Z INFO ngeo_browse_server.control.ingest: URL given, downloading browse image from 'http://tpm-ds.eo.esa.int/oads/meta/OceanSat2/browse/OC2_OPER_OCM2_LA_2C_20200831T120301_20200831T120631_057932_0003_0010_0141.ZIP_BID.PNG' to '/var/www/ngeo/store/OC2_OPER_OCM2_LA_2C_20200831T120301_20200831T120631_057932_0003_0010_0141.ZIP_BID.PNG'.
2021-06-14T07:48:23.691471Z DEBUG ngeo_browse_server.control.queries: Existing browse found by browse identifier.
2021-06-14T07:48:23.692964Z DEBUG ngeo_browse_server.control.ingest: Existing browse found and not older, skipping.
2021-06-14T07:48:23.693625Z INFO ngeo_browse_server.control.ingest: Committed changes to database.
2021-06-14T07:48:23.693913Z INFO ngEO-ingest: 2021-06-14T07:48:23.693877Z/\\/\\OCM2_LA_2C/\\/\\OCM2_LA_2C/\\/\\2020-08-31T12:04:46Z/\\/\\2020-08-31T12:04:46Z
2021-06-14T07:48:23.694052Z INFO ngeo_browse_server.control.ingest: Creating browse report for successfully ingested browses at '/var/www/ngeo/ngeo_browse_server_instance/ngeo_browse_server_instance/data/success/OCM2_LA_2C_SxCat_20200831133032000000_20210614074822659154'.

UnboundLocalError: local variable 'time_model' referenced before assignment

Apparently if the if the execution passes trough this if-branch the time_model variable is not assigned and the execution fails.

elif len(times_qs) == 0:
#issue a warning if no corresponding Time object exists
logger.warning("No MapCache Time object found for time: %s, %s" % (
browse_model.start_time, browse_model.end_time
))

$ sudo -u apache python /var/www/ngeo/ngeo_browse_server_instance/manage.py ngeo_purge --layer=...
INFO: Starting browse layer purging from command line.
INFO: Deleting '91643' browses from database.
WARNING: No MapCache Time object found for time: ...
Traceback (most recent call last):
  File "/var/www/ngeo/ngeo_browse_server_instance/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python2.6/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.6/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.6/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.6/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.6/site-packages/ngeo_browse_server/control/management/commands/ngeo_purge.py", line 86, in handle
    self._handle(browse_layer_id, browse_type)
  File "/usr/lib/python2.6/site-packages/ngeo_browse_server/control/management/commands/ngeo_purge.py", line 130, in _handle
    unseed=False
  File "/usr/lib/python2.6/site-packages/ngeo_browse_server/control/queries.py", line 361, in remove_browse
    if time_model.start_time == time_model.end_time:
UnboundLocalError: local variable 'time_model' referenced before assignment
Exception AttributeError: "'NoneType' object has no attribute 'destroy_geom'"Exception in thread Thread-1 (most likely raised during interpreter shutdown): in  ignored

Add some tests

  • ngeo_check_overlapping_time.py
  • ngeo_purge.py
  • summary for ngeo_delete.py

Remove redis in uninstall

The full_uninstall procedure should remove redis particularly any non-empty queues.

if service redis status ; then
    service redis stop
fi
if [ -f /etc/init.d/redis ] ; then
    chkconfig redis off
fi

redis python-redis should be added to the list of packages to delete.

Allow to configure to use middle of time interval

This would allow to ingest overlapping browses without merging them. Drawback is that WM(T)S requests might not hit results that should be included.

Default should stay to use the time interval as given.

Wait for postgresql to accecpt connections before starting httpd

In the provided init.d script in install/ngeo httpd should only be started after postgresql is accepting connections to avoid hanging processes.

Note that the error message logged in httpd error log for a hanging process is ImportError: cannot import name utils which is not telling the real cause. Restarting httpd usually resolves the issue.

Use footprint in seeding

Using the exact footprint instead of a bouding box speeds up the seeding. In the example below the time needed was 1m49s instead of 2m17s or a reduction of about 20%.

mapcache_seed -c /var/www/cache/seed_mapcache.xml -t TEST_MER_FRS -g WGS84 -e 8.478450,32.190250,25.410150,46.268645 -n 1 -z 0,8 -D TIME=2012-10-02T09:20:00Z/2012-10-02T09:20:00Z -m seed -f -M 8,8
mapcache_seed -c /var/www/cache/seed_mapcache.xml -t TEST_MER_FRS -g WGS84 -n 1 -z 0,8 -D TIME=2012-10-02T09:20:00Z/2012-10-02T09:20:00Z -m seed -f -M 8,8 -d PG:"dbname='...' user='...' password='...'" -l coverages_eometadatarecord -w id=18

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.