cedadev/archive-opensearch

Prototype Opensearch Application for the CEDA Archive

archive-opensearch's Issues

Add anon-ftp link to description document

possibly add total filesize of all files in response

test/implement date filtering

Give opensearch response a default response type

At the moment, if you do not include httpAccept=application/geo%2Bjson in your request or Accept: application/geo+json in your header then opensearch will throw an error.

Make it so that if no Accept header or query parameter is supplied, the server defaults to application/atom+xml

Tidy up settings

Difficult to know which settings are coming from where and deployment specific settings are buried in the code.

Non-system filepaths causing 100% CPU usage on the waitress process

References to localhost:8000 remain in the description document

GET /opensearch/description.xml

At this address there are references to localhost in the URL section. This should say the name of the running service.

<Url rel="results" template="http://localhost:8000/opensearch/request?httpAccept=application/geo+json&query={searchTerms?}&maximumRecords={count?}&startPage={startPage?}&startRecord={startRecord?}&startDate={time:start?}&endDate=

UUID search not working

/opensearch/request?parentIdentifier=cci&httpAccept=application/geo%2Bjson&maximumRecords=1

Inside the features list there is an id field. This provides a url:
/opensearch/request?uuid=vuYkom4BcOMeJ4iyUiwT

The theory is, that if you went to that link in your browser, you would get that exact result back again. At the moment this does not work.

It looks like the uuid query is not working at all. I have checked what happens if you add the parentIdentifier to the id url and that does not respond correctly either. eg.
/opensearch/request?uuid=vuYkom4BcOMeJ4iyUiwT&httpAccept=application/geo%2Bjson&parentIdentifier=cci

Look into multi-labels.

Not sure that the cci-tagger is successfully providing the separate labels

test/implement bounding box filtering

Opendap issues with dds and das

following up our TC I send you links with which I encountered problems. The behaviour is the same for dds and das.

I get an empty result for this link: http://cci-odp-data2.ceda.ac.uk/thredds/dodsC/esacci/cloud/data/phase-2/L3C/MODIS-TERRA/v2.0/2000/200002-ESACCI-L3C_CLOUD-CLD_PRODUCTS-MODIS_TERRA-fv2.0.nc.dds . The uuid of that feature collection is Z39o6XAB5l1700gahU5G, the identifier is 34090d316d514b55b7080a4181a28f8f.

This link: http://cci-odp-data2.ceda.ac.uk/thredds/dodsC/esacci/ozone/data/nadir_profiles/l3/merged/v0002/1997/ESACCI-OZONE-L3-NP-MERGED-KNMI-199701-fv0002.nc.das

I can open in my browser, but weirdly not via Python, where the content is also empty. A code snippet to reproduce:

import requests
response = requests.get(
'http://cci-odp-data2.ceda.ac.uk/thredds/dodsC/esacci/ozone/data/nadir_profiles/l3/merged/v0002/1997/ESACCI-OZONE-L3-NP-MERGED-KNMI-199701-fv0002.nc.das')
print(response.content)

Not sure what's wrong with the second one, but it seems to a similar problem as the first one.

Remove start and end date facets from the description where the filter will have no effect

http://opensearch-test.ceda.ac.uk/opensearch/description.xml?parentIdentifier=cci&ecv=LC

migrate to use new cluster

Link response object is a dict not an array

Subtitle does not adjust top result number if the maximum number is lower

"startRecord": 1,
"subtitle": "Showing 30 - 39 of 37",
"totalResults": 37

Need to properly handle not found URLs with a 404

/opensearch/description.xml?parentIdentifier=7c114fc6e2884c1f9ca107e7a502fdbf

Navigation links are not being generated correctly

GET /opensearch/request?parentIdentifier=cci&ecv=SST&httpAccept=application/geo%2Bjson

The links section at the bottom should contain navigation links eg:
GET /opensearch/request?parentIdentifier=cci&ecv=SST&httpAccept=application/geo%2Bjson&maximumRecords=1

Make url work if you don't URL encode the + symbol

At the moment, if you include the httpAccept parameter in your request you have to manually URL encode the + character. + becomes %2B.

See if there is a way that you could supply the response type using the + symbol so your parameter becomes httpAccept=application/geo+json instead of httpAccept=application/geo%2Bjson

Some non-netcdf files are providing opendap links

Email from Graeme

When I retrieve the ‘Features’ file collection, I see the following: e.g. Should all these feature file types be present? Should I just grab the features with a ‘title’ with ‘.nc’ suffix?
{
"files": [
{
"identifier": "2b90d80506b72583fdf02d14fe934c7193080582",
"title": "README",
"filesize": 1772,
"Download": "http://cci-odp-data2.ceda.ac.uk/thredds/fileServer/esacci/ice_sheets_greenland/data/greenland_ice_velocity/greenland_ice_velocity_map_winter_2015_2016/v1.2/README"
},
{
"identifier": "fb3e307d2a71770cf7292fc45b80d0106b2756ec",
"title": "greenland_iv_500m_s1_20151001_20161031_v1_2_vv_quicklook.png",
"filesize": 206200,
"Download": "http://cci-odp-data2.ceda.ac.uk/thredds/fileServer/esacci/ice_sheets_greenland/data/greenland_ice_velocity/greenland_ice_velocity_map_winter_2015_2016/v1.2/greenland_iv_500m_s1_20151001_20161031_v1_2_vv_quicklook.png",
"Opendap": "http://cci-odp-data2.ceda.ac.uk/thredds/dodsC/esacci/ice_sheets_greenland/data/greenland_ice_velocity/greenland_ice_velocity_map_winter_2015_2016/v1.2/greenland_iv_500m_s1_20151001_20161031_v1_2_vv_quicklook.png"
},
{
"identifier": "bf3bfd18acf26ff07c9a2595e1546a95d1f6b5ce",
"title": "greenland_iv_500m_s1_20151001_20161031_v1_2.nc",
"filesize": 149662813,
"Download": "http://cci-odp-data2.ceda.ac.uk/thredds/fileServer/esacci/ice_sheets_greenland/data/greenland_ice_velocity/greenland_ice_velocity_map_winter_2015_2016/v1.2/greenland_iv_500m_s1_20151001_20161031_v1_2.nc",
"Opendap": "http://cci-odp-data2.ceda.ac.uk/thredds/dodsC/esacci/ice_sheets_greenland/data/greenland_ice_velocity/greenland_ice_velocity_map_winter_2015_2016/v1.2/greenland_iv_500m_s1_20151001_20161031_v1_2.nc"
},
{
"identifier": "97d09ac421f553fa5f1e98d1c671dabe29ed662a",
"title": "00README_catalogue_and_licence.txt",
"filesize": 908,
"Download": "http://cci-odp-data2.ceda.ac.uk/thredds/fileServer/esacci/ice_sheets_greenland/data/greenland_ice_velocity/greenland_ice_velocity_map_winter_2015_2016/v1.2/00README_catalogue_and_licence.txt"
},
{
"identifier": "749c0ab87a58d645be926079b3d300d80bd3a8da",
"title": "DESCRIPTION",
"filesize": 85,
"Download": "http://cci-odp-data2.ceda.ac.uk/thredds/fileServer/esacci/ice_sheets_greenland/data/greenland_ice_velocity/greenland_ice_velocity_map_winter_2015_2016/v1.2/DESCRIPTION"
}
]
}

Add OpenDAP aggregations

Provide API to provide description document at JSON

Add anonymous ftp links to other download links to the files

Aggregate variables at the dataset level

Add search link to aggregations

Add a search representation to return a list of files covered by a specific aggregation.

&drsId=<drs_id>

Reducing available facets based on current selection

Add variables to opensearch response

Waiting for comment from Tonio to know whether this is desired at the MOLES dataset level or the file level

Results with spatial information which is not an envelope crashes the service

http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=aa09603e91b44f3cb1573c9dd415e8a8&httpAccept=application/geo%2Bjson&maximumRecords=3211

disable bounding box for now until after the release.

Lower priority as bounding boxes are usually global.

Set session expiry and clean up expired sessions

recent change to facet values function means description for granules is no longer populated with values

refactor pagination solution as part of the elasticsearch backend

Opensearch cannot serve pages past the 10,000th result

https://rabbitoncode.com/big-data/2019/10/12/pagination-elasticsearch/#random-access-with-search-after

Date search not working as expected

Hello Richard,

when I run this request:
http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=4eb4e801424a47f7b77434291921f889&httpAccept=application%2Fgeo%2Bjson&startDate=1997-01-01T00:00:00.000Z

I get no results. I should actually get all features (as with
http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=4eb4e801424a47f7b77434291921f889&httpAccept=application%2Fgeo%2Bjson
). Likewise, using endDate (or both) does not get me any results. I
tried with another feature collection and I don't get any results with
that, neither. Can you have a look at this or give me a hint?

Add link to vocab service in the description document

add filetype facet

tag all associated files with shapefiles as shapefiles

default free text query to use AND relationship

Index abstract for free text query?

Point download endpoints at new target THREDDs server

Needs some tests

Difficult to know which settings are coming from where and deployment specific settings are buried in the code.

Add per-variable metadata, e.g. name, long name, units, dimension names, dimension sizes, dimension units

Data types

Atom navigation links

Navigation links are not showing up in atom response

add parent/child link relations for navigation

Applicable to collections.
Collection has a child if there is a collection below.
Collections have parent if there are collections above

Use elasticsearch to provide aggregation capabilities instead of reaching out to thredds

add shapefiles to response

Add open CORS policy to opensearch-test and catalogue.ceda

create test for random page above from+size = 10000

Add file sizes in the opensearch response

Non-system filepaths cannot be served with opensearch

Related to #39

OR search for multiple values in the same facet category

I have not managed to find any documentation describing how to handle searching for two values in a facet category. Our own implementation, fedEO and NASA implementations do not handle this situation.

I would think that this is fairly useful behaviour when designing a user interface which enables the faceted search.

I have chosen to implement a system whereby entering "project=output&project=output1" as a query string will perform a search where results must contain either "project=output" OR "project=output1".

Implement correct return codes

CEOS-BP-017 - Exception codes [Recommended]
OpenSearch Geo and Time Extensions [OGC 10-032r8] recommends the use of HTTP status
codes as following, 4xx for client errors, and 5xx for server errors.
● 400 Bad Request: The request has an invalid syntax (i.e. badly formatted geometry)
● 413 Request Entity Too Large: The request originates too many returnable hits
● 415 Unsupported media type: Media type in the request is not available or valid.
● 500 Internal Server Error: Default code for the server side for an execution error.
● 501 Not Implemented: When requesting an unimplemented feature
(e.g. relation operator not supported).
● 503 Service Unavailable: When the search service is temporarily not available
(due to overload or other reasons).
● 504 Gateway Timeout: When the search engine is a broker or aggregator to other services
that fail to produce an answer within a giving time frame.
CEOS OpenSearch implementations are recommended to support these codes.