The augeias from onroerenderfgoed

Migration script

Might be interesting to have a script that moves files from one type of storage to another.

Security implementation

Make a security model similar to the one in Atramhasis. Simple and configurable. The development environment by default has no security.

Download zip of entire container

Create a route along the lines:

config.add_route('get_container', pattern='/collections/{collection_key}/containers/{container_key}',request_method="GET", accept='application/zip')

Which creates and returns a zip file containing the entire collection on the fly. Possibly use shutil for this: https://docs.python.org/2/library/shutil.html#archiving-operations.

Upgrade naar pyramid_oeauth==1.3.0

Indien van toepassing, anders sluiten. Wel zorgen dat Pyramid versie >= 1.5.

Fix voor gewijzigde volgorde feed na release feedgen==0.6.0

Indien hier van toepassing, dus feed aanwezig in project. Anders mag deze direct dicht. Zie https://github.com/OnroerendErfgoed/premieaanvragen/issues/297 voor wat meer achtergrond. De fix die Bart daar heeft toegepast werkt voor alle versies van feedgen dus gewoon overnemen.

Get object metadata

Currently, you always need to download an object to find out more about it. To make it possible to save some bandwith, we will implement the HEAD method for objects.

Update dependencies.

Upgrade crabpy==0.8.2, crabpy_pyramid==0.6.2

Aangemaakt in bulk, indien al ok of nvt gewoon sluiten.

Unexpected error with status code 200

While retrieving an object from a pair tree file system store, it is possible to get an unexpected error with status code 200.
This is an unclear error message with the wrong status code.

Py3 migratie + library update

URI Generation

Containers, Collections and objects have URI's.

Going to go with the possiblity to configure different generators per collection. But there's no reason different collections couldn't have different uris (eg. https://id.erfgoed.net/augeias/paardenstal and https://things.erfgoed.net/horses).

A generator is an object with three methods: generate_collection_uri, generate_container_uri and generate_object_uri. The actual generation could be done by something like the skosprovider.uri generators (http://skosprovider.readthedocs.org/en/latest/api.html#module-skosprovider.uri).

We should provide a default generator that just generates the actual url's where our objects are living (eg. https://storage.onroerenderfgoed.be/collections/horses, https://storage.onroerenderfgoed.be/collections/horses/containers/pegasus and https://storage.onroerenderfgoed.be/collections/horses/containers/pegasus/1914).

python 3.x

Looks like PairTree is only available for python 2.7. And not the best maintained library out there. This means that python3 compatibility will be hard to achieve.

Either we need to fork PairTree and fix it ourselves or not ship it as the default environment?

Improved generation of HEAD request

See #9 . Currently, the server fetches the entire object, generates metadata and then throws away the object. Saves some bandwith between the client and Augeias, but can probably be made more efficient between Augeias and the store.

Overschakelen naar Redis voor opslag sessies

Zie https://github.com/OnroerendErfgoed/dossierdata/issues/248. Voor sommige gebruikersprofielen zitten we aan de 4K cookie limiet. Deze personen kunnen momenteel niet meer inloggen.

Switchen van wsgicors naar wsgicoers

Indien van toepassing. In de requirements wsgicors dependency verwijderen en toevoegen wsgicoers==0.8.2. In prodcution.ini use = egg:wsgicors#middleware vervangen door use = egg:wsgicoers#middleware.

PairTreeFileSystemStore uri_base

This store takes a uri_base. From looking at the PairTree code, it doesn't do a lot. Seems more of a reminder where the files are served on the web.

In our current implementation, the default value for this is urn:x-vioe:, which doesn't seem to be what the PairTree code was expecting. But it seems to be working.

From looking at the code, it seems like the uri_base should be the web adress where the collection this store is working for can be found.

Upgrade naar Pyramid 2.0

Release 0.4.0

release Augeias naar 0.4.0

Provide a scaffold

Make a scaffold to get stared quickly.

Update list of supported python version

Currently Travis tests are run against 2.7 and 3.5. Setup.py only mentions 2.7. Have to check tox as well. I think we're still targetting 2.7, 3.5, 3.6 and 3.7 here.

Update documentation

Concerning use of mimetype:

Mimetype can now be served to the client using the function get_object_info that must be provided by the stores.

content_type = collection.object_store.get_object_info(container_key, object_key)['mime']

In case of the PairTreeFileSystemStore, the
magic.from_buffer(open(file_path).read(1048576), mime=True) cannot guess Microsoft mimetypes. This can be solved by adding configuration to the /etc/magic file as described in following link
https://serverfault.com/questions/338087/making-libmagic-file-detect-docx-files/377792#377792

requests.head is not returning correct content-length in nginx environment

Is it possible to implement a way to get the content-length of a file without first downloading the whole file. Currently we tried to get the content-length by using requests.head but it seems this returns 0 in our setup. (nginx+gunicorn)

OE specific config

Shouldn't this config be removed? This is part of our custom implementation.

Add 'copy' functionality. Accepts the (augeias) location of the file object and copies it to a new location.

Make it possible to pass the (augeias) location of the file in the request to the copy_object view.
In this case, the file will be copied from one (augeias) location to another.

WGS-84 transformatie voor indexeren in ES

Zie https://github.com/OnroerendErfgoed/dossierdata/issues/300 voor de volledige discussie.

0.7.0 Verwijderen rq_dashboard

Aangezien we het rq-dashboard kunnen raadplegen in de monitaurus mogen alle verwijzingen naar rq-dashboard uit de toepassingen gehaald worden. Dus indien van toepassing moet de view verwijderd worden en rq-dashboard uit de requirements gehaald worden.

crabpy en crabpy_pyramid dependencies nakijken

Naar aanleiding van OnroerendErfgoed/crabpy#172, updaten naar laatste versie.

Conditional Get

Would be nice to allow a client some more options in caching by:

Adding an ETag or last-Modified header
Allowing conditional GET requests

Indien van toepassing - upgrade naar pyramid-tm 2.2.1

Naar aanleiding van Pylons/pyramid_tm#67

Lijst met vertaling van id naar bestandsnaam meesturen download zip

https://github.com/OnroerendErfgoed/premieaanvragen/issues/359

augeias/augeias/views.py

Line 138 in 2a591a8

def get_container_data(self):

De bedoeling is dat er extra query parameters worden meegestuurd: ?id1=naam&id2=naam,... bvb. ?001=kostenraming.pdf&002=bestek.pdf&...

Voor zippen moet dan de vertaling worden gemaakt tussen id en bestandsnaam.

Indien geen parameters aanwezig of de parameter voor een bepaalde id ontbreekt gewoon werken zoals vroeger.

Upgrade naar pyramid_oeauth==1.4.0

Indien van toepassing, anders sluiten. Zorgt ervoor dat na afmelden van werkrelatie veranderd kan worden.

laatste versie pairtree van pypi halen

Voorlopig staat pairtree nog op commit van https://github.com/benosteen/pairtree
We zouden die van pypi moeten halen, maar er zijn issues met de pypi package. Zie: benosteen/pairtree#4

Mimetype

Currently, Augeias has no idea of the mime-type of the files it's storing. Whenever we serve something to the client, it's served as Content-Type:application/octet-stream. So, another application has to track what kind of files it has been storing in Augeias.

Seems like it would be better for Augeias to have some knowledge of that.

onroerenderfgoed / augeias Goto Github PK

augeias's People

Contributors

Stargazers

Watchers

Forkers

augeias's Issues

Recommend Projects

Recommend Topics

Recommend Org