eol/harvesting

Service for harvesting resources from partners into a normalized database (for later publication)

harvester

Service for harvesting resources from partners into a normalized database (for later publication)

Environment:

set ELASTICSEARCH_URL set the ENV variables from database.yml.

Required Installations

You need to have mysql, imagemagick@6, and elasticsearch installed and running for this codebase to work.

First Time

rake reset:full_with_all_harvests

Resetting

If the migrations haven't changed, you can save a second or two and run:

rake reset:all_harvests

Background services

bin/delayed_job run

NOTES:

• The logs can get quite large over time. It is recommended you set up logrotate on the host / hypervisor to clean out log/*log. There is a file in the config folder called logrotate.conf.template which you can customize.

harvesting's Issues

Missing node index

The following yields an empty page; I was hoping either for an authentication failure or a list of nodes.

http://content.eol.org/resources/157/nodes

Looking at the code (app/controllers/nodes_controller.rb) I wonder if the resource_id parameter needs to be coerced to an integer?

@resource = Resource.find(params[:resource_id].to_i)

as opposed to

@resource = Resource.find(params[:resource_id])

? I don't know. This could easily be a deficit in my understanding.

DH draft version 1.2 uses <core> which isn't supported by harvester

The Darwin Core Archive version 1.0 documentation that I found describes and elements and a few others that don't seem to be supported by the harvester. For an example in the EOL opendata archive, see version 1.2 of the dynamic hierarchy.

Is this intentional? Are DwCA meta.xml files supposed to be run through a preprocessor before being deposited in opendata? Or is this just a feature that isn't yet supported by the harvester?

(I haven't tried this, I just make a conclusion based on the code in app/models/resource/from_meta_xml.rb.)

Broken links

To reproduce:

• go to http://content.eol.org/formats/2722
• try to follow any of the links under 'fields' (e.g. http://content.eol.org/formats/2722/fields/34283/edit )

Result: "The page you were looking for doesn't exist."

Either this is a bug (I should see something, even if I can't edit anything because I'm unauthorized) or the error message is wrong (if other people can edit, then it can't be the case that the page doesn't exist.

Missing resource metadata

The content repository's id for GloBI is supposed to be 20. If I look at content.eol.org/resources/20 I can see the content resource just fine. But if I look at
https://content.eol.org/resources.json?per_page=10000
there are many resources, but none with id 20. And:
https://content.eol.org/resources/20.json
doesn't work. How do I get machine-readable metadata for this resource, the same stuff I see in the HTML version? (in particular the opendataUrl field?)