Integrate with third-party web services

[tsega]

Integrate with third-party web services

[alanorth]

Let me know when you need more information about CGSpace. I am the admin
there... There's no API but you can scrape XML pretty reliably.
On Apr 25, 2014 7:10 PM, "Tsegaselassie Tadesse" [email protected]
wrote:

Ability to integrate with third party web services; e.g. consume images
from Flickr, link documents via CGSpace.

Reply to this email directly or view it on GitHubhttps://github.com//issues/4
.

[tsega]

OK, I will certainly do that! However, this would be the last feature to be implemented for Phase 1 of the project (which is not that far away) but issues 1-3 are the first ones to be implemented.

[tsega]

@alanorth and @swebsheet now that we have the DAGRIS forms created and the DB migrated, I need your help in linking our two systems. I'm guessing you have every Bibliography entry that is already in DAGRIS, there are about 923 of them.

As discussed earlier, I'm expecting you would have a REST web service I can hit to fetch Bibliography data. I'm I right? Or do you have another way to do it?

Thanks

[alanorth]

Hey, not sure what kind of references in DAGRIS you're referring to? The
bibliography references are probably in CGSpace (cgspace.cgiar.org), for
which there is no REST yet. If they are indeed there then you'll be able to
use REST once I upgrade DSpace to version 4.1 in the next few weeks.
On Jun 19, 2014 8:20 PM, "Tsegaselassie Tadesse" [email protected]
wrote:

@alanorth https://github.com/alanorth and @swebsheet
https://github.com/swebsheet now that we have the DAGRIS forms created
and the DB migrated, I need your help in linking our two systems. I'm
guessing you have every Bibliography entry that already in DAGRIS as well,
there are about 923 of them.

As discussed earlier, I'm expecting you would have a REST web service I
can hit to fetch Bibliography data. I'm I right? Or do you have another way
to do it?

Thanks

Reply to this email directly or view it on GitHub
#4 (comment).

[tsega]

I don't if you've seen the database structure of DAGRIS but Bibliographies are one of the essential data sets it keeps track of.

Peter mentioned that these should not be redundantly keep in DAGRIS but should be referenced from DSpace (or is it CGSpace?). Hence, the requirement for the integration through a REST API. From your response I see that it will take a while before you can update the system to the latest version and for us to be able to use the REST interface.

At the moment this is not a top priority so it can wait until the update. There is another web service we want to make use of as well, Flickr for our images, so we will work on that in the mean time.

Thanks for the help.

[alanorth]

@tsega no, I hadn't seen the database structure yet. Now I see the bibliographies are essential.

DSpace is the software. CGSpace is what we call our website which is running DSpace (haha). Unfortunately, I'm not sure how these Bibliographies would link to CGSpace. Yes, we'll have REST soon, but I don't know what unique identifier you'd use to link the two... Peter should probably clarify where in CGSpace the data is... I am only the sysadmin for the CGSpace server really, so I'm not sure about the data.

[tsega]

Wow, that was one quick reply! Yes I believe Sisay, @swebsheet, is the person responsible for CGSpace, right? I will talk to him about it; I'm hoping he would get these comments as well. Thanks again for the assistance.

[alanorth]

btw, I still don't really understand what "bibliographies" are in this context... I searched for DAGRIS on CGSpace and found 48 items...

https://cgspace.cgiar.org/search

[tsega]

The Bibliographies are the source documents from which the DAGRIS data is extracted, particularly a Breed's trait information. The population data for breeds is also referenced from bibliography entries; at least that's what I could gather from the database table relationships.

In context of CGSpace, these Bibliographies (or any document sources) are maintained as document entries. For example, in DAGRIS we have species data on Yak, this data can be extracted from a document kept in CGSpace; Performance characteristics of the Yak in Nepal and its crosses with Mountain cattle could be one such source.

Currently, what is done in DAGRIS is that the information that you see on the linked page above is kept in the system database. This is makes the data in DAGRIS redundant to what is already in CGSpace.

Now, what Peter suggested was to keep the data at one place, CGSpace and for DAGRIS to just reference the document page. Therefore, when new entries for a Breed's trait are keyed in, all the data encoder needs to do is look for the source document in CGSpace and add its link as the source of the data. How to exactly find the document and link it is the real question we want to tackle on our side.

Hope this helps.

[alanorth]

Oops, forgot to respond. Yep, that makes sense.

I was basically confused because there's no unique id shared by an entry in DAGRIS and an entry in CGSpace. So basically a DAGRIS item's bibliography has to be search for (and found) in CGSpace and entered manually, eh?

[tsega]

@alanorth I wanted to try out the REST API for DSpace. I installed DSpace 4.2 locally but I was not able to configure it. The documentation under the REST: Deploying the DSpace REST API in your Servlet Container says,

The dspace-rest module is automatically configured to compile and build with DSpace 4.0, so a
mvn+ant process will create the webapp. To make it work in your environment, you would just need to
add a context entry for it in your servlet container. For example, in tomcat, one might alter
$CATALINA_HOME/conf/server.xml and add:

<Context path="/rest" docBase="/dspace/webapps/rest" allowLinking="true"/>

I have tried that but that doesn't work. Neither does placing that XML in context.xml. What should I do?

[alanorth]

Assuming this is Tomcat7 on Ubuntu, I put my contexts in /etc/tomcat7/Catalina/localhost/rest.xml:

<Context docBase="/path/to/dspace/webapps/rest"/>

Then the REST application will be deployed at http://localhost/rest. Otherwise, you could try the DSpace demo app (demo.dspace.org)?

[tsega]

@alanorth I have successfully installed and configured DSpace on my local machine and have checkout the demo site. Regarding the RestAPI, however, both installations redirect to an error page while trying to retrieve items list.

What could be the case?

[alanorth]

You MIGHT be in luck. We deployed 4.2 on DSpace Test last week and the REST
API is there. We're still doing some background processing/upgrading of
Solr data before we really start testing 4.2, but you can try looking at:

http://dspacetest.cgiar.org/rest

I'm getting on a plane, but this will be ready in the next few days.

Alan
On Oct 21, 2014 11:08 AM, "Tsegaselassie Tadesse" [email protected]
wrote:

@alanorth https://github.com/alanorth I have successfully installed and
configured DSpace on my local machine and have checkout the demo site
http://demo.dspace.org/. Regarding the RestAPI, however, both
installations redirect to an error page while trying to retrieve items
list http://demo.dspace.org/rest/items.

What could be the case?

Reply to this email directly or view it on GitHub
#4 (comment).

[tsega]

Same case with the new demo site, it redirect to the same error page. Is there anyone else I can ask, maybe in forum or something like that?

[alanorth]

Ok, I figured as much. We are still doing things on the back end with the
actual data in those communities, so I would wait a few days. The actual
upgrade from 3.x to 4.2 was the easy part, we had to re-process 20,000,000
Solr usage events into a new format for one of the custom modules we are
using, and that is still underway. Check back in a few days...

Alan
On Oct 21, 2014 2:35 PM, "Tsegaselassie Tadesse" [email protected]
wrote:

Same case with the new demo site http://dspacetest.cgiar.org/rest/items,
it redirect to the same error page. Is there anyone else I can ask, maybe
in forum or something like that?

Reply to this email directly or view it on GitHub
#4 (comment).

[alanorth]

Ok, try again... we just finished the indexing of items, now there is
actual data:

http://dspacetest.cgiar.org/rest/

Alan

On Tue Oct 21 2014 at 3:15:09 PM Alan Orth [email protected] wrote:

Ok, I figured as much. We are still doing things on the back end with the
actual data in those communities, so I would wait a few days. The actual
upgrade from 3.x to 4.2 was the easy part, we had to re-process 20,000,000
Solr usage events into a new format for one of the custom modules we are
using, and that is still underway. Check back in a few days...

Alan
On Oct 21, 2014 2:35 PM, "Tsegaselassie Tadesse" [email protected]
wrote:

Same case with the new demo site http://dspacetest.cgiar.org/rest/items,
it redirect to the same error page. Is there anyone else I can ask, maybe
in forum or something like that?

Reply to this email directly or view it on GitHub
#4 (comment).

[tsega]

Still not working, only the first four links work. I need the /items and /items/1 links to work. I have not been able to see it work in any of the DSpace demo instances.

[alanorth]

Ok, I had a quick play... it seems the items in the /rest link are just
suggestions, and not necessarily meant to work. /rest/items/1 won't
work, because handle "1" is not an item, it's a community.

Hmmm, seems pretty confusing. Apparently you have to get the ID first via
/rest/handle/, ie for handle 10568/43784:

http://dspacetest.cgiar.org/rest/handle/10568/43784

Then use the ID returned there to request the item using /rest/items/:

http://dspacetest.cgiar.org/rest/items/43311

The best docs I've found so far are from the README on github:
https://github.com/DSpace/DSpace/tree/master/dspace-rest

Play around with that and get back to me...

On Fri Oct 31 2014 at 10:02:54 PM Tsegaselassie Tadesse <
[email protected]> wrote:

Still not working, only the first four links work. I need the /items
http://dspacetest.cgiar.org/rest/items and /items/1
http://dspacetest.cgiar.org/rest/items/1 links to work. I have not been
able to see it work in any of the DSpace demo instances.

Reply to this email directly or view it on GitHub
#4 (comment).

[alanorth]

I just ran into this REST API explorer web app...

https://github.com/peterdietz/dspace-rest-play

Might be interesting to help understand how the REST API is implemented. There's a deployed version here:

http://dspace-rest-client-play.herokuapp.com/

[alanorth]

Ok, so this is pretty cool. I played with it a bit more and found some useful tidbits.

For example, given a collection 10568/3547, we can get a list of its items using the /handle/ endpoint: /handle/10568/3547

<collection>
<expand>parentCommunityList</expand>
<expand>parentCommunity</expand>
<expand>license</expand>
<expand>logo</expand>
<expand>all</expand>
<handle>10568/3547</handle>
<ID>313</ID>
<name>CCAFS Journal Articles</name>
<type>collection</type>
<copyrightText/>
<introductoryText>
<strong><a href="http://cgspace.cgiar.org/handle/10568/3547/browse?type=dateissued&sort_by=2&order=DESC&rpp=30">View all journal articles</a></strong>
</introductoryText>
<items>
<expand>metadata</expand>
<expand>parentCollection</expand>
<expand>parentCollectionList</expand>
<expand>parentCommunityList</expand>
<expand>bitstreams</expand>
<expand>all</expand>
<handle>10568/33749</handle>
<ID>34886</ID>
<name>
Integrated management practices significantly affect N2O emissions and wheat–maize production at field scale in the North China Plain
</name>
<type>item</type>
<archived>true</archived>
<lastModified>2014-11-04 18:25:11.469</lastModified>
<withdrawn>false</withdrawn>
</items>

From there you can get the handles for the individual items, then drill down to get their metadata (still using the /handle endpoint): /handle/10568/33749

<item>
<expand>metadata</expand>
<expand>parentCollection</expand>
<expand>parentCollectionList</expand>
<expand>parentCommunityList</expand>
<expand>bitstreams</expand>
<expand>all</expand>
<handle>10568/33749</handle>
<ID>34886</ID>
<name>
Integrated management practices significantly affect N2O emissions and wheat–maize production at field scale in the North China Plain
</name>
<type>item</type>
<archived>true</archived>
<lastModified>2014-11-04 18:25:11.469</lastModified>
<withdrawn>false</withdrawn>
</item>

It gets interesting when you expand the result set. For example, you can expand the items to get their full metadata, ?expand=all, or perhaps the bitstreams, ?expand=bitstreams. From there, getting their thumbnails is super easy: /bitstreams/22635/retrieve

[tsega]

Good this looks very promising, I'll work on this.