dspace-labs / dspace-docker-images Goto Github PK
View Code? Open in Web Editor NEW**(OBSOLETE)** Repository for DSpace 4.x and 5.x Docker images. NO LONGER MAINTAINED
Home Page: https://github.com/DSpace/DSpace#running-dspace-7-in-docker
**(OBSOLETE)** Repository for DSpace 4.x and 5.x Docker images. NO LONGER MAINTAINED
Home Page: https://github.com/DSpace/DSpace#running-dspace-7-in-docker
@pnbecker has embedded a sample Dockerfile in https://jira.duraspace.org/browse/DS-4160 to illustrate a bug in a specific version of Linux. I recommend that we implement something like this with the following conventions.
The goal should be to make the image produced from this dockerfile to be swap-able with the dspace/dspace images referenced in the DSpace Docker compose files.
This is a paid feature, we would need to present a case to the steering group for approval.
The cost should be modest.
export DSPACE_SRC=$(pwd)
cd DSpace-Docker-Images/docker-compose-files/dspace-compose
docker-compose -p d7 -f docker-compose.yml -f d7solr.override.yml -f src.override.yml build
docker-compose -p d7 -f docker-compose.yml -f d7solr.override.yml -f src.override.yml up -d
Ideally, I would like to use the same Docker Compose file to start up a DSpace 5, a DSpace 6 and a DSpace 7 instance of DSpace.
In order to do this, I would like to manage separate volumes for the database and the dspace-install directories and intelligently manage those volumes.
I wounder if an ENV variable could be introduced when setting the volume names in the Docker Compose file to assist with the management of this information.
This will require the following changes
If so, there volumes would already contain sample users, communities, collections, items, and bitstreams.
These files would provide a quick restart point for someone deploying a new container.
These volumes would need to be tagged to specific DSpace releases.
When the "dspace version" command is run, "SCM revision" and "SCM branch" are set. These values are not getting set in our Docker builds. I wonder if there is a way to pass these values through environment variables. Do you know what mechanism grabs these values?
Kim Shepherd [12:56 PM]
@terrywbrady the file that holds the vars is dspace-api/src/main/resources/scm.properties
branch = ${scmBranch}```
and it seems to be maven that populates it, as the `target` version in dspace-api/target/classes/scm.properties looks like this
```revision = 929312d7e8728d33d3425e2b106c444444ea0468
branch = DS-4136_oai_import_master```
so, not sure how docker is missing out.. the maven build is fairly straight forward isn't it?
(you can see the full class at `org.dspace.app.util.Version`
Mark Wood [1:13 PM]
Yes, Maven creates that file for us. The resource plugin fills in values from the effective POM while copying the template over to target. It's not easy to see, but see dspace-api/pom.xml (edited)
Likewise Ant write its version into 'ant.properties'.
Kim Shepherd [1:54 PM]
hm, interestingly, though my docker-dev-compose instance works ok (it's an old one) for `dspace version`, it lists each webapp in Applications at least 10 times.. i wonder if there's some caching in the UtilServiceFactory stuff, or some filtering/deduplication needed in the results for that...
Mark Wood [1:56 PM]
That's an old problem that I need to fix someday. The duplicates for webapp X *should* all be deleted when X starts, but for some reason they persist.
https://jira.duraspace.org/browse/DS-3000
Terry Brady [2:06 PM]
Since the published images are built on Dockerhub, those values are probably not set. I will add an issue to our Docker project to find a way to set these values.
Based on this report, I would expect a locally built instance to possibly have the branch.
The config framework introduced in DSpace 6 allows environment variables to override config values set in dspace.cfg or local.cfg.
I did some testing of this with one of our Docker compose files.
The environment variable did get passed into tomcat and worked as expected.
When running a command line task using docker exec dspace bash
, the environment variables were not carried into the shell environment.
If so, I would imagine that we would build and re-publish images each time that there is a major DSpace release.
Once a user has deployed a container, the user would have the option to re-deploy the code within the container.
If so, where should the source for these scripts reside?
How should these scripts be invoked? (From the host computer, or within a bash window in Docker)?
To enable registration emails
Make this invokable from dockerhub dspace/dspace
How could we re-use the existing published images for DSpace?
The compose file will bring in the proper database image.
We will need to insert the right Oracle properties into local.cfg before tomcat is started.
Could we create a volume to override local.cfg?
This should allow us to collapse the multiple Dockefiles that we have on each branch (jdk7 vs jdk8) or (test vs no test).
In DSpace 4, the indexer does not run automatically. This should be added to the startup script.
o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: no field name specified in query and no default specified via 'df' param
Create a docker compose file that will run 3 containers
Re-use as much of the DSpace 6 docker file as possible
Next up: As you may have seen, the team at CNRI has reached out to help us upgrade our Handle server: https://jira.duraspace.org/browse/DS-4205 and DSpace/DSpace#2394
I discovered/fixed a bug in the base image.
We currently save /dspace/solr to the solr volume.
Because this contains both the conf and data directories for each repo, this could interfere with updates to solr conf files.
Our docker compose files should persist each solr data directory into its own repo.
We currently maintain a separate docker compose file to support ingest of content into a DSpace instance.
Rather than using an alternate compose file, I recommend the following changes.
In the DSpace Dockerfile, provide one (or more) optional entrypoint scripts that provide the functionality that currently resides here:
https://github.com/DSpace-Labs/DSpace-Docker-Images/tree/master/add-ons/mount_ingest_tools
Consider installing these to /entrypoint
# Uncomment the following line to trigger the init of DSpace solr repos
entrypoint: "/entrypoint/createAdmin.sh"
Call localhost to hit services
Verify expected status codes from requests
Hello,
I have installed the DSpace7 using this image. Its showing me REST and angular interface on browser.
When i try to sign in/sign up or forgot password, its just come back to home page. There are errors in the console. I am uploading the image as well.
I have installed the docker on window 10 professional. i am using the docker machine ip not the localhost.
PORT=8081 docker-compose ... up -d
(-e PORT=8081 is not supported)
We could use the docker-compose file for DSpace 6.
Since pgcrypto is not needed, we could create a separate compose file that uses the postgres image directly.
Just noting that the ingetAIP.sh
script currently does its own parsing/looping over Community/Collection/Item AIPs (instead of restoring from a Site-level AIP), e.g.
https://github.com/DSpace-Labs/DSpace-Docker-Images/blob/master/add-ons/mount_ingest_tools/ingestAIP.sh
We may want to consider updating to do an import starting at the SITE level AIP (this is the same command we use on demo.dspace.org to do a full site "reset"):
/dspace/bin/dspace packager -u -r -a -f -t AIP -e [email protected] -o skipIfParentMissing=true -o createMetadataFields=true SITE@[handle].zip
The extra benefit here is that this will also create any permissions/users/groups that are stored in the SITE AIP. It also includes flags to auto-create any missing/custom metadata fields, and to skip any "parent not found errors".
The full reset script for demo.dspace.org site is in GitHub here (in case it's useful to see the entire restore process run on demo): https://github.com/DSpace-Labs/demo.dspace.org-site/blob/master/scripts/linux/reset-dspace-content#L234
This will depend on https://jira.duraspace.org/browse/DS-4167
Add the call to rdfize in the script
Also update tutorials
When we initially added Dockerfiles to the DSpace repo, there was a desire to place them in dspace/src/main/docker.
The DockerHub classic build system did not work with this approach.
With the new dockerhub build system, this might be possible
Having followed the instructions and set DSPACE_VER=dspace-4_9
and $DPROJ=d4
having run : docker-compose -p $DPROJ up
the database seems to initialise and lots of logging is coming from catalina, but then:
dspace_1 | INFO [org.dspace.servicemanager.DSpaceKernelImpl] DSpace kernel startup completed in 1594 ms and registered as MBean: org.dspace:name=c778448b-c238-4a75-bf16-ee78694931af,type=DSpaceKernel
dspace_1 | INFO [org.dspace.core.ConfigurationManager] Loading provided config file: /dspace/config/dspace.cfg
dspace_1 | INFO [org.dspace.core.ConfigurationManager] Using dspace provided log configuration (log.init.config)
dspace_1 | INFO [org.dspace.core.ConfigurationManager] Loading: /dspace/config/log4j.properties
dspacedb_1 | 2018-09-18 13:47:14.238 UTC [85] ERROR: syntax error at or near ")" at character 22
dspacedb_1 | 2018-09-18 13:47:14.238 UTC [85] STATEMENT: INSERT INTO webapp ( ) VALUES ( ) RETURNING null
dspace_1 | Loading catalog: file:/usr/local/tomcat/work/Catalina/localhost/xmlui/cocoon_xml_resolver_entities/catalog
dspace_1 | 18-Sep-2018 13:47:16.085 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/xmlui] has finished in [8,824] ms
dspace_1 | 18-Sep-2018 13:47:16.085 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/jspui]
dspace_1 | 18-Sep-2018 13:47:22.138 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
dspace_1 | log4j:WARN No appenders could be found for logger (org.dspace.servicemanager.DSpaceKernelInit).
dspace_1 | log4j:WARN Please initialize the log4j system properly.
dspace_1 | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
dspace_1 | INFO: Loading provided config file: /dspace/config/dspace.cfg
dspace_1 | INFO: Using dspace provided log configuration (log.init.config)
dspace_1 | INFO: Loading: /dspace/config/log4j.properties
dspacedb_1 | 2018-09-18 13:47:23.902 UTC [87] ERROR: syntax error at or near ")" at character 22
dspacedb_1 | 2018-09-18 13:47:23.902 UTC [87] STATEMENT: INSERT INTO webapp ( ) VALUES ( ) RETURNING null
dspace_1 | 18-Sep-2018 13:47:23.970 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/jspui] has finished in [7,885] ms
dspace_1 | 18-Sep-2018 13:47:23.970 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/rest]
dspace_1 | 18-Sep-2018 13:47:29.545 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
dspace_1 | INFO: Loading provided config file: /dspace/config/dspace.cfg
dspace_1 | INFO: Using dspace provided log configuration (log.init.config)
dspace_1 | INFO: Loading: /dspace/config/log4j.properties
dspace_1 | 18-Sep-2018 13:47:30.811 INFO [localhost-startStop-1] com.sun.jersey.api.core.PackagesResourceConfig.init Scanning for root resource and provider classes in the packages:
dspace_1 | org.dspace.rest
dspace_1 | 18-Sep-2018 13:47:30.834 INFO [localhost-startStop-1] com.sun.jersey.api.core.ScanningResourceConfig.logClasses Root resource classes found:
dspace_1 | class org.dspace.rest.HandleResource
dspace_1 | class org.dspace.rest.BitstreamResource
dspace_1 | class org.dspace.rest.CommunitiesResource
dspace_1 | class org.dspace.rest.RestIndex
dspace_1 | class org.dspace.rest.ItemsResource
dspace_1 | class org.dspace.rest.CollectionsResource
dspace_1 | 18-Sep-2018 13:47:30.834 INFO [localhost-startStop-1] com.sun.jersey.api.core.ScanningResourceConfig.init No provider classes found.
dspace_1 | 18-Sep-2018 13:47:30.898 INFO [localhost-startStop-1] com.sun.jersey.spi.spring.container.servlet.SpringServlet.getContext Using default applicationContext
dspace_1 | 18-Sep-2018 13:47:30.902 INFO [localhost-startStop-1] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.17.1 02/28/2013 12:47 PM'
dspace_1 | 18-Sep-2018 13:47:31.532 WARNING [localhost-startStop-1] com.sun.jersey.spi.inject.Errors.processErrorMessages The following warnings have been detected with resource and/or provider classes:
dspace_1 | WARNING: A sub-resource method, public org.dspace.rest.common.Community[] org.dspace.rest.CommunitiesResource.list(java.lang.String), with URI template, "/", is treated as a resource method
dspace_1 | WARNING: A sub-resource method, public org.dspace.rest.common.Collection[] org.dspace.rest.CollectionsResource.list(java.lang.String,java.lang.Integer,java.lang.Integer), with URI template, "/", is treated as a resource method
dspace_1 | 18-Sep-2018 13:47:31.540 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/rest] has finished in [7,570] ms
dspace_1 | 18-Sep-2018 13:47:31.556 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
dspace_1 | 18-Sep-2018 13:47:31.585 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
dspace_1 | 18-Sep-2018 13:47:31.589 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 49792 ms
dspacedb_1 | 2018-09-18 13:47:49.618 UTC [88] ERROR: relation "community" does not exist at character 15
dspacedb_1 | 2018-09-18 13:47:49.618 UTC [88] STATEMENT: SELECT * FROM community WHERE NOT community_id IN (SELECT child_comm_id FROM community2community) ORDER BY name
Something goes wrong in the database. There are three different errors logged from the database (dspacedb_1), the first one repeated twice.
http://localhost:8080/jspui/ - has an Internal error page
The primary advantage of this approach is that it would require one fewer volume file to test.
On the other hand. it would make it more complicated to load up a large scale database volume.
Example: DSpace/DSpace#2379
The current dockerfile has made the entire dspace-install directory into a volume.
This can be convenient because it allows you to stop and start a container and the state of the solr directories is retained.
Because the installed code is in a volume, if you swap versions (dspace-5_x to dspace-5.9), the code is not updated.
Consider creating separate volumes for the assetstore and for solr. Let the code remain within the image.
When I change admin variables by e.g.
$ cat .env
[email protected]
ADMIN_PASS=admin2
the logging still accept the original one not those.
Eliminate the test_database check
Determine the minimum setting needed to run DSpace 6 and DSpace 7.
Update the windows and macos install pages with these details.
Terry Brady [7:52 AM]
Since I have never seen Sword in action, it would be interesting to create a docker compose file that runs a sword client alongside DSpace. Perhaps both a v1 and v2.
Does such a client exist?
Tim Donohue [7:53 AM]
DSpace XMLUI has a built in SWORDv1 client. But, yes, I agree
I'm not aware of many SWORDv2 clients -- there surely are some, but I think that's part of the reason why SWORDv1 is still in wide use
James Creel [7:54 AM]
The use case for the XMLUI SWORDv1 client isn't super strong, as I recall. You could also do export/import, harvesting, other things.
But Vireo uses SWORDv1
Tim Donohue [7:54 AM]
SWORDv2 implementations: http://swordapp.org/sword-v2/sword-v2-implementations/
James Creel [7:54 AM]
Just a curl request and a METS structure sitting next door could suffice for demonstration
Tim Donohue [7:55 AM]
Yes, technically you can send content to either SWORDv1 or SWORDv2 via curl
. We have examples of both in our DSpace Docs
Mark Wood [7:55 AM]
I'm beginning to imagine a bunch of projects standing around, all thinking "SWORDv1 should be deprecated," but none of them wanting to be the first to do so.
Tim Donohue [7:56 AM]
Example curl
commands for SWORDv1 are in our docs here: https://wiki.duraspace.org/display/DSDOC5x/SWORDv1+Server#SWORDv1Server-DeposittoSWORDServer
Mark Wood [7:56 AM]
Meanwhile ResourceSync is hovering in the background, looking for an opening.
Terry Brady [7:56 AM]
I started a github issue. I will copy this notes there. #73
Tim Donohue [7:56 AM]
Example curl
commands for SWORDv2 are in our docs here: https://wiki.duraspace.org/display/DSDOC6x/SWORDv2+Server#SWORDv2Server-DeposittoSWORDv2Server
I was closely watching my Docker logs on startup (docker logs -f dspace
) and noticed that it threw a syntax error on this line:
https://github.com/DSpace-Labs/DSpace-Docker-Images/blob/master/add-ons/dspace-docker-tools/ingestAIP.sh#L76
I believe the variable either needs to be surrounded by quotes, or we might need to use -eq
. I haven't had a chance to dig into the correct syntax yet though.
Would it be possible to make the instructions so simple that any repository manager with a local Docker instance could test a specific version of the code?
When travis performs the automated build of code for a specific PR, could the built assets be retrievable for quick testing? Or, would the download of built assets be so slow that it would be advisable just to build locally?
If a user must build locally, what can we do to make it as easy as possible?
This would be especially useful if we intend to deploy pre-built images for Docker.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.