bioatlas/ala-docker

Dockerized version of Atlas of Living Australia components

ala-docker's Issues

Upgrade solr to 7.3

Due to this vulnerability with more details here perhaps it would be good to upgrade to solr 7.3 for external facing solr services.

Scatterplot download not working

I produce the same scatterplot as described in issue #53. Then I try to download the data by pressing the download button in the left margin. The download appears to produce an empty zip file; nothing shows up if I do "tar tvf" in the unix command window of my Mac, and the file expands to nothing. When I try to open the file from the Firefox download dialog with the Archive utility, I get the following warning message: "Kan inte expandera "ScatterplotCreate (3662372).zip i "Hämtade filer". (Fel 1 – Åtgärden är inte tillåten.)". I am trying to past a partial screen shot below. The same bug occurs if I try to download the scatterplot data in ALA, so it appears to be an upstream bug. I am using Firefox 72.0.1 on a Mac running macOS 10.14.6 Mojave.

Use .env for credentials for all services

Make sure all services use .env files for credentials and provide an automated way to generate strong credentials

Trouble with CAS locally

Hello!

I'm trying to add CAS to my cluster and im testing it locally. this is how my docker-compose looks like:

version: '3.7'

services:
  cas:
    image: cas
    ports:
      - 8080:8080
      - 8089:8443
    volumes:
      - ./config/cas.properties:/tmp/cas.properties
    command: /bin/ash -c  "sleep 80 && envsubst < /tmp/cas.properties > /data/cas/config/cas.properties && catalina.sh run"
    links:
      - mysqldbcas
    depends_on:
 
      - mysqldbcas

  mysqldbcas:
    image: mysql:5.7
    env_file: ./env/envcas.template
    container_name: mysqldbcas
    volumes:
      - ./db:/docker-entrypoint-initdb.d
      - db_data_mysqldbcas:/var/lib/mysql

  userdetails:
    image: userdetails
    container_name: userdetails
    domainname: localhost
    env_file: ./env/.envcas
    volumes:
      - ./config/userdetails-config.properties:/tmp/userdetails-config.properties 
    command: /bin/ash -c "sleep 80 && envsubst < /tmp/userdetails-config.properties > /data/userdetails/config/userdetails-config.properties && catalina.sh run"
    depends_on:
 
      - mysqldbcas

volumes:
  cas_data:
  mongodb_data:
    driver: local
  db_data_mysqldbcas:

i've created the images with the Dockerfiles. I dont know how to correctly setup the conf files. My CAS container keeps showing that

cas_1 | 2019-04-17 21:53:03,369 INFO [org.jasig.cas.services.DefaultServicesManagerImpl] - <Loaded 1 services.> cas_1 | 2019-04-17 21:55:02,991 INFO [org.jasig.cas.services.DefaultServicesManagerImpl] - <Reloading registered services.> cas_1 | 2019-04-17 21:55:02,992 DEBUG [org.jasig.cas.services.DefaultServicesManagerImpl] - <Adding registered service ^(https?|imaps?)://.*> cas_1 | 2019-04-17 21:55:02,992 INFO [org.jasig.cas.services.DefaultServicesManagerImpl] - <Loaded 1 services.> cas_1 | 2019-04-17 21:57:02,991 INFO [org.jasig.cas.services.DefaultServicesManagerImpl] - <Reloading registered services.> cas_1 | 2019-04-17 21:57:02,991 DEBUG [org.jasig.cas.services.DefaultServicesManagerImpl] - <Adding registered service ^(https?|imaps?)://.*> cas_1 | 2019-04-17 21:57:02,992 INFO [org.jasig.cas.services.DefaultServicesManagerImpl] - <Loaded 1 services.>

And i cant accesse cas throw localhost:8089/cas/login

bioatlas/ala-geoserver:v0.4 not found when doing "make pull" from develop branch

Has this one not been pushed to DH?

cassandra db image

the ddb container from cassandra db image seem to lack the occ keyspace (schema created during building of image) when started from make up (docker-compose up).
However creating container from the per-built image using docker exec -it has the schema.

Evaluate authentication mechanism

Test the authentication mechanism to make sure it works well with the various components and uses https etc.

Self-signed certs trouble

Using commercial certs is different from using self-signed certs. It seems that some of the Java based services may not trust the ca.pem-file for the CA "test-ca" we use by default - but the trust chain would be there if using a commercial cert (which we have been doing internally so far).

So we may need to install the CA cert into some truststores first for getting this to work on initial startup.... pretty annoying.... Let's Encrypt certs doesn't seem to be an option. It would be nice to have https with self-signed certs from the get-go...

Ideas?

Some of these links may provide more detail:

https://rootsquash.com/2016/05/02/inserting-certificates-into-java-keystore-via-dockerfile/
https://thomas-leister.de/en/how-to-import-ca-root-certificate/
https://stackoverflow.com/questions/46923699/ssl-client-certs-with-docker-container
anapsix/docker-alpine-java#27

Refactor for shared data volumes

Refactor the setup to use shared data volumes, there is documentation here which might be relevant for the docker-compose.yml version 2 format: https://docs.docker.com/compose/compose-file/#/volume-configuration-reference

Troubles with name resolution when running ala-docker on networks that block certain DNS traffic...

If running on a corporate or guest network that is strict and blocks DNS traffic and forces usage of specific DNS servers, containers may not be able to reach out to the Internet using default settings.

A possible workaround could be to modify and hard-code settings for the local name resolution servers in the docker-compose.yml file within the dnsmasq section.

On some Linux OSes this command can be used to find the DNS servers in use:

nmcli dev show | grep DNS

SO post on finding what DNS servers are in use

for example the startup command for dnsmasq could be edited to use specific hard coded name DNS server to look like this assuming DNS ips in the corporate network are 130.238.96.1 and 130.238.98.11:

command: --log-facility=- --server=127.0.0.11 --server=130.238.96.1 --server=130.238.98.11 --server=193.10.57.11 --server=8.8.8.8 --server=172.16.0.72 --server=172.16.0.7 --neg-ttl=3600 --cache-size=1000 --max-cache-ttl=3600 --min-cache-ttl=3600 --all-servers

Dashboard state province

GDay, gentlemen

In Dashboard, there is a graph called "Records by state and territory". It returns me "Not Supplied" and "Other" with 100% of records as Not Supplied. I do not know what is missing to have that graph showing the states and records by state.
https://dashboard.ala-hml.vertigo.com.br/dashboard/
My dashboard configs:

serverName=https://dashboard.ala-dev.vertigo.com.br
contextPath=/dashboard
security.cas.casProperties=casServerLoginUrl,serverName,centralServer,casServerName,uriFilterPattern,uriExclusionFilter,authenticateOnlyIfLoggedInFilterPattern,casServerLoginUrlPrefix,gateway,casServerUrlPrefix,contextPath

security.cas.uriFilterPattern=
security.cas.uriExclusionFilterPattern=/images.*,/css.*,/js.*,/less.*


biocache.baseURL=https://biocache-service.ala-dev.vertigo.com.br/biocache-service
biocache.webappURL=https://ala-hub.ala-dev.vertigo.com.br/ala-hub
spatial.baseURL=https://portal-espacial.ala-dev.vertigo.com.br/ws
alahub.baseURL=https://ala-hub.ala-dev.vertigo.com.br/ala-hub
collectory.baseURL=https://collectory.ala-dev.vertigo.com.br/collectory
volunteer.baseURL=
images.baseURL=https://images.ala-dev.vertigo.com.br/images
userDetails.baseURL=http://cas.ala-dev.vertigo.com.br/cas/login

bie.baseURL=https://ala-bie.ala-dev.vertigo.com.br/ala-bie/
bie.searchPath=/search
bie.webappURL=https://ala-bie.ala-dev.vertigo.com.br/ala-bie/

bie.index.baseURL=https://bie-webservice.ala-dev.vertigo.com.br/bie-index

ala.skin=ala
ala.baseURL=https://ala-dev.vertigo.com.br
headerAndFooter.baseURL=https://ala-dev.vertigo.com.br

skin.layout=ala-main
skin.fluidLayout=true
skin.homeUrl=https://ala-dev.vertigo.com.br
skin.orgNameLong=SiBBr
skin.orgNameShort=SiBBr
skin.favicon=https://ala-dev.vertigo.com.br/img/sibbr.ico

Image Service losing image id data.

Me again, gentlemen.

So, if the data inside the /data/images/store/* is lost the image service will only show empty thumbnails e the meta data of the image files. There is a way, like a backend command, to recreate or reload all info into the image service again without creating duplicate artifacts?

Use Alpine images where possible

./baptize does not work on Mac

Steins-MBP:ala-docker steinarildhoem$ make theme-dl
Downloading bioatlas wordpress theme...
Steins-MBP:ala-docker steinarildhoem$ ./baptize.sh
grep: warning: recursive search of stdin
^C
Steins-MBP:ala-docker steinarildhoem$

Works ok on linux - no-ssl branch

Regions Layer Indexing

G'Day Gentleman,

I'm trying to deploy regions service. It looks like i can load the layers, yet when i try to filter the species in a selected area it's returning:

Error 500: Cannot invoke method getAt() on null object
Servlet: grailsDispatcherServlet
URI: /regions/region/showGroups
Exception Message: Cannot invoke method getAt() on null object
Caused by: Cannot invoke method getAt() on null object
Class: UrlMappingsInfoHandlerAdapter
At Line: [90]
Code Snippet:

It seems that i did not the right load of the layers indexing occurrences with it on Geoserver, wich makes me ask How is the right way to load layers in order to plot correctly in regions?
In case you gentlemen want to see it for yourselves here is the link

Develop branch cannot start the spatialhub due to logback classpath issue

The spatialhub does not start due to a SEVERE error, the log says "LoggerFactory is not a Logback LoggerContext but Logback is on the classpath", this may be related to transitive dependencies, see https://stackoverflow.com/questions/30792268/loggerfactory-is-not-a-logback-loggercontext-but-logback-is-on-the-classpath#42641450

manifest for bioatlas/ala-imageservice:v0.5 not found

docker-compose.yml & Makefile seem to reference unavailable image versions

Pulling imageservice (bioatlas/ala-imageservice:v0.5)...
ERROR: manifest for bioatlas/ala-imageservice:v0.5 not found: manifest unknown: manifest unknown
Makefile:251: recipe for target 'up' failed
make: *** [up] Error 1

Similar error for imagestore:v0.4

Load Uptime with preset configuration for the services

Trouble implementing edited nameindex

Hi,
I've added a record to the Dyntaxa-index, and get an exact match in the new image:
docker run --rm -it bioatlas/ala-dyntaxaindex:v0.3-Bifid-sh nameindexer -testSearch "Bifidobacteriaceae"
...
Match type: exactMatch

I have restarted services, and my new image seems to be used:
docker inspect nameindex
"Image": "bioatlas/ala-dyntaxaindex:v0.3-Bifid-sh",

I have deleted, re-loaded, re-processed and re-indexed my occurrence records.
Yet, my Bifidobacteriaceae-occurrence is matched as Bacteria only, i.e. my index seems not have been used for indexing after all.

What am I doing wrong?

Develop branch: upload of layer runs into size limit

While uploading one of the layers, a size limit in the spatialservice exceeding 524288000 (bytes?) prevented the upload from succeeding.

Log messages:

webserver | 2019/09/24 11:12:22 [warn] 6#6: *3835 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000084, client: 172.18.0.39, server: beta.bioatlas.se, request: "POST /spatial-service/manageLayers/upload HTTP/1.1", host: "beta.bioatlas.se", referrer: "https://beta.bioatlas.se/spatial-service/manageLayers/uploads?error=MALFORMED+-+see+logs+for+details"

spatialservice        | [SPATIAL-SERVICE] 11:12:25.893 StackTrace : Full Stack Trace:
spatialservice        | 
spatialservice        | org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is java.lang.IllegalStateException: org.apache.tomcat.util.http.fileupload.FileUploadBase$SizeLimitExceededException: the request was rejected because its size (579193424) exceeds the configured maximum (524288000)

biocache Download Issue

Hey gentlemen, it's me again.
So, now im rtrying to make the Download module in biocache.ala.org.au/download work.

looks like i cannot provide such resource because it depends on the DOI module (doi.ala.org.au). So, will i really need to set my ow DOI or there is a way to me to use other one's DOI to be able to make the download option work?

Consistent use of .travis.yml for all services where this is relevant

The URL_BIOCACHE_SERVICE needs to be built from sources.

bioatlas docker images missing from docker hub

Great work on dockerising the ALA platform. I am trying to run it in my development environment but images referred to in the docker-compose.yml file do not exist in docker hub. Is there an alternative image I can use? https://hub.docker.com/search/?isAutomated=0&isOfficial=0&page=1&pullCount=0&q=bioatlas&starCount=0

On develop branch, some Java 8 services warn about memory opts

For example the geonetwork service and also the ones below:

spatialhub            | OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
regions               | OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0

Should this OPT be removed?

Use https everywhere

Configure all services to use https

solr image

the solr image when created does not create the biocache core. the other 2 core bie and bie-offline are created and loaded.

Develop branch: spatial-batch upload succeeds but layer view then fails

After uploading a set of geodata, I click a link to view one of the layers:

https://beta.bioatlas.se/spatial-service/layers/view/more/naturgeografiska_regioner

It produces a log entry:

spatialservice        | Caused by: java.lang.NullPointerException: Cannot invoke method split() on null object
spatialservice        | 	at gsp_spatial_service_layershow_gsp$_run_closure2.doCall(gsp_spatial_service_layershow_gsp.groovy:109)

It may be happening around here:

https://github.com/AtlasOfLivingAustralia/spatial-service/blob/master/grails-app/views/layer/show.gsp#L120

Indicating that there is no layer.metadatapath or rather it is null. Does this come from content or a config?

Make use of dockers embedded DNS

It would be nice if the setup uses Docker embedded DNS capabilities. If we have container 1 and container 2 (c1, c2) running in the host, can c2 ping c1 under the name "gbifsweden.se" resolved locally by the embedded Docker DNS capabilities?

The container needs to "see" and resolve the hostname used. That can be tested with "ping" for example. If this doesn't work from inside the container, then there is a problem.

Perhaps https://docs.docker.com/engine/userguide/networking/default_network/configure-dns/ can provide instructions on how to make the container see itself under a given hostname?

What about --link or --hostname and the docker compose equivalents, whatever those might be?

Perhaps extra_hosts or external_links? It seems you can do this in docker-compose.yml:

domainname: foo.com
hostname: foo

Details: https://docs.docker.com/compose/compose-file/#/cpu-shares-cpu-quota-cpuset-domainname-hostname-ipc-mac-address-mem-limit-memswap-limit-privileged-read-only-restart-shm-size-stdin-open-tty-user-working-dir

make dotfiles - does not work on Mac

output:

Steins-MBP:ala-docker steinarildhoem$ sudo make dotfiles
Password:
#echo "# Make this unique, and don't share it with anybody.\n# This value was autogenerated." > secrets
#rm -f secrets env/{.envapi,.envcollectory,.envimage,.envlogger}
printf "export SECRET_MYSQL_ROOT_PASSWORD=%b\n"
$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 50) >> secrets
tr: Illegal byte sequence
printf "export SECRET_MYSQL_PASSWORD=%b\n"
$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 50) >> secrets
tr: Illegal byte sequence
printf "export SECRET_POSTGRES_PASSWORD=%b\n"
$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 50) >> secrets
tr: Illegal byte sequence
printf "export SECRET_MIRROREUM_PASSWORD=%b\n"
$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 50) >> secrets
tr: Illegal byte sequence
bash -c ". secrets && envsubst < env/envapi.template > env/.envapi"
bash -c ". secrets && envsubst < env/envcollectory.template > env/.envcollectory"
bash -c ". secrets && envsubst < env/envimage.template > env/.envimage"
bash -c ". secrets && envsubst < env/envlogger.template > env/.envlogger"
bash -c ". secrets && envsubst < env/envmirroreum.template > env/.envmirroreum"
bash -c ". secrets && envsubst < env/envwordpress.template > env/.envwordpress"
rm -f secrets

output from image mysqldb:
error: database is uninitialized and password option is not specified
You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD and MYSQL_RANDOM_ROOT_PASSWORD
... might be related to above problem

Develop branch has issue with geonetwork service

The geonetwork service on develop branch has some errors in the log, this is an excerpt:

WARN  [geonetwork.databasemigration] -       Database does not contain any version information. Check that the database is a GeoNetwork database with data. The database is probably empty, no migration required.

WARN  [jeeves.config.springutil.JeevesApplicationContext] - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'MetadataRegionsDAO': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.fao.geonet.kernel.search.SearchManager org.fao.geonet.services.region.MetadataRegionDAO.searchManager; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'SearchManager' defined in URL [jar:file:/usr/local/tomcat/webapps/geonetwork/WEB-INF/lib/core-3.4.1-0.jar!/config-spring-geonetwork.xml]: Instantiation of bean failed; nested exception is java.lang.StackOverflowError

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'MetadataRegionsDAO': Injection of autowired dependencies failed; nested exception is

org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.fao.geonet.kernel.search.SearchManager org.fao.geonet.services.region.MetadataRegionDAO.searchManager; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'SearchManager' defined in URL [jar:file:/usr/local/tomcat/webapps/geonetwork/WEB-INF/lib/core-3.4.1-0.jar!/config-spring-geonetwork.xml]: Instantiation of bean failed; nested exception is java.lang.StackOverflowError

Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.fao.geonet.kernel.search.SearchManager org.fao.geonet.services.region.MetadataRegionDAO.searchManager; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'SearchManager' defined in URL [jar:file:/usr/local/tomcat/webapps/geonetwork/WEB-INF/lib/core-3.4.1-0.jar!/config-spring-geonetwork.xml]: Instantiation of bean failed; nested exception is java.lang.StackOverflowError

FATAL [jeeves.engine] -    Message   : Error creating bean with name 'MetadataRegionsDAO': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.fao.geonet.kernel.search.SearchManager org.fao.geonet.services.region.MetadataRegionDAO.searchManager; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'SearchManager' defined in URL [jar:file:/usr/local/tomcat/webapps/geonetwork/WEB-INF/lib/core-3.4.1-0.jar!/config-spring-geonetwork.xml]: Instantiation of bean failed; nested exception is java.lang.StackOverflowError

A "make pull" on the develop branch gives "access denied" to bioatlas/ala-sds

The make pull errors out with pull access denied on bioatlas/ala-sds

Error response from daemon: pull access denied for bioatlas/ala-sds, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
make: *** [Makefile:284: pull] Error 1

Cannot start service proxy (on Mac)

dag$ make up
Starting services...
Removing nginx-proxy
psqldbimage is up-to-date
Recreating a4b16899e86b_nginx-proxy ... 
mysqldbwordpress is up-to-date
solr is up-to-date
mysqldblogger is up-to-date
mysqldbapi is up-to-date
Starting cassandradb                ... 
static is up-to-date
Starting wordpress                  ... 
Recreating a4b16899e86b_nginx-proxy ... error
Starting cassandradb                ... done
Starting wordpress                  ... done
Starting nameindex                  ... done
Starting imageservice               ... done
Starting loggerservice              ... done
Starting collectory                 ... done
Starting biocacheservice            ... 

ERROR: for a4b16899e86b_nginx-proxy  Cannot start service proxy: driver failed programming external connectivity on endpoint nginx-proxy (c3461adaf570cac0106885528646be562762a7e9d6da33066a8aefad2f307822): Error starting userland proxy: Bind fStarting biocacheservice            ... done
biocachehub is up-to-date
Starting biocachebackend            ... 
Starting biocachebackend            ... done

ERROR: for proxy  Cannot start service proxy: driver failed programming external connectivity on endpoint nginx-proxy (c3461adaf570cac0106885528646be562762a7e9d6da33066a8aefad2f307822): Error starting userland proxy: Bind for 0.0.0.0:80: unexpected error (Failure EADDRINUSE)
ERROR: Encountered errors while bringing up the project.
make: *** [up] Error 1

The API role

It seems that the webapi module and the apikey is the way to enable one module to interact with other. For example, in case to "import" some area into the spatial-hub map, i need to set the geoserver, the spatial service and hub apikeys. Am i right?

How could i define CAS and Application roles?

THere is a way to register an user in the userdetails with role association?

Scatterplot bug?

I tried to create a scatterplot on beta.bioatlas.se using the following procedure:

1. Click on spatial portal on beta.bioatlas.se
2. Select Tools -> Scatterplot - single
3. In box 2, Search for species: "Aglais" and click "Aglais Dalman, 1816"
4. Select environmental layers: Worldclim Annual Mean Temperature and Annual Precipitation
5. Click on Next

I got a new box opening up and a progress bar with some moving colors, but no plot was generated (at least not for a very long time...).

Add language translation files - gives Swedish language to several UIs

Background map tiles do not display correctly

I opened an issue here AtlasOfLivingAustralia/ala-collectory#15 with the Atlas people in Australia.

Trouble with biocache-backend

In the first place, thank you all for this work. It's really a big thing.
I'm having some problem trying to execute in my dev station the biocache components. Biocachebackend exits code 0 and no logs is displayed. I have a docker-compose.yml to build and run Cassandra, Solr, and biocaches images, already downloaded the .war and .zips but don't know what is happening in biocachebackend.

Load data

Loading datasets from the IPT and document any issues that appear in the process, troubleshoot and fix and problems discovered.

Load frontpage ("ghost") with content

Automatically bring up the webpage configured with content and good looks

Build process for localized ALA components (with Swedish language translations)

Missing "mongodump" utility in the Alpine mongo image

The Makefile for the Uptime image requires that the mongo database where content is persisted has a "mongodump" utility installed. Either this image is extended with that utility or we roll back to the official mongo container, which has it bundled.

Develop branch has an issue with the top navigation section for some services

Top navigation does not work for: explore by polygon (https://beta.bioatlas.se/ala-hub/search#tab_spatialSearch), explore by location (https://beta.bioatlas.se/ala-hub/explore/your-area), api (https://beta.bioatlas.se/api/), could this be a XHR issue?

THE JS Console reports "Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/"

Develop branch has issue with the "images" component

Got JS error Error: cannot call methods on autocomplete prior to initialization; attempted to call method 'https://beta.bioatlas.se/bie-index/search/auto.jsonp' when navigating to https://beta.bioatlas.se/images/

Scatterplot environmental layers bug

I tried to create a scatterplot on beta.bioatlas.se using the following procedure:

1. Click on spatial portal on beta.bioatlas.se
2. Select Tools -> Scatterplot - single
3. In box 2, Search for species: "Aglais" and click "Aglais Dalman, 1816"
4. Select environmental layers: Worldclim Annual Mean Temperature and Annual Precipitation

In step 4, all the environment layers that are not checked should be colored either red yellow or green. The selected ones are always colored in red I think. This works properly at ALA but in the bioatlas, the green color is usually missing completely. If you try the two layers indicated above, no layer is colored in green; a few are colored in yellow.