./docker/prepare-dev-environment/gradle-cache/run.sh -r -l gradle-cache.log && \
./docker/prepare-dev-environment/volumes/run.sh -r -l volumes.log && \
./docker/prepare-dev-environment/postgres/run.sh -r -l pg-anndata.log && \
./docker/prepare-dev-environment/postgres/run.sh -r -a -l pg-no-anndata.log && \
./docker/prepare-dev-environment/solr/run.sh -r -l solr.log &&
./build-and-deploy-webapp.shPoint your browser at http://localhost:8080/gxa/sc and voilà!
- Docker v20+ with the Compose plugin
- 60 GB of available storage for the following Docker volumes:
- Experiment files
- Bioentity properties (i.e. gene annotations)
- PostgreSQL
- SolrCloud and ZooKeeper
- Tomcat configuration files
Files written by Solr, PostgreSQL and Tomcat are kept in volumes which will be reused even if the containers are
removed (e.g. when running docker compose down). If you want to start afresh each script can remove all volumes
if you add the -r option . You can find the volume names used by each service in the volumes section of its Docker
Compose YAML file.
The full list of volumes is:
scxa_gradle-wrapper-distsscxa_gradle-ro-dep-cachescxa_atlas-data-bioentity-propertiesscxa_atlas-data-expscxa_atlas-data-scxa-expdesignscxa_postgres-11-pgdata-18scxa_postgres-11-pgdata-latestscxa_solrcloud-0-data-solr-8.7scxa_solrcloud-1-data-solr-8.7scxa_solrcloud-zookeeper-0-data-solr-8.7scxa_solrcloud-zookeeper-0-datalog-solr-8.7scxa_solrcloud-zookeeper-1-data-solr-8.7scxa_solrcloud-zookeeper-1-datalog-solr-8.7scxa_solrcloud-zookeeper-2-data-solr-8.7scxa_solrcloud-zookeeper-2-datalog-solr-8.7scxa_tomcat-conf
Clone this repository with submodules:
git clone --recurse-submodules https://github.com/ebi-gene-expression-group/atlas-web-single-cell.gitIf you have already cloned the project ensure it’s up-to-date:
git pull
git submodule update --remoteTo speed up builds and tests it is strongly encouraged to create a Docker volume to back a Gradle read-only dependency cache.
./docker/prepare-dev-environment/gradle-cache/run.sh -r -l gradle-cache.logIn order to run integration tests and a development instance of Single Cell Expression Atlas you will need a few Docker Compose volumes first. They will be populated with data that will be indexed in Solr and Postgres. Single Cell Expression Atlas needs all three of: file bundles in the volumes, Solr collections and Postgres data. This step takes care of the first requirement:
./docker/prepare-dev-environment/volumes/run.sh -r -l volumes.logYou can get detailed information about which volumes are created if you run the script with the -h flag.
This script, unless it’s run with the -r flag, can be interrupted without losing any data. The container mirrors
directories via FTP, and can resume after cancellation. It can be re-run to update the data in the volumes should the
contents of the source directories change. This is especially useful when experiments are re-analysed/re-annotated,
or the bioentity properties directory is updated after a release of Ensembl, WormBase ParaSite, Reactome, Gene
Ontoloy, Plant Ontology or InterPro.
To enable easy switching between anndata support and earlier versions of the database, the script sets the
environment variable SCHEMA_VERSION to either latest or 18
(latest version before anndata was
introduced),
respectively. The value is appended to the volume name mounted by the Postgres service.
Run the script twice and then choose later the appropriate version:
./docker/prepare-dev-environment/postgres/run.sh -r -l pg-anndata.log # anndata support
./docker/prepare-dev-environment/postgres/run.sh -a -r -l pg-no-anndata.log # no anndata supportTo run the Postgres service with support for anndata experiments:
SCHEMA_VERSION=latest \
docker compose --env-file ./docker/dev.env \
-f ./docker/docker-compose-postgres.yml \
upTo run the Postgres service without support for anndata experiments:
SCHEMA_VERSION=18 \
docker compose --env-file ./docker/dev.env \
-f ./docker/docker-compose-postgres.yml \
upThe Solr script can optionally be given a location for the Single Cell Expression Atlas ontology OWL file. Otherwise,
it will download the published scatlas.owl file in the EBI SPOT
repository. It will also generate an RSA keypair
to sign and verify Solr packages that you can keep
for reference and to sign other packages (or later versions of BioSolr). Be aware that Solr can store multiple keys, so
they are not strictly necessary; it is possible to generate a new keypair and store its public key every time you add a
package. Run the script with the -h flag for more details.
./docker/prepare-dev-environment/solr/run.sh -r -l solr.logYou may want to speed up the process by raising the value of the environment variable NUM_DOCS_PER_BATCH (L126 of the
run.sh script). On a fairly powerful laptop at the time of
writing 20,000 has been
found to be a reliable number via painstaking trail and error, but your mileage may vary. Ensure that there are no
errors in the script logs. Some tests may fail due to incomplete annotations; grep for DistributedUpdatesAsyncException in
particular, which signals a problem storing the document batch, which in turn stops processing the current file. If
found, try again with a lower value for NUM_DOCS_PER_BATCH.
Add or change the necessary species names and experiment accessions in the test-data.env file and rebuild the
development environment.
./execute-all-tests.sh 18 # For non-anndata DB
./execute-all-tests.sh latest # For anndata DB
./execute-single-test.sh TEST_NAME [ 18 ]
./debug-single-test.sh TEST_NAME [ 18 ]
./stop-and-remove-containers.shThe scxa-gradle service in docker/docker-compose-gradle.yml executes all tests and writes reports to
atlas-web-core/build and app/build in the host machine. It requires the SolrCloud service described earlier, and a
Postgres container with the following differences compared to the development service, scxa-postgres: it doesn’t use
named volumes to ensure the database is clean before running any tests, and its name (as well as the dependency
declared in docker-compose-gradle.yml) has been changed to scxa-postgres-test. We don’t want to use
scxa-postgres by mistake and wipe the tables from the dev instance when cleaning fixtures... such an unfortunate
accident is known to have happened.
The job is split in the following six phases:
- Clean build directory
- Compile test classes
- Run unit tests
- Run integration tests
- Run end-to-end tests
- Generate JaCoCo reports
You should specify which SCHEMA_VERSION you want to test against (i.e. anndata or non-anndata database, see the
PostgreSQL section above).
For non-anndata:
SCHEMA_VERSION=18 \
docker compose \
--env-file ./docker/dev.env \
-f ./docker/docker-compose-gradle.yml \
-f ./docker/docker-compose-postgres-test.yml \
-f ./docker/docker-compose-solrcloud.yml \
upFor anndata:
SCHEMA_VERSION=latest \
docker compose \
--env-file ./docker/dev.env \
-f ./docker/docker-compose-gradle.yml \
-f ./docker/docker-compose-postgres-test.yml \
-f ./docker/docker-compose-solrcloud.yml \
upYou will eventually see these log messages:
scxa-gradle | BUILD SUCCESSFUL in 2s
scxa-gradle | 3 actionable tasks: 1 executed, 2 up-to-date
scxa-gradle exited with code 0
Press Ctrl+C to stop the container and clean any leftovers:
SCHEMA_VERSION=18 \
docker compose \
--env-file ./docker/dev.env \
-f ./docker/docker-compose-gradle.yml \
-f ./docker/docker-compose-postgres-test.yml \
-f ./docker/docker-compose-solrcloud.yml \
downOr run ./stop-and-remove-containers.sh.
You will find very convenient to use the script execute-all-tests.sh. By default, it runs the anndata database, but
the schema version can be provided as an argument. E.g.:
./execute-all-tests.sh [ 18 ]The script uses docker compose run, and control returns to your shell once the tasks have finished.
Many times you will find yourself working in a specific test case or class. Running all tests in such cases is
impractical. In such situations you can use
Gradle’s continuous build execution. See the example below for
e.g. ExperimentFileLocationServiceIT.java:
docker compose \
--env-file ./docker/dev.env \
-f ./docker/docker-compose-gradle.yml \
-f ./docker/docker-compose-postgres-test.yml \
-f ./docker/docker-compose-solrcloud.yml \
run --rm --service-ports \
scxa-gradle bash -c '
./gradlew :app:clean &&
./gradlew \
-PdataFilesLocation=/atlas-data \
-PexperimentFilesLocation=/atlas-data/scxa \
-PexperimentDesignLocation=/atlas-data/expdesign \
-PjdbcUrl=jdbc:postgresql://${POSTGRES_HOST}:5432/${POSTGRES_DB} \
-PjdbcUsername=${POSTGRES_USER} \
-PjdbcPassword=${POSTGRES_PASSWORD} \
-PzkHosts=${SOLR_CLOUD_ZK_CONTAINER_1_NAME}:2181,${SOLR_CLOUD_ZK_CONTAINER_2_NAME}:2181,${SOLR_CLOUD_ZK_CONTAINER_3_NAME}:2181 \
-PsolrHosts=http://${SOLR_CLOUD_CONTAINER_1_NAME}:8983/solr,http://${SOLR_CLOUD_CONTAINER_2_NAME}:8983/solr \
app:testClasses &&
./gradlew --continuous :app:test --tests ExperimentFileLocationServiceIT
'After running the test Gradle stays idle and waits for any changes in the code. When it detects that the files in your
project have been updated it will recompile them and run the specified test again. Notice that you can specify multiple
test files after --tests (by name or with wildcards).
Again, a convenience script can be used:
./execute-single-test.sh TEST_NAME [ 18 ]The second argument is optional and can be used to specify a database version. As before, if unspecified it defaults to
latest.
If you want to use a debugger, add the option -PremoteDebug to the task test line. For instance:
./gradlew -PremoteDebug :app:test --tests CellPlotDaoITBe aware that Gradle won’t execute the tests until you attach a remote debugger to port 5005. It will notify you when it’s ready with the following message:
> Task :app:test
Listening for transport dt_socket at address: 5005
<===========--> 90% EXECUTING [5s]
> IDLE
> IDLE
> IDLE
> IDLE
> IDLE
> IDLE
> IDLE
> :app:test > 0 tests completed
> IDLE
> IDLE
> IDLE
> IDLE
You can combine --continuous with -PremoteDebug, but the debugger will be disconnected at the end of the test. You
will need to start and attach the remote debugger every time Gradle compiles and runs the specified test.
The script debug-single-test.sh is a shortcut for this task. It takes the same arguments as executing a single test.
The web application is compiled in two stages:
- Front end JavaScript packages are transpiled into “bundles” with Webpack
- Bundles and back end Java code are built as a WAR file
Lastly, Tomcat deploys the WAR file according to app/src/main/webapp/META-INF/context.xml; other Java EE web servers
might work but no testing has been carried out in this regard.
For the first step you can run the following script:
./compile-front-end-packages.sh -iuThe sceond step is simply:
./gradlew :app:warThe script build-and-deploy-webapp.sh puts it altogether and will eventually launch a Tomcat container with a running
dev instance of Single Cell Expression Atlas. The script before launching the web application can build only the back-end or front-end component or both.
Here is the usage of this script:
- -f Use this flag if you would like to build the front-end javascript packages.
- -b Use this flag if you would like to build the back-end of the web application.
- -h Displaying the help file of this script.
If you don't give any flags, or you add both then the script is going to build both front and back-end part of the web application.
It also creates a service with one Gradle container that you can attach to your terminal:
docker attach scxa-gradle-shell-1You can run tests within and any Gradle task in that container (a running Gradle daemon is provided). Every time you
re-run the war task (e.g. in the Gradle shell container) the web app will be automatically re-deployed by Tomcat.
You can also set a Docker Compose Run configuration in IntelliJ IDEA with SCHEMA_VERSION in environment variables
and dev.env in environment files.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent
