{"id":44540808,"url":"https://github.com/ebi-gene-expression-group/atlas-web-bulk","last_synced_at":"2026-02-13T18:58:45.055Z","repository":{"id":35144810,"uuid":"178305267","full_name":"ebi-gene-expression-group/atlas-web-bulk","owner":"ebi-gene-expression-group","description":"Bulk Expression Atlas web application","archived":false,"fork":false,"pushed_at":"2026-01-30T17:59:22.000Z","size":129659,"stargazers_count":2,"open_issues_count":58,"forks_count":2,"subscribers_count":8,"default_branch":"develop","last_synced_at":"2026-01-31T10:43:21.098Z","etag":null,"topics":["java","spring"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ebi-gene-expression-group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-03-29T00:52:40.000Z","updated_at":"2026-01-30T17:59:50.000Z","dependencies_parsed_at":"2023-10-13T06:30:24.510Z","dependency_job_id":"35eb3319-4936-4b6d-885d-50aa1bf426c9","html_url":"https://github.com/ebi-gene-expression-group/atlas-web-bulk","commit_stats":{"total_commits":163,"total_committers":9,"mean_commits":18.11111111111111,"dds":"0.45398773006134974","last_synced_commit":"558590f7493dbbcf7bc9c9348b694030d95312e9"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/ebi-gene-expression-group/atlas-web-bulk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebi-gene-expression-group%2Fatlas-web-bulk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebi-gene-expression-group%2Fatlas-web-bulk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebi-gene-expression-group%2Fatlas-web-bulk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebi-gene-expression-group%2Fatlas-web-bulk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ebi-gene-expression-group","download_url":"https://codeload.github.com/ebi-gene-expression-group/atlas-web-bulk/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebi-gene-expression-group%2Fatlas-web-bulk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29414313,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T06:24:03.484Z","status":"ssl_error","status_checked_at":"2026-02-13T06:23:12.830Z","response_time":78,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java","spring"],"created_at":"2026-02-13T18:58:44.392Z","updated_at":"2026-02-13T18:58:45.046Z","avatar_url":"https://github.com/ebi-gene-expression-group.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Expression Atlas\n\n## CI/CD\n\nJenkins Build job is [Bulk Expression Atlas - Develop](https://gene-expression.ebi.ac.uk/jenkins/job/Bulk%20Expression%20Atlas%20-%20Develop/)\n\n## IDE Setup\n\nThese instructions are for VS Code\n\n1. install Extension Pack for Java\n\n2. set workspace to use a jdk 11 by adding your jdk 11 path in [the workspace's settings.json](.vscode/settings.json)\n\n```json\n\"java.configuration.runtimes\": [\n        {\n            \"name\": \"JavaSE-11\",\n            \"path\": \"/usr/local/opt/openjdk@11/\"\n        }\n    ]\n```\n\n## Prepare your development environment\n\n### TL;DR\n\n```bash\n./docker/prepare-dev-environment/gradle-cache/run.sh -r -l gradle-cache.log \u0026\u0026 \\\n./docker/prepare-dev-environment/volumes/run.sh -r -l volumes.log \u0026\u0026 \\\n./docker/prepare-dev-environment/postgres/run.sh -r -l pg.log \u0026\u0026 \\\n./docker/prepare-dev-environment/solr/run.sh -r -l solr.log\n```\n\n### Requirements\n- Docker v20+ with the [Compose plugin](https://docs.docker.com/compose/install/)\n- 100 GB of available storage for the following Docker volumes:\n  - Experiment files\n  - Bioentity properties (i.e. gene annotations)\n  - PostgreSQL\n  - SolrCloud and ZooKeeper\n  - Tomcat configuration files\n\nFiles written by Solr, PostgreSQL and Tomcat are kept in volumes which will be reused even if the containers are\nremoved (e.g. when running `docker-compose down`).  If you want to start afresh delete the old volume(s) (e.g. for\nPostgres `docker volume rm gxa-pgdata`) and re-run the necessary script to return to the\ninitial state. You can find the volume names used by each service in the `volumes` section of its Docker Compose YAML\nfile.\n\nThe full list of volumes is:\n- `gxa_atlas-data-bioentity-properties`\n- `gxa_atlas-data-gxa`\n- `gxa_atlas-data-gxa-expdesign`\n- `gxa_gradle-ro-dep-cache`\n- `gxa_gradle-wrapper-dists`\n- `gxa_pgdata`\n- `gxa_solrcloud-1-data`\n- `gxa_solrcloud-2-data`\n- `gxa_zk-1-data`\n- `gxa_zk-1-datalog`\n- `gxa_zk-2-data`\n- `gxa_zk-2-datalog`\n- `gxa_zk-3-data`\n- `gxa_zk-3-datalog`\n- `gxa_tomcat-conf`\n- `gxa_webapp-properties`\n\n### Code\nClone the repository of Bulk Expression Atlas with submodules:\n```bash\ngit clone --recurse-submodules https://github.com/ebi-gene-expression-group/atlas-web-bulk.git\n```\nIf you have already cloned the project ensure it’s up-to-date:\n```bash\n  git pull\n  git submodule update --remote\n```\n\n### Create a Gradle read-only dependency cache\nTo speed up builds and tests it is strongly encouraged to create a Docker volume to back a [Gradle read-only dependency\ncache](https://docs.gradle.org/current/userguide/dependency_resolution.html#sub:ephemeral-ci-cache).\n```bash\n./docker/prepare-dev-environment/gradle-cache/run.sh -r -l gradle-cache.log\n```\n\n### Prepare volumes\nIn order to run integration tests and a development instance of Bulk Expression Atlas you will need a few Docker\nvolumes first. They will be populated with data that will be indexed in Solr and Postgres. Bulk Expression Atlas\nneeds all three of: file bundles in the volumes, Solr collections and Postgres data. This step takes care of the first\nrequirement:\n```bash\n./docker/prepare-dev-environment/volumes/run.sh -r -l volumes.log\n```\n\nYou can get detailed information about which volumes are created if you run the script with the `-h` flag.\n\nThis script, unless it’s run with the `-r` flag, can be interrupted without losing any data. The container mirrors\ndirectories via FTP, and can resume after cancellation. It can be re-run to update the data in the volumes should the\ncontents of the source directories change. This is especially useful when experiments are re-analysed/re-annotated,\nor the bioentity properties directory is updated after a release of  Ensembl, WormBase ParaSite, Reactome, Gene\nOntology, Plant Ontology or InterPro.\n\n### How to add a private experiment bundle\n\nPrivate experiments are not available to download from our FTP site. You can download them from the `codon-cluster` by using the following steps:\n\n1. These steps should be done before the `PostGreSQL` and `Solr` steps. \n2. After logged in to the `codon-cluster` check if the experiment bundle can be found under this path:\n`/nfs/production/irene/ma/experiments/`.\n3. If it is there, then go to the folder on your local computer where you would like to download the bundle.\n4. Download it by this command:\n    ```bash\n    scp -r codon-login:/nfs/production/irene/ma/experiments/\u003cEXPERIMENT_ACCESSION_ID\u003e .\n    ```\n5. Create a temp container with mounting the already existing data volume for our local experiments:\n    ```bash\n    docker container create --name expVol -v gxa_atlas-data-exp:/atlas-data/exp ubuntu:jammy\n    ```\n6. Copy the file bundles of the downloaded private experiment into the volume: \n    ```bash \n    docker cp \u003cEXPERIMENT_ACCESSION_ID\u003e expVol:/atlas-data/exp/magetab/\n    ```\n7. Add `\u003cEXPERIMENT_ACCESSION_ID\u003e` into the `PRIVATE_EXP_IDS` variable. It is in to `test-data.env` file under the `docker/prepare-dev-environment` folder. If it is not there, then please create it.\n8. The experiment accession IDs in that variable should be separated by SPACE.\n\n\n\n### PostGreSQL\n\nTo create our PostGreSQL database and run the schema migrations up to the latest version please execute this script:\n```bash\n./docker/prepare-dev-environment/postgres/run.sh -r -l pg.log\n```\n\n### Solr\nTo create the collections, their schemas and populate them, please run the following scripts.\n\nCurrently, this step is separated into 2 sub-steps by Solr collections. \nThere is an inconsistency in our web apps and various shell scripts - that we use together with the Data Prod Team -\nhow we use the `SOLR_HOST` and `SOLR_HOSTS` variables. We need to sort this out, \nbut while it is not solved we probably have to keep this 2 sub-steps, unless we find a way to merge them.\n\nTo create and populate the `bioentities` collection:\n```bash\n./docker/prepare-dev-environment/solr-bioentities/run.sh -r -l solr-bioentities.log\n```\n\nTo create and populate the `bulk-analytics` collection:\n```bash\n./docker/prepare-dev-environment/solr-analytics/run.sh -l solr-analytics.log\n```\n\nRun the script with the `-h` flag for more details.\n\nYou may want to speed up the process by raising the value of the environment variable `NUM_DOCS_PER_BATCH` (L81 of the\n`run.sh` script). On [a fairly powerful laptop at the time of\nwriting](https://www.lenovo.com/gb/en/p/laptops/thinkpad/thinkpadx1/x1-extreme-gen-2/22tp2txx1e2) 20,000 has been\nfound to be a reliable number via painstaking trail and error, but your mileage may vary. Ensure that there are no\nerrors in the script logs, or update your test data by add the necessary species names and experiment accessions in the `test-dev.env` file and rebuild the development\nenvironment. Some tests may fail due to incomplete annotations; `grep` for `DistributedUpdatesAsyncException` in\nparticular, which signals a problem storing the document batch, which in turn stops processing the current file. If\nfound, try again with a lower value for `NUM_DOCS_PER_BATCH`.\n\n### Update test data\nAdd or change the necessary species names and experiment accessions in the `test-data.env` file and rebuild the\ndevelopment environment.\n\n## Testing\n\n### TL;DR\n```bash\n./execute-all-tests.sh      \n./execute-single-test.sh TEST_NAME \n./debug-single-test.sh TEST_NAME\n./stop-and-remove-containers.sh\n```\n\n### Execute all tests\nThe `gxa-gradle` service in `docker/docker-compose-gradle.yml` executes all tests and writes reports to\n`atlas-web-core/build` and `app/build` in the host machine. It requires the SolrCloud service described earlier, and a\nPostgres container with the following differences compared to the development service, `gxa-postgres`: it doesn’t use\nnamed volumes to ensure the database is clean before running any tests, and its name (as well as the dependency\ndeclared in `docker-compose-gradle.yml`) has been changed to `gxa-postgres-test`. We don’t want to use\n`gxa-postgres` by mistake and wipe the tables from the dev instance when cleaning fixtures... such an unfortunate\naccident is known to have happened.\n\nThe job is split in the following six phases:\n1. Clean build directory\n2. Compile test classes\n3. Run unit tests\n4. Run integration tests\n5. Run end-to-end tests\n6. Generate JaCoCo reports\n\n\nYou will eventually see these log messages:\n```\ngxa-gradle         | BUILD SUCCESSFUL in 2s\ngxa-gradle         | 3 actionable tasks: 1 executed, 2 up-to-date\ngxa-gradle exited with code 0\n```\n\nPress `Ctrl+C` to stop the container and clean any leftovers:\n```bash\ndocker-compose \\\n--env-file ./docker/dev.env \\\n-f ./docker/docker-compose-gradle.yml \\\n-f ./docker/docker-compose-postgres-test.yml \\\n-f ./docker/docker-compose-solrcloud.yml \\\ndown\n```\n\nOr run `./stop-and-remove-containers.sh`.\n\nYou will find very convenient to use the script `execute-all-tests.sh`.\n```bash\n./execute-all-tests.sh\n```\n\nThe script uses `docker-compose run`, and control returns to your shell once the tasks have finished, but you’ll need\nto clean up the service containers anyway.\n\n### Execute a single test\nMany times you will find yourself working in a specific test case or class. Running all tests in such cases is\nimpractical. In such situations you can use\n[Gradle’s continuous build execution](https://blog.gradle.org/introducing-continuous-build). See the example below for\ne.g. `GenePageControllerIT.java`:\n```bash\ndocker-compose \\\n--env-file ./docker/dev.env \\\n-f ./docker/docker-compose-gradle.yml \\\n-f ./docker/docker-compose-postgres-test.yml \\\n-f ./docker/docker-compose-solrcloud.yml \\\nrun --rm --service-ports \\\ngxa-gradle bash -c '\n./gradlew :app:clean \u0026\u0026\n./gradlew \\\n-PdataFilesLocation=/atlas-data \\\n-PexperimentFilesLocation=/atlas-data/gxa \\\n-PjdbcUrl=jdbc:postgresql://${POSTGRES_HOST}:5432/${POSTGRES_DB} \\\n-PjdbcUsername=${POSTGRES_USER} \\\n-PjdbcPassword=${POSTGRES_PASSWORD} \\\n-PzkHosts=${SOLR_CLOUD_ZK_CONTAINER_1_NAME}:2181,${SOLR_CLOUD_ZK_CONTAINER_2_NAME}:2181,${SOLR_CLOUD_ZK_CONTAINER_3_NAME}:2181 \\\n-PsolrHosts=http://${SOLR_CLOUD_CONTAINER_1_NAME}:8983/solr,http://${SOLR_CLOUD_CONTAINER_2_NAME}:8983/solr \\\napp:testClasses \u0026\u0026\n./gradlew --continuous :app:test --tests GenePageControllerIT\n'\n```\n\nAfter running the test Gradle stays idle and waits for any changes in the code. When it detects that the files in your\nproject have been updated it will recompile them and run the specified test again. Notice that you can specify multiple\ntest files after `--tests` (by name or with wildcards).\n\nAgain, a convenience script can be used:\n```bash\n./execute-single-test.sh TEST_NAME\n```\n\n### Debug tests\nIf you want to use a debugger, add the option `-PremoteDebug` to the command of the test. For instance:\n```bash\n./gradlew -PremoteDebug :app:test --tests GenePageControllerIT\n```\n\nBe aware that Gradle won’t execute the tests until you attach a remote debugger to port 5005. It will notify you when\nit’s ready with the following message:\n```\n\u003e Task :app:test\nListening for transport dt_socket at address: 5005\n\u003c===========--\u003e 90% EXECUTING [5s]\n\u003e IDLE\n\u003e IDLE\n\u003e IDLE\n\u003e IDLE\n\u003e IDLE\n\u003e IDLE\n\u003e IDLE\n\u003e :app:test \u003e 0 tests completed\n\u003e IDLE\n\u003e IDLE\n\u003e IDLE\n\u003e IDLE\n```\n\nYou can combine `--continuous` with `-PremoteDebug`, but the debugger will be disconnected at the end of the test. You\nwill need to start and attach the remote debugger every time Gradle compiles and runs the specified test.\n\nTo attach a remote debugger to your gradle test you can add following configuration in your IntelliJ:\n\n[![RMoIhF.md.png](https://iili.io/RMoIhF.md.png)](https://freeimage.host/i/RMoIhF)\n\n\nThe script `debug-single-test.sh` is a shortcut for this task. It takes the same arguments as executing a single test.\n\n```bash\n./debug-single-test.sh TEST_NAME\n```\n\n## Remote Debugging\n\nTomcat can be run in debug mode when started with `catalina jpda run`. The default debug port is 8000. You then need to run locally a java remote debug session and connect to the webapp tomcat host.\n\n### Debugging in VS Code\n\nSet a launch configuration that connects to the Tomcat host on the debug port.\n\n## Run web application\n\nPlease check this first in the troubleshooting session: [Known Build Issue](#known-build-issue)\n\nThe web application is compiled in two stages:\n1. Front end JavaScript packages are transpiled into “bundles” with [Webpack](https://webpack.js.org/)\n2. Bundles and back end Java code are built as a WAR file\n\nLastly, Tomcat deploys the WAR file according to `app/src/main/webapp/META-INF/context.xml`; other Java EE web servers\nmight work but no testing has been carried out in this regard.\n\nFor the first step you can run the following script:\n```bash\n ./compile-front-end-packages.sh -iu\n```\n\nThe second step is simply:\n```bash\n./gradlew :app:war\n```\n\nThe script `build-and-deploy-webapp.sh` puts it altogether and will eventually launch a Tomcat container with a running\ndev instance of Single Cell Expression Atlas. The script before launching the web application can build only the back-end or front-end component or both.\n\nHere is the usage of this script:\n\n- -n Use this flag if you would not like to do any build, just execute the application.\n- -f Use this flag if you would like to build the front-end javascript packages.\n- -b Use this flag if you would like to build the back-end of the web application.\n- -h Displaying the help file of this script.\n\nIf you don't give any flags, or you add both then the script is going to build both front and back-end part of the web application.\n\n## Troubleshooting\n\n### \u003ca name=\"known-build-issue\"\u003e\u003c/a\u003eKnown Build Issue\n\nThis current version of our developer env has a bug when we build and execute the application\nwith the `build-and-deploy-webapp.sh` script.\nWe have a ticket to fix this in our backlog: [Update bulk with the latest webpack and its dependencies](https://github.com/ebi-gene-expression-group/atlas-web-bulk/issues/176)\n\nYou can build a working WAR with the following steps:\n1. Manually build the UI:\n```bash\n./compile-front-end-packages.sh -iu\n```\n\n2. Manually build the backend:\n```bash\n./gradlew :app:war\n```\n\n3. Use this script to start up the web app on your local environment:\n```bash\n./build-and-deploy-webapp.sh -n\n```\n\n### SolrCloud nodes shut down on macOS\nDocker for macOS sets fairly strict resource limits for all Docker containers. If your containers require e.g. more\nmemory you need to increase the available amount in the Docker Dashboard. For bulk Expression Atlas, please set Memory\nto between 8-12 GB and disk image to 100 GB or more. Please see the screenshot below for reference:\n\n![Screenshot-2021-02-18-at-18-27-40](https://user-images.githubusercontent.com/4425744/109644570-8ccee680-7b4d-11eb-9db0-7a29fb4d9e2b.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Febi-gene-expression-group%2Fatlas-web-bulk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Febi-gene-expression-group%2Fatlas-web-bulk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Febi-gene-expression-group%2Fatlas-web-bulk/lists"}