{"id":24527245,"url":"https://github.com/biobakery/firecloud-graphql","last_synced_at":"2026-01-27T09:17:10.735Z","repository":{"id":104687841,"uuid":"172793873","full_name":"biobakery/firecloud-graphql","owner":"biobakery","description":"GraphQL server and utilities scripts for the BIOM-Mass portal","archived":false,"fork":false,"pushed_at":"2024-11-16T01:29:17.000Z","size":464,"stargazers_count":1,"open_issues_count":2,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-01-22T06:17:42.725Z","etag":null,"topics":["biobakery","misc"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/biobakery.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-26T21:32:12.000Z","updated_at":"2024-12-03T15:21:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"8474f8cd-7c42-47e4-87a7-9772bd6c2ad5","html_url":"https://github.com/biobakery/firecloud-graphql","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biobakery%2Ffirecloud-graphql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biobakery%2Ffirecloud-graphql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biobakery%2Ffirecloud-graphql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biobakery%2Ffirecloud-graphql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/biobakery","download_url":"https://codeload.github.com/biobakery/firecloud-graphql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243757374,"owners_count":20343216,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["biobakery","misc"],"created_at":"2025-01-22T06:17:45.627Z","updated_at":"2026-01-27T09:17:05.713Z","avatar_url":"https://github.com/biobakery.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Firecloud GraphQl\n\n## GraphQL Server\n\nTo install: \n`` $ pip install firecloud flask graphene\u003e=2.0.0 flask-graphql flask-cors sqlalchemy gunicorn gevent``\n\n* Also [firecloud-tools](https://github.com/broadinstitute/firecloud-tools) are required for the Firecloud API queries.\n\nTo run:\n`` $ nohup gunicorn -b 0.0.0.0:5000 -w 2 \"server:main()\" --timeout 1200 --log-level=debug -k gevent \u0026 ``\n\nThe file named `schema.graphql` contains the current schema (implemented in the module of the same name). Use the data utility script in the Portal UI repository to update this schema when needed.\n\n## External API Utilities\n\nThe server will load data from a local database. This database is created by running a utility script which queries the Firecloud and BigQuery APIs.\n\n### Run \n\nThe script that loads the local database with data from the Firecloud workspaces and BigQuery database is run as follows.\n\n``python load_local_database.py``\n\nScript calls functions from other 2 scripts query_firecloud.py and query_bigquery.py. These 2 scripts can be run to view data in Firecloud and BigQuery\n\nBefore running the script first set up the environment for the external API calls and have mariadb running. This script requires the same three\nenvironment variables to query the local database as the server.\n\nAlso make sure the service account has been added to all Terra workspaces so that the script can pull the metadata through the Firecloud API.\n\nFor projects with metadata in local csv files add the options `--file-participant participant.csv --file-sample sample.csv`.\n\n### Firecloud API\n\n#### Installation \n\nA registered service account must be set up prior to using the Firecloud API script. \n\nFollow these steps to set up an account registered with Firecloud:\n\n1. In Google Cloud go to IAM -\u003e Service accounts (and create a new service account or use the existing compute if running on a VM) and create a key.\n2. Download this key to the machine you will run the Firecloud API script.\n3. Install firecloud-tools and run the following command (replacing $PATH_TO_KEY and $EMAIL with specific values)\n```\n$ ./run.sh scripts/register_service_account/register_service_account.py -j $PATH_TO_KEY -e $EMAIL\n```\n4. Set the environment variable to point to the KEY\n```\n$ export GOOGLE_APPLICATION_CREDENTIALS=$PATH_TO_KEY\n```\n5. Login to the account (if not using a compute engine service account on a VM) with `gsutil config`\n6. Add the service account to each Firecloud workspace it will need to query. This can be done by adding the service account email id to the \"Share\" page as a new \"User ID\" with READER level access (no share and no compute permissions).\n\nWhen running the Firecloud API script make sure the environment variable is set and firecloud-tools, plus all of its dependencies are installed.\n\n### BigQuery API\n\n#### Installation\n\nTo install the client library run\n\n``pip install --upgrade google-cloud-bigquery``\n\nTo set up authentication\n\n1. In the Google Console of your project create service account\n2. Download generated json file that contains your credentials key\n3. Either set path to your credentials file as environmental variable\n\n ``export GOOGLE_APPLICATION_CREDENTIALS=$PATH_TO_KEY``\n  \n  or pass path directly when defining client within code\n  \n ``from google.cloud import bigquery``\n ``client = bigquery.Client.from_service_account_json($PATH_TO_KEY)``\n \n More information \n https://cloud.google.com/bigquery/docs/reference/libraries#client-libraries-install-python\n \n## Data Repositories\n\n### BigQuery Dataset\n\nBiom-mass project Demo dataset 'HPFS_Demo_Clean' has 2 tables\n\n1. participant (has all fileds that describe participant of study)\n2. sample (has all fileds that describe measurements/tests taken on participant during study\n\nsample table has participant field that connects it to participant by id.\n\n### Firecloud Workspaces\n\nFirecloud workspaces have\n\n1. All raw files and processed files of study uploaded to google backet associated with workspace\n2. Information about files is stored as key/value pairs in 'sample' section\n\naccess \ndata_category\ndata_format\nexperimental_strategy\nfile_id\nfile_name\nfile_size\nparticipant\nplatform\nsample\ntype\n\n3. Participant ids are stored in 'participant' section of workspace\n\nFiles and related information are associated to participants by key 'participant' that matches participant id.\n\n### Local Maria DB\n\nLocal to portal mariadb combines all data from BigQuery and Firecloud Workspaces and stores information in 3 tables\n\n1. participant (all participant information from BigQuery)\n2. sample (all sample information from BigQuery)\n3. file_sample (all file information from all Firecloud Workspaces)\n\nfile_sample and sample tables have 'participant' field that connects them to participant table by id.\n\nTo install mariadb on ubuntu/debian run\n\n``sudo apt-get update -y``\n``sudo apt-get install mariadb-server mariadb-client``\n\nTo set root password (it is none by default) run\n\n``sudo mysql_secure_installation`` \n\nMore information\nhttps://websiteforstudents.com/students-tutorial-install-mariadb-ubuntu-16-10/\n\n\nTo create non root user in mysql  login as root user\n``sudo mysql --user=root -p``\nthen run\n1. ``CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';``\n2. ``GRANT ALL PRIVILEGES ON *.* TO 'username'@'localhost' WITH GRANT OPTION;``\n3. ``flush privileges;``\n\nMore information https://dev.mysql.com/doc/refman/5.5/en/adding-users.html\n\nload_local_database.py script needs mysql.connect module.\n\nTo install run\n``pip install mysql-connector-python``\n\n\n## Production\n\nThis server runs in a docker container that is hosted by nginx running in the portal-ui container. \n\nTo build: `$ sudo docker sudo docker build -t graphql . `\n\nTo run: `$ sudo docker run -d --name graphql -e MYSQL_ROOT_PASSWORD=biobakery -e MYSQL_USER=biom_mass -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=portal_ui -p 5000:5000 -v /opt/database:/var/lib/mysql -v /opt/logs:/usr/local/src/logs/ graphql`\n\nWhen running a new container replace passwords in demo command above. This command will reuse the existing database at /opt/database.\n\nThe steps to build a new container are included in the script `build.bash`.\n\nTo start:\n```\n$ sudo docker exec -it graphql bash\n$ cd /usr/local/src\n$ gsutil config -e (# then provide the path to the key at $GOOGLE_APPLICATION_CREDENTIALS and the project id)\n$ python load_local_database.py (if needed as based on prior db with mount)\n$ nohup python server.py \u003e server.log \u0026\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbiobakery%2Ffirecloud-graphql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbiobakery%2Ffirecloud-graphql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbiobakery%2Ffirecloud-graphql/lists"}