{"id":28591463,"url":"https://github.com/thehyve/fairspace-metadata-testdata","last_synced_at":"2026-07-10T22:31:41.828Z","repository":{"id":45601495,"uuid":"313861444","full_name":"thehyve/fairspace-metadata-testdata","owner":"thehyve","description":"Testdata generator for metadata in Fairspace","archived":false,"fork":false,"pushed_at":"2021-12-06T08:04:06.000Z","size":244,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-11T09:14:53.838Z","etag":null,"topics":["fairspace"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thehyve.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-18T07:53:24.000Z","updated_at":"2021-12-06T08:04:07.000Z","dependencies_parsed_at":"2022-09-11T15:41:31.180Z","dependency_job_id":null,"html_url":"https://github.com/thehyve/fairspace-metadata-testdata","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/thehyve/fairspace-metadata-testdata","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thehyve%2Ffairspace-metadata-testdata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thehyve%2Ffairspace-metadata-testdata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thehyve%2Ffairspace-metadata-testdata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thehyve%2Ffairspace-metadata-testdata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thehyve","download_url":"https://codeload.github.com/thehyve/fairspace-metadata-testdata/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thehyve%2Ffairspace-metadata-testdata/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35345661,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fairspace"],"created_at":"2025-06-11T09:14:51.908Z","updated_at":"2026-07-10T22:31:41.804Z","avatar_url":"https://github.com/thehyve.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Testdata generator for metadata in Fairspace\n\n## Dependencies\n\nRequires Python 3.7 or newer.\nPreferably run within a virtual environment:\n```\npython3 -m venv venv/\nsource venv/bin/activate\n```\nInstall:\n```\npip install .\n```\n\n## Testdata generation\n\nConfigure the connection settings using environment variables (either specify using the shell or create a file `.env`). \nMake sure that the user specified in the config has an `Admin`, `Add shared metadata` and `Query metadata` roles.\nExample for a local development environment:\n```shell\nFAIRSPACE_URL=http://localhost:8080\nKEYCLOAK_URL=http://localhost:5100\nKEYCLOAK_REALM=fairspace\nKEYCLOAK_CLIENT_ID=workspace-client\nKEYCLOAK_CLIENT_SECRET=**********\nKEYCLOAK_USERNAME=organisation-admin\nKEYCLOAK_PASSWORD=fairspace123\n```\n\nRun the script:\n```shell\nupload_test_data\n```\nThis uploads 1,000 subjects, 1,500 tumor pathology events,\n3,000 samples and creates 5 collection, each containing 50\ndirectories with 500 files.\n\nThese numbers can be changed using environment variables:\n```shell\nSUBJECT_COUNT=1000\nEVENT_COUNT=1500\nSAMPLE_COUNT=3000\nCOLLECTION_COUNT=5\nDIRS_PER_COLLECTION=50\nFILES_PER_DIR=500\n```\n\nOr run with different parameters:\n```python\nfrom metadata_scripts.upload_test_data import TestData\ndef testrun():\n    testdata = TestData()\n    testdata.subject_count = 100\n    testdata.event_count = 200\n    testdata.sample_count = 500\n    testdata.collection_count = 5\n    testdata.dirs_per_collection = 5\n    testdata.files_per_dir = 10\n    testdata.run()\n\ntestrun()\n```\n\nAt the end of the script a recreation of the Fairspace view database will be triggered, based on the updated RDF database.\nThis can take up to 2 minutes for the default count parameters configuration.\n\n## Run queries\n\nThe `retrieve_view` command retrieves the first page of samples by default,\nuse `retrieve_view Subject` to retrieve a page of subjects, etc.\n\nThe API for retrieving pages can be used directly to specify the page and\nfilters:\n```python\nfrom fairspace_api.api import FairspaceApi\nimport json\nimport time\n\napi = FairspaceApi()\n\nfilters = [{'field': 'gender', 'values': ['male']}]\nstart = time.time()\ntry:\n    count = api.count('samples', filters=filters)\n    print(f'Took {time.time() - start:.0f}s.')\n    print(count)\n    start = time.time()\n    page = api.retrieve_view_page('samples', page=1, size=2, filters=filters)\n    print(f'Took {time.time() - start:.0f}s.')\n    print(page)\nexcept Exception as e:\n    print('Error', e)\n\n\n# Retrieve first 2 female subjects for which blood samples with ChIP-on-Chip analysis are available\npage = api.retrieve_view_page('Sample', page=1, size=2, include_counts=True, filters=[\n  {'field': 'Subject.gender', 'values': ['Female']},\n  {'field': 'Sample.nature', 'values': ['Blood']},\n  {'field': 'Collection.analysisType', 'values': ['ChIP-on-Chip']}\n])\nprint(json.dumps(page.__dict__, indent=2))\n```\n\nSPARQL queries:\n```python\nfrom fairspace_api.api import FairspaceApi\napi = FairspaceApi()\nquery = \"\"\"\nPREFIX rdfs:  \u003chttp://www.w3.org/2000/01/rdf-schema#\u003e\nPREFIX curie: \u003chttps://institut-curie.org/ontology#\u003e\nPREFIX fs:    \u003chttps://fairspace.nl/ontology#\u003e\n\nSELECT ?sample ?sampleTopography ?sampleNature ?sampleOrigin ?tumorCellularity ?event ?tumorTopography ?morphology ?eventType\n    ?laterality ?tumorGradeType ?tumorGradeValue ?tnmType ?tnmT ?tnmN ?tnmM ?ageAtDiagnosis ?subject ?gender\n    ?species ?ageAtLastNews ?ageAtDeath\nWHERE {\n    OPTIONAL {?sample curie:topography ?sampleTopography }\n    OPTIONAL {?sample curie:isOfNature ?sampleNature }\n    OPTIONAL {?sample curie:hasOrigin ?sampleOrigin }\n    OPTIONAL {?sample curie:tumorCellularity ?tumorCellularity }\n    \n    ?sample curie:subject ?subject .\n    OPTIONAL {?subject curie:isOfSpecies ?species}\n    OPTIONAL {?subject curie:gender ?gender}\n    OPTIONAL {?subject curie:ageAtLastNews ?ageAtLastNews}\n    OPTIONAL {?subject curie:ageAtDeath ?ageAtDeath}\n\n    OPTIONAL {\n        ?sample curie:diagnosis ?event .\n        \n        OPTIONAL { ?event curie:tumorMorphology ?morphology }\n        OPTIONAL { ?event curie:eventType ?eventType }\n        OPTIONAL { ?event curie:topography ?tumorTopography }\n        OPTIONAL { ?event curie:tumorLaterality ?laterality }\n        OPTIONAL { ?event curie:ageAtDiagnosis ?ageAtDiagnosis }\n        OPTIONAL { ?event curie:tumorGradeType ?tumorGradeType }\n        OPTIONAL { ?event curie:tumorGradeValue ?tumorGradeValue }\n        OPTIONAL { ?event curie:tnmType ?tnmType }\n        OPTIONAL { ?event curie:tnmT ?tnmT }\n        OPTIONAL { ?event curie:tnmN ?tnmN }\n        OPTIONAL { ?event curie:tnmM ?tnmM }\n    }\n\n    ?sample a curie:BiologicalSample .\n    FILTER NOT EXISTS { ?sample fs:dateDeleted ?anyDateDeleted }\n}\nLIMIT 10\n\"\"\"\nresults = api.query_sparql(query)\nsample_ids = [sample['sample']['value'] for sample in results['results']['bindings']]\nprint(len(sample_ids), 'samples,', len(set(sample_ids)), 'unique:', set(sample_ids))\n```\n\n## License\n\nCopyright (c) 2021 The Hyve B.V.\n\nThis program is free software: you can redistribute it and/or modify it under the terms of the Apache 2.0 \nLicense published by the Apache Software Foundation, either version 2.0 of the License, or (at your option) any later version.\n\nThis program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty\nof MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Apache 2.0 License for more details.\n\nYou should have received a copy of the Apache 2.0 License along with this program (see [LICENSE](LICENSE)). If not, see https://www.apache.org/licenses/LICENSE-2.0.txt.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthehyve%2Ffairspace-metadata-testdata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthehyve%2Ffairspace-metadata-testdata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthehyve%2Ffairspace-metadata-testdata/lists"}