{"id":21660215,"url":"https://github.com/slub/musiconn-performance-data-pipeline","last_synced_at":"2026-04-09T23:52:57.883Z","repository":{"id":150196282,"uuid":"563333921","full_name":"slub/musiconn-performance-data-pipeline","owner":"slub","description":"musiconn.performance data transformation pipelines and utilities","archived":false,"fork":false,"pushed_at":"2022-12-01T13:17:20.000Z","size":722,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-01-25T07:08:58.360Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/slub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-08T11:58:35.000Z","updated_at":"2024-04-16T20:11:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"e560d0ea-6a95-4b85-957f-64c4fea6db06","html_url":"https://github.com/slub/musiconn-performance-data-pipeline","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slub%2Fmusiconn-performance-data-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slub%2Fmusiconn-performance-data-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slub%2Fmusiconn-performance-data-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slub%2Fmusiconn-performance-data-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/slub","download_url":"https://codeload.github.com/slub/musiconn-performance-data-pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244555951,"owners_count":20471535,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T09:32:30.097Z","updated_at":"2026-04-09T23:52:52.852Z","avatar_url":"https://github.com/slub.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"musiconn.performance - data pipelines and schemata\n===\n\nThis is a collection of schemata and processing utilities for the [Musiconn Performance](https://performance.musiconn.de/) catalog project.\nIts purpose is to provide a common vocabulary for describing\nmusical performances. It can be bootstraped from the elastic search\nindex of the Musiconn.perfomance database (as of november 2022). Its main purpose\nis to bring the data into an intermediate RDF format and provide utilities\nfor further processing and allow bulk imports to concrete databases (e.g. triple stores, neo4j,...).\n\n## Dumping the database\n\nAssumes elasticsearch to be running on `localhost:9200`.\n\nIf you want to use localhost for a remote ES instance, you need to\nset up a tunnel to the remote host:\n\n    ssh user@example.de -L 9200:localhost:9200 -N\n\n1. get all indices\n\n```shell\ncurl http://localhost:9200/_cat/indices?h=i | grep -v \"\\.\"\n```\n2. create a dump of all indices\n\n```shell\nINDICES=(date subject work corporation source series person authority location event )\nfor INDEX in \"${INDICES[@]}\";do \n  npx elasticdump --input=http://localhost:9200/${INDEX} --output=./data/${INDEX}_data.json --type=data\ndone\n```\n\n3.  optionally slurp json stream and make it valid json\n\n```shell\ncat ./data/${INDEX}_data.json | jq -s '.' \u003e ./data/${INDEX}_data_pretty.json\n```\n\n## Datatypes and Schemata\n\nTypes can automatically be inferred from the data dumps, so that\nchanges in the data model will cause the data pipeline to fail,\nonce the data model drifts away from the schemata.\n\nTo learn json-schemata from the data\n\n```shell\nnpm run build:jsonSchemas\n```\n\nTo generate typescript types from json-schema\n\n```shell\nnpm run build:types\n```\n\n## RDF data generation\n\ngenerate rdf graph from the dumped data\n\n ```shell\n npm run build:graph\n ```\n\n\n## Dataprocessing Pipeline\n\nThe stream processor takes the json data dump transforms them into RDF as intermediate\nformat and creates Cypher statements, that can be streamed directly to a neo4j database.\n\nyou could stream it to a neo4j instance\n\n```shell\nnpm run build:neo4j\n```\n\n## reproducable build environment using nix\n\n### on Linux ans MacOSX with nix installed\n\non any system, where the nix package manager is installed with experimental flake support\nturned on, you can just run\n\n```shell\nnix develop\n```\n\n### other OSs and Linux without nix installed\n\nyou can use a docker image containing the nix package manager to build the project\nor get a development environment with all dependencies installed.\n\n```shell\ndocker run  -v /var/run/docker.sock:/var/run/docker.sock \\\n            -v \"$(pwd):/app\" \\\n            -it -w /app --rm \\\n            nixos/nix \\\n            bash -c \"nix develop -L --extra-experimental-features nix-command --extra-experimental-features flakes --no-sandbox\"\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslub%2Fmusiconn-performance-data-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fslub%2Fmusiconn-performance-data-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslub%2Fmusiconn-performance-data-pipeline/lists"}