{"id":17429837,"url":"https://github.com/pacuna/discogs-bq-importer","last_synced_at":"2026-05-02T13:34:04.800Z","repository":{"id":179700853,"uuid":"663999738","full_name":"pacuna/discogs-bq-importer","owner":"pacuna","description":"Scio job to import discogs xml dumps into BigQuery","archived":false,"fork":false,"pushed_at":"2023-07-10T00:06:54.000Z","size":44,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-28T00:41:28.517Z","etag":null,"topics":["apache","beam","discogs","discogs-dump","discogs-importer","scala","scio","xml"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pacuna.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-08T16:39:26.000Z","updated_at":"2023-07-18T09:36:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"ae5817a2-90aa-4edf-91cf-0c8bb7779fcd","html_url":"https://github.com/pacuna/discogs-bq-importer","commit_stats":null,"previous_names":["pacuna/discogs-bq-importer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pacuna/discogs-bq-importer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pacuna%2Fdiscogs-bq-importer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pacuna%2Fdiscogs-bq-importer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pacuna%2Fdiscogs-bq-importer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pacuna%2Fdiscogs-bq-importer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pacuna","download_url":"https://codeload.github.com/pacuna/discogs-bq-importer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pacuna%2Fdiscogs-bq-importer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32536578,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T12:25:33.646Z","status":"ssl_error","status_checked_at":"2026-05-02T12:24:51.733Z","response_time":132,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache","beam","discogs","discogs-dump","discogs-importer","scala","scio","xml"],"created_at":"2024-10-17T07:10:02.447Z","updated_at":"2026-05-02T13:34:04.780Z","avatar_url":"https://github.com/pacuna.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Discogs BigQuery importer\n\n## Raison d'être\n\nScio Dataflow job to load the [discogs xml dumps](https://discogs-data-dumps.s3.us-west-2.amazonaws.com/index.html) into BigQuery tables.\n\n## Running\n\nCurrent available jobs: \n\n- [x] Labels\n- [x] Artists\n- [x] Masters\n- [x] Releases\n\nTo run the jobs, download the compressed XML dumps and upload them to your own gcp bucket.\nFollow the [Scio instructions](https://spotify.github.io/scio/Getting-Started.html) to set up your GCP project.\n\nAll the times were measured using the default run arguments for Dataflow.\n\n### Releases (~ 1 hour)\n\n```\nsbt \"runMain discogs.ReleasesJob\n  --project=your-gpc-project-id\n  --runner=DataflowRunner\n  --region=us-central1\n  --input=gs://your-bucket/releases.xml.gz\n  --output=your-project.bq-dataset.releases-bq-table\"\n```\n\n### Artists (~ 11 minutes)\n\n```\nsbt \"runMain discogs.ArtistsJob\n  --project=your-gpc-project-id\n  --runner=DataflowRunner\n  --region=us-central1\n  --input=gs://your-bucket/artists.xml.gz\n  --output=your-project.bq-dataset.artists-bq-table\"\n```\n\n### Masters (~ 6 minutes)\n\n```\nsbt \"runMain discogs.MastersJob\n  --project=your-gpc-project-id\n  --runner=DataflowRunner\n  --region=us-central1\n  --input=gs://your-bucket/masters.xml.gz\n  --output=your-project.bq-dataset.masters-bq-table\"\n```\n\n### Labels (~ 6 minutes)\n\nBecause of an Apache Beam [XmlIO](https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html) limitation regarding\nnested tags with the same outer label (`\u003clabel\u003e`), the original labels file cannot be processed as is. There's a small\nscript to convert the nested `\u003clabels\u003e` tags into `\u003csublabels\u003e` in `src/main/java/utils/LabelRenamer.java`. Download the original final, decompress it\nand run the converter. Then compress the output file and upload it to your GCP bucket. Then use that file as the input for the job.\n\n```\nsbt \"runMain discogs.MastersJob\n  --project=your-gpc-project-id\n  --runner=DataflowRunner\n  --region=us-central1\n  --input=gs://your-bucket/labels-renamed.xml.gz\n  --output=your-project.bq-dataset.labels-bq-table\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpacuna%2Fdiscogs-bq-importer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpacuna%2Fdiscogs-bq-importer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpacuna%2Fdiscogs-bq-importer/lists"}