{"id":24884512,"url":"https://github.com/wuvt/mbslave","last_synced_at":"2026-02-27T00:32:07.455Z","repository":{"id":149675384,"uuid":"89880344","full_name":"wuvt/mbslave","owner":"wuvt","description":null,"archived":false,"fork":false,"pushed_at":"2020-07-30T06:14:30.000Z","size":777,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-11-09T12:22:54.983Z","etag":null,"topics":["dockerfile","musicbrainz"],"latest_commit_sha":null,"homepage":null,"language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wuvt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-30T22:11:29.000Z","updated_at":"2020-07-30T06:14:34.000Z","dependencies_parsed_at":"2023-04-29T08:16:29.046Z","dependency_job_id":null,"html_url":"https://github.com/wuvt/mbslave","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/wuvt/mbslave","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wuvt%2Fmbslave","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wuvt%2Fmbslave/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wuvt%2Fmbslave/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wuvt%2Fmbslave/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wuvt","download_url":"https://codeload.github.com/wuvt/mbslave/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wuvt%2Fmbslave/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29879005,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-26T23:51:21.483Z","status":"ssl_error","status_checked_at":"2026-02-26T23:50:46.793Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dockerfile","musicbrainz"],"created_at":"2025-02-01T14:28:31.930Z","updated_at":"2026-02-27T00:32:07.432Z","avatar_url":"https://github.com/wuvt.png","language":"PLpgSQL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MusicBrainz Database Setup\n\nThis repository contains a collection of scripts that can help you setup a local\ncopy of the MusicBrainz database and keep it up to date. There is no script that\ndoes everything for you though. The main motivation for writing these scripts was\nto customize the database, so the way to install the database might differ from\nuser to user.\n\n## Installation\n\n 0. Make sure you have [Python 2](http://python.org/) and [psycopg2](http://initd.org/psycopg/) installed.\n    On Debian and Ubuntu, that means installing these packages:\n\n        sudo apt install python python-psycopg2\n\n 1. Create `mbslave.conf` by copying and editing `mbslave.conf.default`.\n    You will need to get the API token on the [MetaBrainz website](https://metabrainz.org/supporters/account-type).\n\n 1. Setup the database. If you are starting completely from scratch,\n    you can use the following commands to setup a clean database:\n\n        sudo su - postgres\n        createuser musicbrainz\n        createdb -l C -E UTF-8 -T template0 -O musicbrainz musicbrainz\n        createlang plpgsql musicbrainz\n        psql musicbrainz -c 'CREATE EXTENSION cube;'\n        psql musicbrainz -c 'CREATE EXTENSION earthdistance;'\n\n 2. Prepare empty schemas for the MusicBrainz database and create the table structure:\n\n        echo 'CREATE SCHEMA musicbrainz;' | ./mbslave-psql.py -S\n        echo 'CREATE SCHEMA statistics;' | ./mbslave-psql.py -S\n        echo 'CREATE SCHEMA cover_art_archive;' | ./mbslave-psql.py -S\n        echo 'CREATE SCHEMA wikidocs;' | ./mbslave-psql.py -S\n        echo 'CREATE SCHEMA documentation;' | ./mbslave-psql.py -S\n\n        ./mbslave-remap-schema.py \u003csql/CreateTables.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/statistics/CreateTables.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/caa/CreateTables.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/wikidocs/CreateTables.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/documentation/CreateTables.sql | ./mbslave-psql.py\n\n 3. Download the MusicBrainz database dump files from\n    http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/\n\n 4. Import the data dumps, for example:\n\n        ./mbslave-import.py mbdump.tar.bz2 mbdump-derived.tar.bz2\n\n 5. Setup primary keys, indexes and views:\n\n        ./mbslave-remap-schema.py \u003csql/CreatePrimaryKeys.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/statistics/CreatePrimaryKeys.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/caa/CreatePrimaryKeys.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/wikidocs/CreatePrimaryKeys.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/documentation/CreatePrimaryKeys.sql | ./mbslave-psql.py\n\n        ./mbslave-remap-schema.py \u003csql/CreateIndexes.sql | grep -v musicbrainz_collate | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/CreateSlaveIndexes.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/statistics/CreateIndexes.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/caa/CreateIndexes.sql | ./mbslave-psql.py\n\n        ./mbslave-remap-schema.py \u003csql/CreateViews.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/CreateFunctions.sql | ./mbslave-psql.py\n\n 6. Vacuum the newly created database (optional)\n\n        echo 'VACUUM ANALYZE;' | ./mbslave-psql.py\n\n## Tips and Tricks\n\n### Single Database Schema\n\nMusicBrainz used a number of schemas by default. If you are embedding the MusicBrainz database into\nan existing database for your application, it's convenient to merge them all into a single schema.\nThat can be done by changing your config like this:\n\n    [schemas]\n    musicbrainz=musicbrainz\n    statistics=musicbrainz\n    cover_art_archive=musicbrainz\n    wikidocs=musicbrainz\n    documentation=musicbrainz\n\nAfter this, you only need to create the `musicbrainz` schema and import all the tables there.\n\n### No PostgreSQL Extensions\n\nYou can avoid installing the `cube` and `earthdistance` extensions if you map the `CUBE` column to `TEXT`\nand remove some indexes.\n\nYou might do a similar thing with `JSONB` columns, if you don't want to upgrade to PostgreSQL 9.5 yet.\n\nReplace the commands above with something like his:\n\n    ./mbslave-remap-schema.py \u003csql/CreateTables.sql | \\\n        perl -pe 's{\\b(CUBE|JSONB)\\b}{TEXT}' | \\\n        ./mbslave-psql.py\n\n    ./mbslave-remap-schema.py \u003csql/CreateIndexes.sql | \\\n        grep -v musicbrainz_collate | \\\n        grep -v ll_to_earth | \\\n        grep -v medium_index | \\\n        perl -pe 's{\\bUSING BRIN\\b}{}' | \\\n        perl -pe 'BEGIN { undef $/; } s{^CREATE INDEX edit_data_idx_link_type .*?;}{}smg' | \\\n        ./mbslave-psql.py\n\n### Full Import Upgrade\n\nYou can use the schema mapping feature to do zero-downtime upgrade of the database with full\ndata import. You can temporarily map all schemas to e.g. `musicbrainz_NEW`, import your new\ndatabase there and then rename it.\n\n    echo 'BEGIN; ALTER SCHEMA musicbrainz RENAME TO musicbrainz_OLD; ALTER SCHEMA musicbrainz_NEW RENAME TO musicbrainz; COMMIT;' | ./mbslave-psql.py -S\n\n## Replication\n\nAfter the initial database setup, you might want to update the database with the latest data.\nThe `mbslave-sync.py` script will fetch updates from MusicBrainz and apply it to your local database:\n\n```sh\n./mbslave-sync.py\n```\n\nIn order to update your database regularly, add a cron job like this that runs every hour:\n\n```cron\n15 * * * * $HOME/mbslave/mbslave-sync.py \u003e\u003e/var/log/mbslave.log\n```\n\n## Upgrading\n\nWhen the MusicBrainz database schema changes, the replication will stop working.\nThis is usually announced on the [MusicBrainz blog](http://blog.musicbrainz.org/).\nWhen it happens, you need to upgrade the database.\n\n### Release 2019-05-13 (25)\n\nRun the upgrade scripts:\n\n```\n./mbslave-remap-schema.py \u003csql/updates/schema-change/25.slave.sql | ./mbslave-psql.py\necho 'UPDATE replication_control SET current_schema_sequence = 25;' | ./mbslave-psql.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwuvt%2Fmbslave","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwuvt%2Fmbslave","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwuvt%2Fmbslave/lists"}