{"id":18681880,"url":"https://github.com/gravityblast/mbslave","last_synced_at":"2025-11-07T16:30:30.480Z","repository":{"id":66386678,"uuid":"11927410","full_name":"gravityblast/mbslave","owner":"gravityblast","description":"Fork of https://bitbucket.org/lalinsky/mbslave","archived":false,"fork":false,"pushed_at":"2014-07-12T16:08:23.000Z","size":784,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-05T22:39:40.311Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gravityblast.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-08-06T15:24:23.000Z","updated_at":"2022-10-14T11:56:53.000Z","dependencies_parsed_at":"2023-02-20T11:15:34.853Z","dependency_job_id":null,"html_url":"https://github.com/gravityblast/mbslave","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gravityblast%2Fmbslave","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gravityblast%2Fmbslave/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gravityblast%2Fmbslave/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gravityblast%2Fmbslave/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gravityblast","download_url":"https://codeload.github.com/gravityblast/mbslave/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239533064,"owners_count":19654617,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T10:10:03.200Z","updated_at":"2025-11-07T16:30:30.411Z","avatar_url":"https://github.com/gravityblast.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MusicBrainz Database Setup\n\nThis repository contains a collection of scripts that can help you setup a local\ncopy of the MusicBrainz database and keep it up to date. There is no script that\ndoes everything for you though. The main motivation for writing these scripts was\nto customize the database, so the way to install the database might differ from\nuser to user.\n\n## Installation\n\n 0. Make sure you have [Python](http://python.org/) and [psycopg2](http://initd.org/psycopg/) installed.\n\n 1. Setup a database and create `mbslave.conf` by copying and editing\n    mbslave.conf.default. If you are starting completely from scratch,\n    you can use the following commands to setup a clean database:\n\n        sudo su - postgres\n        createuser musicbrainz\n        createdb -l C -E UTF-8 -T template0 -O musicbrainz musicbrainz\n        createlang plpgsql musicbrainz\n\n 2. Prepare empty schemas for the MusicBrainz database and create the table structure:\n\n        echo 'CREATE SCHEMA musicbrainz;' | ./mbslave-psql.py -S\n        echo 'CREATE SCHEMA statistics;' | ./mbslave-psql.py -S\n        echo 'CREATE SCHEMA cover_art_archive;' | ./mbslave-psql.py -S\n        echo 'CREATE SCHEMA wikidocs;' | ./mbslave-psql.py -S\n        echo 'CREATE SCHEMA documentation;' | ./mbslave-psql.py -S\n        ./mbslave-remap-schema.py \u003csql/CreateTables.sql | sed 's/CUBE/TEXT/' | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/statistics/CreateTables.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/caa/CreateTables.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/wikidocs/CreateTables.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/documentation/CreateTables.sql | ./mbslave-psql.py\n\n 3. Download the MusicBrainz database dump files from\n    http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/\n\n 4. Import the data dumps, for example:\n\n        ./mbslave-import.py mbdump.tar.bz2 mbdump-derived.tar.bz2\n\n 5. Setup primary keys, indexes and views:\n\n        ./mbslave-remap-schema.py \u003csql/CreatePrimaryKeys.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/statistics/CreatePrimaryKeys.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/caa/CreatePrimaryKeys.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/wikidocs/CreatePrimaryKeys.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/documentation/CreatePrimaryKeys.sql | ./mbslave-psql.py\n\n        ./mbslave-remap-schema.py \u003csql/CreateIndexes.sql | grep -vE '(collate|page_index|medium_index)' | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/statistics/CreateIndexes.sql | ./mbslave-psql.py\n        ./mbslave-remap-schema.py \u003csql/caa/CreateIndexes.sql | ./mbslave-psql.py\n\n        ./mbslave-remap-schema.py \u003csql/CreateViews.sql | ./mbslave-psql.py\n\n 6. Vacuum the newly created database (optional)\n\n        echo 'VACUUM ANALYZE;' | ./mbslave-psql.py\n\n## Replication\n\nAfter the initial database setup, you might want to update the database with the latest data.\nThe `mbslave-sync.py` script will fetch updates from MusicBrainz and apply it to your local database:\n\n```sh\n./mbslave-sync.py\n```\n\nIn order to update your database regularly, add a cron job like this that runs every hour:\n\n```cron\n15 * * * * $HOME/mbslave/mbslave-sync.py \u003e\u003e/var/log/mbslave.log\n```\n\n## Upgrading\n\nWhen the MusicBrainz database schema changes, the replication will stop working.\nThis is usually announced on the [MusicBrainz blog](http://blog.musicbrainz.org/).\nWhen it happens, you need to upgrade the database.\n\n### Release 2014-05-14 (20)\n\nDownload some new data to import:\n\n```sh\nwget http://ftp.musicbrainz.org/pub/musicbrainz/data/schema-change-2014-05/mbdump-derived.tar.bz2\nwget http://ftp.musicbrainz.org/pub/musicbrainz/data/schema-change-2014-05/mbdump-documentation.tar.bz2\n```\n\nImport the data dumps:\n\n```sh\necho 'TRUNCATE TABLE release_tag;' | ./mbslave-psql.py\n./mbslave-import.py mbdump-derived.tar.bz2\n./mbslave-import.py mbdump-documentation.tar.bz2 \n```\n\nRun the upgrade scripts:\n\n```sh\n./mbslave-remap-schema.py \u003csql-extra/CreateUUIDFunctions.sql | ./mbslave-psql.py\n./mbslave-remap-schema.py \u003csql/updates/schema-change/20.slave.sql | grep -v to_tsvector | ./mbslave-psql.py\n./mbslave-remap-schema.py \u003csql/updates/20140509-place-example-pkeys.sql | ./mbslave-psql.py\necho 'UPDATE replication_control SET current_schema_sequence = 20;' | ./mbslave-psql.py\necho 'VACUUM ANALYZE;' | ./mbslave-psql.py\n```\n\n## Solr Search Index (Work-In-Progress)\n\nIf you would like to also build a Solr index for searching, mbslave includes a script to\nexport the MusicBrainz into XML file that you can feed to Solr:\n\n    ./mbslave-solr-export.py \u003e/tmp/mbslave-solr-data.xml\n\nOnce you have generated this file, you for example start a local instance of Solr:\n\n    java -Dsolr.solr.home=/path/to/mbslave/solr/ -jar start.jar\n\nImport the XML file:\n\n    curl http://localhost:8983/solr/musicbrainz/update -F stream.file=/tmp/mbslave-solr-data.xml -F commit=true\n\nInstall triggers to queue database updates:\n\n    echo 'CREATE SCHEMA mbslave;' | ./mbslave-psql.py -S\n    ./mbslave-remap-schema.py \u003csql-extra/solr-queue.sql | ./mbslave-psql.py -s mbslave\n    ./mbslave-solr-generate-triggers.py | ./mbslave-remap-schema.py | ./mbslave-psql.py -s mbslave\n\nUpdate the index:\n\n    ./mbslave-solr-update.py\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgravityblast%2Fmbslave","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgravityblast%2Fmbslave","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgravityblast%2Fmbslave/lists"}