{"id":13433126,"url":"https://github.com/Deep-Symmetry/crate-digger","last_synced_at":"2025-03-17T10:33:16.409Z","repository":{"id":34294542,"uuid":"156325284","full_name":"Deep-Symmetry/crate-digger","owner":"Deep-Symmetry","description":"Java library for fetching and parsing rekordbox exports and track analysis files.","archived":false,"fork":false,"pushed_at":"2025-02-27T02:10:00.000Z","size":2351,"stargazers_count":146,"open_issues_count":5,"forks_count":20,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-02-27T03:20:00.793Z","etag":null,"topics":["java-library","network","pioneer","rekordbox"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Deep-Symmetry.png","metadata":{"files":{"readme":"README.adoc","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"Deep-Symmetry","liberapay":"deep-symmetry","custom":"https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=LG5NLFL5T372W\u0026source=url"}},"created_at":"2018-11-06T04:20:07.000Z","updated_at":"2025-02-27T02:09:21.000Z","dependencies_parsed_at":"2024-01-19T10:57:30.824Z","dependency_job_id":"50b15794-c949-4250-ac23-10f2dcd1fe02","html_url":"https://github.com/Deep-Symmetry/crate-digger","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deep-Symmetry%2Fcrate-digger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deep-Symmetry%2Fcrate-digger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deep-Symmetry%2Fcrate-digger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deep-Symmetry%2Fcrate-digger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Deep-Symmetry","download_url":"https://codeload.github.com/Deep-Symmetry/crate-digger/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244016945,"owners_count":20384243,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java-library","network","pioneer","rekordbox"],"created_at":"2024-07-31T02:01:21.375Z","updated_at":"2025-03-17T10:33:16.385Z","avatar_url":"https://github.com/Deep-Symmetry.png","language":"Java","funding_links":["https://github.com/sponsors/Deep-Symmetry","https://liberapay.com/deep-symmetry","https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=LG5NLFL5T372W\u0026source=url"],"categories":["Java"],"sub_categories":[],"readme":"= Crate Digger\nJames Elliott \u003cjames@deepsymmetry.org\u003e\n:icons: font\n:experimental:\n\n// Set up support for relative links on GitHub, and give it\n// usable icons for admonitions, w00t! Add more conditions\n// if you need to support other environments and extensions.\nifdef::env-github[]\n:outfilesuffix: .adoc\n:tip-caption: :bulb:\n:note-caption: :information_source:\n:important-caption: :heavy_exclamation_mark:\n:caution-caption: :fire:\n:warning-caption: :warning:\nendif::env-github[]\n\n+++\n\u003cimg align=\"right\" width=\"275\" height=\"250\" alt=\"Record crate\"\nsrc=\"doc/assets/crate-padded-left.png\"\u003e\n+++\n\nimage:https://img.shields.io/badge/chat-on%20zulip-brightgreen[Chat on Zulip,link=\"https://deep-symmetry.zulipchat.com/#narrow/stream/275855-dysentery-.26-crate-digger\"]\n\nA Java library for fetching and parsing rekordbox media exports and\ntrack analysis files.\n\nimage:https://img.shields.io/badge/License-Eclipse%20Public%20License%202.0-blue.svg[link=\"#licenses\"]\nor secondary options\nimage:https://img.shields.io/badge/License-Mozilla%20Public%20License%202.0-blue.svg[link=\"#licenses\"]\nimage:https://img.shields.io/badge/License-LGPL%203-blue.svg[link=\"#licenses\"]\n\nThis project uses the http://kaitai.io[Kaitai Struct] compiler with\nthe help of a https://github.com/valery1707/kaitai-maven-plugin[Maven\nplugin] to create classes that can parse and output binary data\nstructures in a convenient and efficient way. It uses them to create\nJava classes, because it was created to support\nhttps://github.com/Deep-Symmetry/beat-link[Beat Link], but other\nprojects can use them to output code for other languages.\n\nIt also uses the `jrpcgen` tool (which is part of the\nhttps://sourceforge.net/projects/remotetea/[Remote Tea project]),\nvia another plugin, to generate classes that know how to talk to the\nnon-standard NFSv2 file servers running in Link-capable players, so\nthe rekordbox data can be reliably obtained even during big shows\nwhere all four players are in use.\n\n== Getting Help\n\n+++\n\u003ca href=\"http://zulip.com\"\u003e\u003cimg align=\"right\" alt=\"Zulip logo\"\n src=\"doc/assets/zulip-icon-circle.svg\" width=\"128\" height=\"128\"\u003e\u003c/a\u003e\n+++\n\nDeep Symmetry's projects are generously sponsored with hosting by\nhttps://zulip.com[Zulip], an open-source modern team chat app designed\nto keep both live and asynchronous conversations organized. Thanks to\nthem, you can\nhttps://deep-symmetry.zulipchat.com/#narrow/stream/275855-dysentery-.26-crate-digger[chat\nwith our community], ask questions, get inspiration, and share your\nown ideas.\n\n== PDB Database\n\nThe file link:src/main/kaitai/rekordbox_pdb.ksy[`rekordbox_pdb.ksy`] contains\nthe structure definitions needed to parse exported rekordbox databases\n(`export.pdb` files).\n\nNOTE: Huge thanks to https://github.com/flesniak[Fabian Lesniak] for\nfiguring out the details of how to interpret these files in his\nhttps://github.com/flesniak/python-prodj-link[python-prodj-link]\nproject and https://github.com/GreyCat[Mikhail Yakshin] for helping me\nquickly learn the more subtle aspects of Kaitai Struct. And this was\nall started by a question\nhttps://reverseengineering.stackexchange.com/users/4599/evan-purkhiser[Evan\nPurkhiser] posted on\nhttps://reverseengineering.stackexchange.com/questions/4311/help-reversing-a-edb-database-file-for-pioneers-rekordbox-software[Stack\nExchange].\n\nThere is an\nhttps://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html[Export\nStructure Analysis] site describing the details of what we have\nlearned about these file formats. Reading that will help make sense of\nthe exploration tools and the objects returned by this library.\n\n[[exploring-analysis]]\n=== Exploring the Analysis\n\nOne of the amazingly cool things about Kaitai Struct is that you can\nuse its https://ide.kaitai.io/#[Web IDE] to see how the structure\ndefinitions work and visually explore the contents of files you are\nanalyzing. This also means you can look inside your own `.pdb` files\nand check my work, or get a better understanding of how to use the\ngenerated parsers. To do that, simply upload the `.pdb` file you want\nto examine to the Web IDE (it doesn't actually go to the web, it just\ngets put in your local browser storage), then also upload my\nlink:src/main/kaitai/rekordbox_pdb.ksy[`rekordbox_pdb.ksy`] file, and\nthe Web IDE will parse the exported database, letting you explore the\nstructures in the tree view, and see the corresponding raw bytes in\nthe hex viewer.\n\nTIP: You can find the `export.pdb` file on a media stick prepared by\nrekordbox inside the `PIONEER` folder, which may be invisible in the\nFinder, but you can open it using Terminal commands if you have to. To\ndownload my\nlink:src/main/kaitai/rekordbox_pdb.ksy[`rekordbox_pdb.ksy`], click on\nits link, then click on the `Raw` button in the header above the first\nline of the listing, then tell your browser to save it to disk. Be\nsure to keep the `.ksy` extension. Then you can upload it to the\nKaitai Struct Web IDE.\n\nIf you want to look at an `exportExt.pdb` file, follow the procedure above but then also put the following content into an auxiliary file (named\nsomething like `redkordboxExt_pdb.ksy`) and drag it into the web IDE before\nyou drag in the database, that will tell the web IDE which variant of the\nfile it is working with:\n\n```yaml\nmeta:\n  id: rekordbox_pdb_ext\n  imports:\n    - rekordbox_pdb\nseq:\n  - id: root\n    type: rekordbox_pdb(true)\n```\n\n== ANLZ Data\n\nEach track in a rekordbox database also has `ANLZnnnn.DAT` and\n`ANLZnnnn.EXT` files associated with it, containing the beat grid, an\nindex allowing rapid seeking to any time in variable-bit rate audio\nfiles, the waveforms, memory cues and loop points. The paths to these\nfiles are found inside the corresponding track record.\n\nThe structure definitions for these files are in\nlink:src/main/kaitai/rekordbox_anlz.ksy[`rekordbox_anlz.ksy`]. You can\nuse it with the Kaitai Struct Web IDE as described\n\u003c\u003cexploring-analysis,above\u003e\u003e to explore analysis files found in\nyour own exported media.\n\n== Using the Library\n\nCrate Digger is available through Maven Central, so to use it in your\nMaven project, all you need is to include the appropriate dependency.\n\n+++\u003ca href=\"https://maven-badges.herokuapp.com/maven-central/org.deepsymmetry/crate-digger\"\u003e\u003cimg\n src=\"https://maven-badges.herokuapp.com/maven-central/org.deepsymmetry/crate-digger/badge.svg\"\n  alt=\"Maven Central\"\u003e\u003c/a\u003e+++\n\nClick the **maven central** badge above to view the repository entry\nfor crate-digger. The proper format for including the latest release as a\ndependency in a variety of tools, including Leiningen if you are using\nbeat-link from Clojure, can be found in the **Dependency Information**\nsection.\n\nThere are two halves to what Crate Digger offers. The first is an\nability to talk to the nonstandard Network File System servers that\nare running in Pioneer players, and ask them to deliver the rekordbox\ndata export and track analysis files (programs like Beat Link\nneed these files to provide smooth integrations with the music being\nperformed). The second half is the ability to parse the contents of\nthose files, as described above.\n\n=== Retrieving Files\n\nThe class `org.deepsymmetry.cratedigger.FileFetcher` is a singleton,\nso to work with it you will start by calling `getInstance()`, as is\ncustomary. Retrieving a file is then as simple as this:\n\n[source,java]\n----\nFileFetcher fetcher = FileFetcher.getInstance();\nfetcher.fetch(playerAddress, mountPath, sourcePath, destination);\n----\n\n`playerAddress` is an `InetAddress` object holding the address of the\nplayer from which you want to download a file. `mountPath` identifies\nthe media slot you want to get information from, as shown in the table\nbelow. `sourcePath` is the path to the specific file you want within\nthe mounted media, and `destination` is a `File` object identifying\nwhere you want the downloaded data to be stored.\n\n.Media Slot Mount Paths\n|===\n|Media Slot | Mount Path\n|SD\n|`/B/`\n\n|USB\n|`/C/`\n|===\n\n[IMPORTANT]\n====\nThe `FileFetcher` caches information about players to make\nrequests more efficient, so it is important for you to tell it when a\nplayer goes away, or unmounts one of its media slots, by calling:\n\n[source,java]\n----\nfetcher.removePlayer(playerAddress);\n----\n====\n\n=== Parsing Structures\n\nThe class `org.deepsymmetry.cratedigger.Database` provides support for\naccessing the contents of rekordbox database export files. You can create\nan instance to wrap a `File` instance that contains such an export (for\nexample one that you downloaded using the `fetch` method above). Then\nyou can query it for track and other information:\n\n[source,java]\n----\nDatabase database = new Database(downloadedFile);\nRekordboxPdb.TrackRow track = database.findTrack(1);\nSystem.out.println(database.getText(track.title()));\n----\n\nStrings (like titles, artist names, etc.) are represented by a variety\nof structures with different encodings, so a `getText()` method is\nprovided to convert them into ordinary Java strings.\n\nSee the http://deepsymmetry.org/cratedigger/apidocs/[API\ndocumentation] for more details about these classes, and the\nhttps://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html[Export\nStructure Analysis] for more details about the file formats.\n\n== Logging\n\nCrate Digger uses http://www.slf4j.org/manual.html[slf4j] to allow you\nto integrate it with whatever Java logging framework your project is\nusing, so you will need to include the appropriate slf4j binding on\nyour class path.\n\n== Unfinished Tasks\n\n* There are still more tables to be figured out. `Columns` looks like\nthe list of things that can be searched by, so perhaps it will hold\nsome clues for how to find and use the index tables, which must exist\nbecause it would be horribly slow for the players to do a linear scan\nthrough the main sparse tables whenever they wanted a record.\n\n* If we could figure out how to use the indices ourselves, we could\navoid having to load the whole file and index it ourselves.\n\n== Building the source\n\nAs noted above, he Maven project uses a plugin to run the the\n`jrpcgen` tool (which is part of the\nhttps://sourceforge.net/projects/remotetea/[Remote Tea\nproject]) to generate Java classes to implement the\nONC RPC specifications found in `src/main/rpc`.\n(These are used for communicating with the NFS servers\nin CDJs.) It also uses the http://kaitai.io[Kaitai Struct Compiler]\nthrough another plugin to generate Java classes that can parse the\nrekordbox databases it downloads from the players, based on the\nspecifications found in `src/main/kaitai`.\n\nThese things happen for you automatically during the code generation\nphase of the Maven build. If you want to use something other than\nMaven, you will need to figure out how to configure and run the\ntools yourself.\n\n== Building the Structure Analysis\n\nI started out using pdfLaTeX to write and format the document, but\nthen, at the recommendation of one of the Kaitai Struct developers,\nswitched to XeLaTeX in order to take advantage of newer features. But\nover time some of the packages I was using, especially for tables,\nbecame unsupported and started having issues. So this and the\ndysentery project's protocol analysis document have been ported to\nmore modern Asciidoc source in the form of https://antora.org[Antora]\nsites.\n\nTo re-create (and even improve on) the byte field diagrams I was able\nto achieve in LaTeX, I ended up writing my own diagram generator,\nhttps://github.com/Deep-Symmetry/bytefield-svg[bytefied-svg], which\nruns as an Antora plugin with the help of David Jencks'\nhttps://gitlab.com/djencks/asciidoctor-generic-svg-extension.js[generic-svg-extension].\n\nThis documentation site can be built alongside the dysentery project's\nprotocol analysis, by following the directions in\nhttps://github.com/Deep-Symmetry/dysentery/blob/master/doc/#readme[that\nproject].\n\n== Contributing\n\nIf you have ideas, discoveries, or even code you’d like to share,\nthat’s fantastic! Please take a look at the\nlink:CONTRIBUTING.md[guidelines] and get in touch!\n\n== Licenses\n\n+++\u003ca href=\"http://deepsymmetry.org\"\u003e\u003cimg src=\"doc/assets/DS-logo-github.png\"\n      align=\"right\" alt=\"Deep Symmetry logo\" width=\"250\" height=\"150\"\u003e\u003c/a\u003e+++\nCopyright © 2018–2024 http://deepsymmetry.org[Deep Symmetry, LLC]\n\nDistributed under the\nhttps://opensource.org/licenses/EPL-2.0[Eclipse Public License\n2.0]. By using this software in any fashion, you\nare agreeing to be bound by the terms of this license. You must not\nremove this notice, or any other, from this software. A copy of the\nlicense can be found in link:LICENSE[LICENSE] within this project.\n\nSecondary Licenses: This Source Code may also be made available under\nthe following Secondary Licenses when the conditions for such\navailability set forth in the Eclipse Public License, v. 2.0 are\nsatisfied: https://www.mozilla.org/en-US/MPL/2.0/[Mozilla Public\nLicense 2.0], or https://opensource.org/licenses/LGPL-3.0[GNU Lesser\nGeneral Public License v. 3].\n\n\n\n=== Library and Build Tool Licenses\n\nhttps://sourceforge.net/projects/remotetea/[Remote Tea] is licensed under the\nhttps://opensource.org/licenses/LGPL-2.0[GNU Library General\nPublic License, version 2].\n\nThe https://github.com/kaitai-io/kaitai_struct_compiler[Kaitai Struct\nCompiler] is licensed under the\nhttps://opensource.org/licenses/GPL-3.0[GNU General Public License,\nversion 3] and the Kaitai Java runtime embedded in crate-digger is\nlicensed under the https://opensource.org/licenses/MIT[MIT License].\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeep-Symmetry%2Fcrate-digger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDeep-Symmetry%2Fcrate-digger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeep-Symmetry%2Fcrate-digger/lists"}