https://github.com/Deep-Symmetry/crate-digger

Java library for fetching and parsing rekordbox exports and track analysis files.
https://github.com/Deep-Symmetry/crate-digger

java-library network pioneer rekordbox

Last synced: 17 days ago
JSON representation

Java library for fetching and parsing rekordbox exports and track analysis files.

• Host: GitHub
• URL: https://github.com/Deep-Symmetry/crate-digger
• Owner: Deep-Symmetry
• License: other
• Created: 2018-11-06T04:20:07.000Z (about 6 years ago)
• Default Branch: main
• Last Pushed: 2024-08-13T03:54:41.000Z (3 months ago)
• Last Synced: 2024-08-13T04:55:14.315Z (3 months ago)
• Topics: java-library, network, pioneer, rekordbox
• Language: Java
• Homepage:
• Size: 2.22 MB
• Stars: 128
• Watchers: 15
• Forks: 17
• Open Issues: 6
• Metadata Files:
  • Readme: README.adoc
  • Changelog: CHANGELOG.md
  • Contributing: CONTRIBUTING.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

README

        
= Crate Digger

James Elliott 

:icons: font

:experimental:

// Set up support for relative links on GitHub, and give it

// usable icons for admonitions, w00t! Add more conditions

// if you need to support other environments and extensions.

ifdef::env-github[]

:outfilesuffix: .adoc

:tip-caption: :bulb:

:note-caption: :information_source:

:important-caption: :heavy_exclamation_mark:

:caution-caption: :fire:

:warning-caption: :warning:

endif::env-github[]

+++



+++

image:https://img.shields.io/badge/chat-on%20zulip-brightgreen[Chat on Zulip,link="https://deep-symmetry.zulipchat.com/#narrow/stream/275855-dysentery-.26-crate-digger"]

A Java library for fetching and parsing rekordbox media exports and

track analysis files.

image:https://img.shields.io/badge/License-Eclipse%20Public%20License%202.0-blue.svg[link="#licenses"]

or secondary options

image:https://img.shields.io/badge/License-Mozilla%20Public%20License%202.0-blue.svg[link="#licenses"]

image:https://img.shields.io/badge/License-LGPL%203-blue.svg[link="#licenses"]

This project uses the http://kaitai.io[Kaitai Struct] compiler with

the help of a https://github.com/valery1707/kaitai-maven-plugin[Maven

plugin] to create classes that can parse and output binary data

structures in a convenient and efficient way. It uses them to create

Java classes, because it was created to support

https://github.com/Deep-Symmetry/beat-link[Beat Link], but other

projects can use them to output code for other languages.

It also uses the `jrpcgen` tool (which is part of the

https://sourceforge.net/projects/remotetea/[Remote Tea project]),

via another plugin, to generate classes that know how to talk to the

non-standard NFSv2 file servers running in Link-capable players, so

the rekordbox data can be reliably obtained even during big shows

where all four players are in use.

== Getting Help

+++



+++

Deep Symmetry's projects are generously sponsored with hosting by

https://zulip.com[Zulip], an open-source modern team chat app designed

to keep both live and asynchronous conversations organized. Thanks to

them, you can

https://deep-symmetry.zulipchat.com/#narrow/stream/275855-dysentery-.26-crate-digger[chat

with our community], ask questions, get inspiration, and share your

own ideas.

== PDB Database

The file link:src/main/kaitai/rekordbox_pdb.ksy[`rekordbox_pdb.ksy`] contains

the structure definitions needed to parse exported rekordbox databases

(`export.pdb` files).

NOTE: Huge thanks to https://github.com/flesniak[Fabian Lesniak] for

figuring out the details of how to interpret these files in his

https://github.com/flesniak/python-prodj-link[python-prodj-link]

project and https://github.com/GreyCat[Mikhail Yakshin] for helping me

quickly learn the more subtle aspects of Kaitai Struct. And this was

all started by a question

https://reverseengineering.stackexchange.com/users/4599/evan-purkhiser[Evan

Purkhiser] posted on

https://reverseengineering.stackexchange.com/questions/4311/help-reversing-a-edb-database-file-for-pioneers-rekordbox-software[Stack

Exchange].

There is an

https://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html[Export

Structure Analysis] site describing the details of what we have

learned about these file formats. Reading that will help make sense of

the exploration tools and the objects returned by this library.

[[exploring-analysis]]

=== Exploring the Analysis

One of the amazingly cool things about Kaitai Struct is that you can

use its https://ide.kaitai.io/#[Web IDE] to see how the structure

definitions work and visually explore the contents of files you are

analyzing. This also means you can look inside your own `.pdb` files

and check my work, or get a better understanding of how to use the

generated parsers. To do that, simply upload the `.pdb` file you want

to examine to the Web IDE (it doesn't actually go to the web, it just

gets put in your local browser storage), then also upload my

link:src/main/kaitai/rekordbox_pdb.ksy[`rekordbox_pdb.ksy`] file, and

the Web IDE will parse the exported database, letting you explore the

structures in the tree view, and see the corresponding raw bytes in

the hex viewer.

TIP: You can find the `export.pdb` file on a media stick prepared by

rekordbox inside the `PIONEER` folder, which may be invisible in the

Finder, but you can open it using Terminal commands if you have to. To

download my

link:src/main/kaitai/rekordbox_pdb.ksy[`rekordbox_pdb.ksy`], click on

its link, then click on the `Raw` button in the header above the first

line of the listing, then tell your browser to save it to disk. Be

sure to keep the `.ksy` extension. Then you can upload it to the

Kaitai Struct Web IDE.

== ANLZ Data

Each track in a rekordbox database also has `ANLZnnnn.DAT` and

`ANLZnnnn.EXT` files associated with it, containing the beat grid, an

index allowing rapid seeking to any time in variable-bit rate audio

files, the waveforms, memory cues and loop points. The paths to these

files are found inside the corresponding track record.

The structure definitions for these files are in

link:src/main/kaitai/rekordbox_anlz.ksy[`rekordbox_anlz.ksy`]. You can

use it with the Kaitai Struct Web IDE as described

<> to explore analysis files found in

your own exported media.

== Using the Library

Crate Digger is available through Maven Central, so to use it in your

Maven project, all you need is to include the appropriate dependency.

++++++

Click the **maven central** badge above to view the repository entry

for crate-digger. The proper format for including the latest release as a

dependency in a variety of tools, including Leiningen if you are using

beat-link from Clojure, can be found in the **Dependency Information**

section.

There are two halves to what Crate Digger offers. The first is an

ability to talk to the nonstandard Network File System servers that

are running in Pioneer players, and ask them to deliver the rekordbox

data export and track analysis files (programs like Beat Link

need these files to provide smooth integrations with the music being

performed). The second half is the ability to parse the contents of

those files, as described above.

=== Retrieving Files

The class `org.deepsymmetry.cratedigger.FileFetcher` is a singleton,

so to work with it you will start by calling `getInstance()`, as is

customary. Retrieving a file is then as simple as this:

[source,java]

----

FileFetcher fetcher = FileFetcher.getInstance();

fetcher.fetch(playerAddress, mountPath, sourcePath, destination);

----

`playerAddress` is an `InetAddress` object holding the address of the

player from which you want to download a file. `mountPath` identifies

the media slot you want to get information from, as shown in the table

below. `sourcePath` is the path to the specific file you want within

the mounted media, and `destination` is a `File` object identifying

where you want the downloaded data to be stored.

.Media Slot Mount Paths

|===

|Media Slot | Mount Path

|SD

|`/B/`

|USB

|`/C/`

|===

[IMPORTANT]

====

The `FileFetcher` caches information about players to make

requests more efficient, so it is important for you to tell it when a

player goes away, or unmounts one of its media slots, by calling:

[source,java]

----

fetcher.removePlayer(playerAddress);

----

====

=== Parsing Structures

The class `org.deepsymmetry.cratedigger.Database` provides support for

accessing the contents of rekordbox database export files. You can create

an instance to wrap a `File` instance that contains such an export (for

example one that you downloaded using the `fetch` method above). Then

you can query it for track and other information:

[source,java]

----

Database database = new Database(downloadedFile);

RekordboxPdb.TrackRow track = database.findTrack(1);

System.out.println(database.getText(track.title()));

----

Strings (like titles, artist names, etc.) are represented by a variety

of structures with different encodings, so a `getText()` method is

provided to convert them into ordinary Java strings.

See the http://deepsymmetry.org/cratedigger/apidocs/[API

documentation] for more details about these classes, and the

https://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html[Export

Structure Analysis] for more details about the file formats.

== Logging

Crate Digger uses http://www.slf4j.org/manual.html[slf4j] to allow you

to integrate it with whatever Java logging framework your project is

using, so you will need to include the appropriate slf4j binding on

your class path.

== Unfinished Tasks

* There are still more tables to be figured out. `Columns` looks like

the list of things that can be searched by, so perhaps it will hold

some clues for how to find and use the index tables, which must exist

because it would be horribly slow for the players to do a linear scan

through the main sparse tables whenever they wanted a record.

* If we could figure out how to use the indices ourselves, we could

avoid having to load the whole file and index it ourselves.

== Building the source

As noted above, he Maven project uses a plugin to run the the

`jrpcgen` tool (which is part of the

https://sourceforge.net/projects/remotetea/[Remote Tea

project]) to generate Java classes to implement the

ONC RPC specifications found in `src/main/rpc`.

(These are used for communicating with the NFS servers

in CDJs.) It also uses the http://kaitai.io[Kaitai Struct Compiler]

through another plugin to generate Java classes that can parse the

rekordbox databases it downloads from the players, based on the

specifications found in `src/main/kaitai`.

These things happen for you automatically during the code generation

phase of the Maven build. If you want to use something other than

Maven, you will need to figure out how to configure and run the

tools yourself.

== Building the Structure Analysis

I started out using pdfLaTeX to write and format the document, but

then, at the recommendation of one of the Kaitai Struct developers,

switched to XeLaTeX in order to take advantage of newer features. But

over time some of the packages I was using, especially for tables,

became unsupported and started having issues. So this and the

dysentery project's protocol analysis document have been ported to

more modern Asciidoc source in the form of https://antora.org[Antora]

sites.

To re-create (and even improve on) the byte field diagrams I was able

to achieve in LaTeX, I ended up writing my own diagram generator,

https://github.com/Deep-Symmetry/bytefield-svg[bytefied-svg], which

runs as an Antora plugin with the help of David Jencks'

https://gitlab.com/djencks/asciidoctor-generic-svg-extension.js[generic-svg-extension].

This documentation site can be built alongside the dysentery project's

protocol analysis, by following the directions in

https://github.com/Deep-Symmetry/dysentery/blob/master/doc/#readme[that

project].

== Contributing

If you have ideas, discoveries, or even code you’d like to share,

that’s fantastic! Please take a look at the

link:CONTRIBUTING.md[guidelines] and get in touch!

== Licenses

++++++

Copyright © 2018–2024 http://deepsymmetry.org[Deep Symmetry, LLC]

Distributed under the

https://opensource.org/licenses/EPL-2.0[Eclipse Public License

2.0]. By using this software in any fashion, you

are agreeing to be bound by the terms of this license. You must not

remove this notice, or any other, from this software. A copy of the

license can be found in link:LICENSE[LICENSE] within this project.

Secondary Licenses: This Source Code may also be made available under

the following Secondary Licenses when the conditions for such

availability set forth in the Eclipse Public License, v. 2.0 are

satisfied: https://www.mozilla.org/en-US/MPL/2.0/[Mozilla Public

License 2.0], or https://opensource.org/licenses/LGPL-3.0[GNU Lesser

General Public License v. 3].

=== Library and Build Tool Licenses

https://sourceforge.net/projects/remotetea/[Remote Tea] is licensed under the

https://opensource.org/licenses/LGPL-2.0[GNU Library General

Public License, version 2].

The https://github.com/kaitai-io/kaitai_struct_compiler[Kaitai Struct

Compiler] is licensed under the

https://opensource.org/licenses/GPL-3.0[GNU General Public License,

version 3] and the Kaitai Java runtime embedded in crate-digger is

licensed under the https://opensource.org/licenses/MIT[MIT License].