{"id":15497158,"url":"https://github.com/jwerle/dat-shaka-packager","last_synced_at":"2025-06-23T17:36:07.805Z","repository":{"id":148540496,"uuid":"190289636","full_name":"jwerle/dat-shaka-packager","owner":"jwerle","description":"Shaka Packager over the DAT network","archived":false,"fork":false,"pushed_at":"2019-06-06T16:15:33.000Z","size":36,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-04-22T21:49:48.586Z","etag":null,"topics":["dat","node","packager","service","shaka"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jwerle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-04T22:45:33.000Z","updated_at":"2024-03-20T22:29:52.000Z","dependencies_parsed_at":"2023-05-20T11:15:48.178Z","dependency_job_id":null,"html_url":"https://github.com/jwerle/dat-shaka-packager","commit_stats":{"total_commits":23,"total_committers":2,"mean_commits":11.5,"dds":0.04347826086956519,"last_synced_commit":"b199e074ac97d526cf142f2568ffec991f909096"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/jwerle/dat-shaka-packager","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwerle%2Fdat-shaka-packager","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwerle%2Fdat-shaka-packager/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwerle%2Fdat-shaka-packager/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwerle%2Fdat-shaka-packager/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jwerle","download_url":"https://codeload.github.com/jwerle/dat-shaka-packager/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwerle%2Fdat-shaka-packager/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261441157,"owners_count":23158465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dat","node","packager","service","shaka"],"created_at":"2024-10-02T08:31:12.631Z","updated_at":"2025-06-23T17:36:07.780Z","avatar_url":"https://github.com/jwerle.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"dat-shaka-packager\n==================\n\n\u003e [Shaka Packager](https://github.com/google/shaka-packager) over the DAT network.\n\n## Installation\n\n```sh\n$ npm install dat-shaka-packager\n```\n\n## Usage\n\nThe `dat-shaka-packager` module exposes a constructor for creating a\n[hypersource](https://github.com/jwerle/hypersource) server and a peer\nin the DAT network that uses\n[shaka-packager](https://github.com/google/shaka-packager) to package a\nset of inputs into a MPEG-DASH manifest and/or HLS playlist and provide\nthat output as an ephemeral archive returned in the same replication\nstream. The public key for a `dat-shaka-packager` peer is used for\ndiscovery and the initial replication stream. Ephemeral public keys are\nexchanged as user data (`userData`) in the\n[hypercore-protocol](https://github.com/mafintosh/hypercore-protocol)\nhandshake.\n\nSee _[protocol](#protocol)_ for more details on how this works.\n\n### Hypersource Client\n\nThe `dat-shaka-packager` builds on\n[hypersource](https://github.com/jwerle/hypersource) and therefore works\nwell with the client command line tool\n[`hsurl`](https://github.com/jwerle/hypersource-client).\n\nIf you know the endpoint that a node is running on you can send and\nreceive DAT archives easily.\n\n```sh\n$ hsurl wss://some-dat-shaka-packager-service.com \\\n  -i /path/to/dat/archive/input \\\n  -o /path/to/dat/archive/output\n```\n\n## Example\n\n### Create A Node\n\n```js\nconst packager = require('dat-shaka-packager')\nconst crypto = require('crypto')\n\n// An ephemeral \"public key\" without a secret key that can be\n// used for peer discovery and as the first key in a hypercore\n// replication stream. You'll likely want to just give this to a\n// hyperdrive instance\nconst key = crypto.randomBytes(32)\nconst node = packager({\n  discovery: { key },\n  storage: {\n    cache: '/path/to/packager/cache',\n    tmp: '/path/to/packager/tmpdir',\n  },\n})\n```\n\n### Connecting To A Node\n\nThe example below connects to a peer with a known public key, gets a\nephemeral response public key from the replication user data, and\nreplicates that response from the stream tied to that public key. The\nresponse is the packaged output from the input\n\nSee [jwerle/dat-shaka-packager-example](\nhttps://github.com/jwerle/dat-shaka-packager-example) for a complete\nexample.\n\n```js\nconst hyperdrive = require('hyperdrive')\nconst protocol = require('hypercore-protocol')\nconst storage = require('dat-storage')\nconst crypto = require('hypercore-crypto')\nconst rimraf = require('rimraf')\nconst swarm = require('discovery-swarm')\nconst path = require('path')\nconst ram = require('random-access-memory')\n\nconst TIMEOUT = 1000\nconst dirname = './input'\nconst key = '3546e77a133f01721058fb96cd9034d2cbf1e665eb040455e6eec614a8206bb7'\n\nfunction connect(key, dirname, callback) {\n  const opts = require('dat-swarm-defaults')({ hash: false, stream: onstream })\n  const input = hyperdrive(storage(dirname), { latest: true })\n  const discovery = swarm(opts)\n  const discoveryKey = crypto.discoveryKey(Buffer.from(key, 'hex'))\n\n  let connected = false\n  let retries = 3\n\n  input.ready(onready)\n\n  discovery.on('error', onerror)\n  input.on('error', onerror)\n\n  function onerror(err) {\n    connected = false\n    if (0 === --retries) {\n      callback(err)\n    }\n  }\n\n  function onready() {\n    discovery.join(discoveryKey)\n  }\n\n  function onstream() {\n    const stream = protocol({ live: true, userData: input.key })\n    stream.feed(Buffer.from(key, 'hex'))\n    stream.once('handshake', onhandshake)\n    stream.once('error', callback)\n    return stream\n\n    function onhandshake() {\n      if (connected) {\n        return stream.finalize()\n      }\n\n      connected = true\n      const key = stream.remoteUserData.toString('hex')\n      const output = path.join(`./tmp/${key}`)\n      const response = hyperdrive(storage(output), key, { latest: true })\n\n      let timeout = setTimeout(ontimeout, TIMEOUT)\n\n      response.replicate({ stream, live: true })\n      input.replicate({ stream, live: true })\n\n      response.on('error', callback)\n      response.on('sync', onsync)\n      response.on('update', () =\u003e {\n        clearTimeout(timeout)\n        timeout = setTimeout(ontimeout, TIMEOUT)\n      })\n\n      function onsync() {\n        clearTimeout(ontimeout)\n        stream.destroy()\n        discovery.close()\n        callback(null, response)\n      }\n\n      function ontimeout() {\n        connected = false\n        stream.destroy()\n        rimraf(output, (err) =\u003e err \u0026\u0026 console.error(err))\n      }\n    }\n  }\n}\n\nconnect(key, dirname, (err, res) =\u003e {\n  if (err) {\n    console.error('ERR', err.message)\n    return process.exit(1)\n  }\n\n  console.log('packaged %s at %s', dirname, key)\n  res.readdir('/', (err, files) =\u003e {\n    if (err) {\n      console.error('ERR', err.message)\n      return process.exit(1)\n    }\n\n    console.log(files)\n    process.nextTick(process.exit, 0)\n  })\n})\n```\n\n## Protocol\n\nThe `dat-shaka-packager` works by running a\n[hypersource](https://github.com/jwerle/hypersource) server and\nadvertising itself as a service in the DAT network that can be connected to\nwith the intent to send a DAT archive containing inputs and a manifest\nthat should be packaged with the [shaka-packager](\nhttps://github.com/google/shaka-packager).\n\nPeers in the DAT network can discover this service with modules like\n[discovery-swarm](https://github.com/mafintosh/discovery-swarm) using\n[dat-swarm-defaults](https://github.com/datproject/dat-swarm-defaults).\n\n### Discovery\n\n1. Join network based on `discoveryKey(key)` where `key` is the shared\n   public key for a `dat-shaka-packager` peer service\n2. Upon peer discovery, initiate a `live` replication stream with\n   `userData` based on `key` where `key` is the shared public key for a\n   `dat-shaka-packager` peer service and `userData` is the public key\n   for the input DAT archive that will be sent to the service. (_See [key\n   exchange][#key-exchange] for more information_)\n3. Wait for `'handshake'` event on the replication stream.\n\n### Key Exchange\n\nThe key exchange between peers is achieved by leveraging the `userData`\nfield in the [replication\nprotocol](https://github.com/mafintosh/hypercore-protocol). Peers\nconnect to a `dat-shaka-packager` service by leveraging a shared public\nkey to discovery and establish a secure connection between connecting parties.\n\nWhen Alice (peer) connects to Bob (service) over the DAT network, she\ncommunicates the public key of her DAT archive containing her inputs to\nBob through the user data field in the replication protocol. Bob generates\nan ephemeral key pair and communicates the public key to Alice the same\nway. The ephemeral public key is part of the key pair that is used as\nthe response DAT archive Bob will send to Alice.\n\n### Request Archive\n\nThe request (input) DAT archive is the file system containing all of the\ninputs that will be packaged by `shaka-packager`. The archive should\ncontain a `manifest.json` (_See [manifest][#manfiest] for more\ninformation_) that describes how the `shaka-packager` will operate. The\n`manifest.json` file is the first file read in the archive.\n\nThe structure of the archive should at least look something like:\n\n```\n/manifest.json\n/... # other input files\n```\n\nThe `dat-shaka-packager` will read the request archive \"sparsely\"\n(`sparse: true`) and in \"latest\" mode (`latest: true`) attempting to download\nfiles (specified in `manfiest.packager.streams`) specified in the\n`manifest.json` file.\n\nBelow is an example request archive that contains a single `video.mp4` that\ngenerates a MPEG-DASH manifest file `manifest.mpd`\n\n```json\n{\n  \"packager\": {\n    \"mpd_output\": \"manifest.mpd\",\n\n    \"streams\": [\n      {\n        \"in\": \"original.mp4\",\n        \"stream\": \"audio\",\n        \"output\": \"audio.mp4\",\n      },\n      {\n        \"in\": \"original.mp4\",\n        \"stream\": \"video\",\n        \"output\": \"video.mp4\",\n      }\n    ]\n  }\n}\n```\n\n### Response Archive\n\nThe response (output) DAT archive is the file system containing all of\nthe outputs that were generated by `shaka-packager`. For the given\nmanifest above the following output will be generated:\n\n```\n/audio.mp4\n/manifest.mpd\n/video.mp4\n```\n\n## API\n\n### `node = require('dat-shaka-packager')(opts)`\n\nCreates a new `dat-shaka-packager` node where `opts` can be:\n\n```js\n{\n  ignore: [], // An array of file name patterns to ignore in output\n  storage: {\n    cache: '/path/to/cache/dir', // Request archive cache directory\n    tmp: '/path/to/tmp/dir', // Response archive temporary directory\n  },\n  discovery: { // extends 'dat-swarm-defaults'\n    key: 'publicKey', // public key for DAT network so peers can discovery you\n  }\n}\n```\n\nIf you do not specify `opts.discovery.key` then the service will not\njoin the DAT network and is not discoverable through peer discovery.\nClients must connect to you directly.\n\n#### `node.listen(port[, address[, callback]])`\n\nStarts a [hypersource](https://github.com/jwerle/hypersource) server\nlistening on `port` at an optional `address` calling `callback` on error\nor success.\n\n#### `node.discovery`\n\nIf `opts.discovery` is not `false`, then this will be an instance of\n[discovery-swarm](https://github.com/mafintosh/discovery-swarm).\n\n#### `node.key`\n\nA `Buffer` that points back to the `opts.discovery.key` given in the\nconstructor.\n\n#### `node.discoveryKey`\n\nA `Buffer` of the computed discovery key for `node.key`\n\n## Manifest\n\nThe manifest (`manifest.json`) file serves as the arguments for the\n`dat-shaka-packager` service. It is a JSON file that converts its\nproperties into suitable arguments for the `shaka-packager` command line\nprogram. This section outlines some useful high level properties.\n\nBefore reading, you should be familiar with the\n[shaka-packager Documentation](https://google.github.io/shaka-packager/html/documentation.html)\n\n### `manifest.files`\n\nAn array of string patterns used to **white list** files into the output\ndirectory.\n\nThe following is an example of a white list that ensure only `.mp4`,\n`.mpd`, and `.m3u8` files are added to the output.\n\n```json\n...\n  \"files\": [\n    \"*.mp4\",\n    \"*.mpd\",\n    \"*.m3u8\"\n  ],\n...\n```\n\n### `manifest.ignore`\n\nAn array of string patterns used to **black list** files into the output\ndirectory.\n\nThe following is an example of a black list that ensures files with the\n`h264_(baseline|main|high)_(360p|480p|720p|1080p)_(600|1000|3000|6000).mp4`\npattern are not added to the output.\n\n```json\n...\n  \"ignore\": [\n    \"h264_(baseline|main|high)_(360p|480p|720p|1080p)_(600|1000|3000|6000).mp4\"\n  ],\n...\n```\n\n### `manifest.packager`\n\nThe `shaka-packager` configuration as a series of key-value pairs that\nmap directly to the [command line\narguments](https://google.github.io/shaka-packager/html/documentation.html)\n\nAll field names are preserved with the exception of the\n`manifest.packager.streams` and\n`manifest.packager.keys` fields which are transformed into the correct\ninput for the `shaka-packager` command line arguments.\n\nThe following `manifest.json`\n\n```json\n{\n  \"packager\": {\n    \"mpd_output\": \"manifest.mpd\",\n\n    \"streams\": [\n      {\n        \"in\": \"original.mp4\",\n        \"stream\": \"audio\",\n        \"output\": \"audio.mp4\",\n      },\n      {\n        \"in\": \"original.mp4\",\n        \"stream\": \"video\",\n        \"output\": \"video.mp4\",\n      }\n    ]\n  }\n}\n```\n\nis converted into the following command:\n\n```sh\n$ packager \\ \n  in=original.mp4,stream=audio,output=audio.mp4 \\ \n  in=original.mp4,stream=video,output=video.mp4 \\ \n  --mpd_output manifest.mpd\n```\n\n### `manifest.packager.streams[]`\n\nAn array of objects that describe stream inputs, outputs, and other\nmetadata that should be packaged. See [stream\ndescriptors](https://google.github.io/shaka-packager/html/documentation.html#stream-descriptors)\nfor a complete list of all the properties that can appear in this object.\n\n```json\n...\n    \"streams\": [\n      {\n        \"input\": \"original.mp4\",\n        \"stream\": \"video\",\n        \"output\": \"video.mp4\",\n        \"format\": \"MP4\",\n        \"drm_label\": \"audio\"\n      }\n    ]\n...\n```\n\n### `manifest.packager.keys[]`\n\nAn array of objects that map an encryption key to a stream. Below is an\nexample of an encryption key represented as a hex string\n`0884cf8e9445bf2d6e10e5b88a8661d3` for the `\"audio\"` stream specified by\nthe `label` property that maps to the stream specified by the\n`drm_label` properties in the `manfiest.packager.streams` array. The\nencryption key is uniquely identified by the hex string\n`a5308ea1375fb2f240d90fc29bad2c66`\n\n```json\n...\n    \"keys\": [\n      {\n        \"label\": \"audio\",\n        \"key_id\": \"a5308ea1375fb2f240d90fc29bad2c66\",\n        \"key\": \"0884cf8e9445bf2d6e10e5b88a8661d3\"\n      }\n    ]\n...\n```\n\n### Real World Manifest Example\n\nBelow is an example of a `manifest.json` that specifies how to generate\na [clear key encrypted](https://simpl.info/eme/clearkey/) MPEG-DASH manifest\nand a HLS playlist for the following:\n\n```json\n{\n  \"files\": [\n    \"*.mp4\",\n    \"*.mpd\",\n    \"*.m3u8\"\n  ],\n\n  \"ignore\": [\n    \"h264_(baseline|main|high)_(360p|480p|720p|1080p)_(600|1000|3000|6000).mp4\"\n  ],\n\n  \"packager\": {\n    \"enable_raw_key_encryption\": true,\n    \"generate_static_mpd\": true,\n    \"hls_master_playlist_output\": \"playlist.m3u8\",\n    \"mpd_output\": \"manifest.mpd\",\n\n    \"streams\": [\n      {\n        \"in\": \"h264_baseline_360p_600.mp4\",\n        \"stream\": \"audio\",\n        \"output\": \"audio.mp4\",\n        \"drm_label\": \"audio\"\n      },\n      {\n        \"in\": \"h264_baseline_360p_600.mp4\",\n        \"stream\": \"video\",\n        \"output\": \"h264_360p.mp4\",\n        \"drm_label\": \"sd-360p-primary1\"\n      },\n      {\n        \"in\": \"h264_main_480p_1000.mp4\",\n        \"stream\": \"video\",\n        \"output\": \"h264_480p.mp4\",\n        \"drm_label\": \"sd-480p-primary1\"\n      },\n      {\n        \"in\": \"h264_main_720p_3000.mp4\",\n        \"stream\": \"video\",\n        \"output\": \"h264_720p.mp4\",\n        \"drm_label\": \"hd-720p-primary1\"\n      },\n      {\n        \"in\": \"h264_high_1080p_6000.mp4\",\n        \"stream\": \"video\",\n        \"output\": \"h264_1080p.mp4\",\n        \"drm_label\": \"hd-1080p-primary1\"\n      }\n    ],\n\n    \"keys\": [\n      {\n        \"label\": \"audio\",\n        \"key_id\": \"a5308ea1375fb2f240d90fc29bad2c66\",\n        \"key\": \"0884cf8e9445bf2d6e10e5b88a8661d3\"\n      },\n      {\n        \"label\": \"sd-360p-primary1\",\n        \"key_id\": \"445da07a32a2c28de3ee0c00638708a3\",\n        \"key\": \"88a1127789efd85478f21467a7a524e5\"\n      },\n      {\n        \"label\": \"sd-480p-primary1\",\n        \"key_id\": \"6465a536e3a87ec28b594a2a8e20af17\",\n        \"key\": \"2d41d551367a30ed5b71f2dc4f22b802\"\n      },\n      {\n        \"label\": \"hd-720p-primary1\",\n        \"key_id\": \"da54aece6717796b7df729c75bdaa0f9\",\n        \"key\": \"b39c2faae9b3d1406cc06239425536e5\"\n      },\n      {\n        \"label\": \"hd-1080p-primary1\",\n        \"key_id\": \"85d23925199895d562df2edc3a730b2c\",\n        \"key\": \"57d9a65c5ce6ab56c468d26396074b82\"\n      }\n    ]\n  }\n}\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwerle%2Fdat-shaka-packager","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjwerle%2Fdat-shaka-packager","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwerle%2Fdat-shaka-packager/lists"}