{"id":27998486,"url":"https://github.com/illixion/distributed-jxl-converter","last_synced_at":"2026-05-05T20:41:19.922Z","repository":{"id":252992400,"uuid":"842130663","full_name":"illixion/distributed-jxl-converter","owner":"illixion","description":"PoC for bulk converting images to JXL with distribution of work across multiple servers","archived":false,"fork":false,"pushed_at":"2025-02-04T11:26:24.000Z","size":19,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-08T22:53:29.230Z","etag":null,"topics":["distributed","grpc","javascript","jpeg-xl","jpegxl","jxl","nodejs","rabbitmq"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/illixion.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-13T18:22:59.000Z","updated_at":"2025-02-04T11:26:27.000Z","dependencies_parsed_at":"2024-08-15T09:03:01.074Z","dependency_job_id":null,"html_url":"https://github.com/illixion/distributed-jxl-converter","commit_stats":null,"previous_names":["illixion/distributed-jxl-converter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/illixion%2Fdistributed-jxl-converter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/illixion%2Fdistributed-jxl-converter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/illixion%2Fdistributed-jxl-converter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/illixion%2Fdistributed-jxl-converter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/illixion","download_url":"https://codeload.github.com/illixion/distributed-jxl-converter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253160818,"owners_count":21863624,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed","grpc","javascript","jpeg-xl","jpegxl","jxl","nodejs","rabbitmq"],"created_at":"2025-05-08T22:53:33.739Z","updated_at":"2026-05-05T20:41:19.917Z","avatar_url":"https://github.com/illixion.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Distributed JXL Image Converter\n\nThis repository contains a proof-of-concept solution that I wrote to convert a few million images to JPEG-XL with work distributed across multiple computers using RabbitMQ and axios.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Architecture](#architecture)\n- [Prerequisites](#prerequisites)\n- [Configuration](#configuration)\n- [Usage](#usage)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Overview\n\nThis project demonstrates a distributed system for converting images to JXL format. It uses RabbitMQ for job queuing and axios for file transfer between the server and clients. As of writing this, there aren't any JPEG-XL encoders for NodeJS that support writing metadata, which is something that I wanted to fix with this project. This was done by calling `cjxl` via child_process with one instance per machine core, although since it doesn't support pipes in all versions, we also use a temp folder or a RAM disk on each worker.\n\nBoth client and server are simple to read and modify to use any other command that you'd like, for example if you want to use AVIF instead of JPEG-XL.\n\n## Architecture\n\n- **Server**: Reads image files from a specified directory, sends conversion jobs to RabbitMQ, and provides axios services for file transfer.\n- **Client**: Consumes jobs from RabbitMQ, requests the original image from the server, converts it to JXL format, and uploads the converted image back to the server.\n\n## Prerequisites\n\n- Node.js (\u003e= 14.x)\n- RabbitMQ\n- `cjxl` (JPEG XL encoder)\n\nTo setup RabbitMQ, follow instructions on the [official RabbitMQ page here](https://www.rabbitmq.com/docs/download), for example on macOS the commands would be:\n\n```sh\nbrew install rabbitmq\nrabbitmqctl enable_feature_flag all\n```\n\nYou'll also need to create a user for your clients using these commands:\n\n```sh\nrabbitmqctl add_user myuser mypassword\nrabbitmqctl set_permissions -p / myuser \".*\" \".*\" \".*\"\n```\n\n## Configuration\n\nRename the included `config.json.dist` to `config.json` and adjust the following settings:\n\n- rabbitmqUrl (string) — required (client \u0026 server)  \n  Example: \"amqp://user:pass@host\"  \n  Full AMQP connection URL used by clients/servers to connect to RabbitMQ.\n\n- queueName (string) — required (both)  \n  Example: \"image_conversion_jobs\"  \n  Name of the RabbitMQ queue used to send conversion jobs.\n\n- brokenFilesQueueName (string) — required (both)  \n  Example: \"broken_files\"  \n  Name of the queue where clients report files that failed processing.\n\n- imageDir (string) — required (server)  \n  Example: \"/path/to/source/folder\"  \n  Directory on the server containing original images. Originals are deleted on successful conversion.\n\n- restServerUrl (string) — required (client)  \n  Example: \"http://server.local:3000\"  \n  Base URL used by clients to GET original files (server provides /file/:name endpoint).\n\n- uploadRestServerUrl (string) — required (client)  \n  Example: \"http://server.local:3000\"  \n  Base URL used by clients to POST converted files to the server (/upload endpoint).\n\n- ramdiskDir (string) — required (client)  \n  Example: \"/mnt/RAMDisk\"  \n  Local temp folder (ideally a RAM disk) used by clients for intermediate files.\n\n- cjxlPath (string) — optional (client)  \n  Example: \"cjxl\" or \"/usr/local/bin/cjxl\"  \n  Path to the cjxl binary. Required only when imageProcessor is set to \"cjxl\".\n\n- djxlPath (string) — optional (server)  \n  Example: \"djxl\"  \n  Path to the djxl binary (used by server-side utilities if present).\n\n- extension (string) — required (both)  \n  Example: \"jxl\"  \n  Target file extension for converted images.\n\n- imageProcessor (string) — optional (client) — default \"sharp\"  \n  Allowed values: \"sharp\" or \"cjxl\"  \n  Selects the encoder used by clients:\n  - \"sharp\": use the sharp Node.js binding (libvips) to write JXL. Does not support writing JXL metadata.\n  - \"cjxl\": call the external cjxl binary (requires cjxlPath). Use this to preserve metadata / use native encoder.\n\n### Notes\n\n- When using \"cjxl\", ensure cjxlPath points to a working cjxl binary and that system resources (temp/RAM disk) are available, since cjxl uses files rather than pipes.  \n- ramdiskDir should be writable and fast to avoid disk wear and to maximize throughput.  \n- Keep rabbitmqUrl and network URLs reachable from clients.  \n- Sync any new config fields from config.json.dist to your config.json when upgrading.  \n\n### Example `config.json`\n\n```json\n{\n  \"rabbitmqUrl\": \"amqp://myuser:mypassword@server.local\",\n  \"queueName\": \"image_conversion_jobs\",\n  \"brokenFilesQueueName\": \"broken_files\",\n  \"imageDir\": \"/path/to/source/folder\",\n  \"restServerUrl\": \"http://server.local:3000\",\n  \"uploadRestServerUrl\": \"http://server.local:3000\",\n  \"ramdiskDir\": \"/mnt/RAMDisk\",\n  \"cjxlPath\": \"cjxl\",\n  \"djxlPath\": \"djxl\",\n  \"extension\": \"jxl\",\n  \"imageProcessor\": \"sharp\"\n}\n```\n\n## Usage\n\n### Server\n\n1. Install dependencies:\n\n   ```bash\n   npm install\n   ```\n\n2. Start the server:\n\n   ```bash\n   node server.js\n   ```\n\n### Client\n\n1. Install dependencies:\n\n   ```bash\n   npm install\n   ```\n\n2. Start the client:\n\n   ```bash\n   node client.js\n   ```\n\n\n## Contributing\n\nContributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes.\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fillixion%2Fdistributed-jxl-converter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fillixion%2Fdistributed-jxl-converter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fillixion%2Fdistributed-jxl-converter/lists"}