{"id":17397841,"url":"https://github.com/bltavares/colmeia","last_synced_at":"2025-04-15T04:22:58.451Z","repository":{"id":53737721,"uuid":"231459452","full_name":"bltavares/colmeia","owner":"bltavares","description":"Attempt to make an interop layer to connect to dat on hyperswarm in Rust","archived":false,"fork":false,"pushed_at":"2021-05-23T05:26:44.000Z","size":667,"stargazers_count":24,"open_issues_count":7,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-28T15:51:47.067Z","etag":null,"topics":["dat","hypercore","hypercore-protocol","p2p"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bltavares.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-02T21:05:05.000Z","updated_at":"2023-08-29T02:42:17.000Z","dependencies_parsed_at":"2022-08-26T18:50:20.830Z","dependency_job_id":null,"html_url":"https://github.com/bltavares/colmeia","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bltavares%2Fcolmeia","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bltavares%2Fcolmeia/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bltavares%2Fcolmeia/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bltavares%2Fcolmeia/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bltavares","download_url":"https://codeload.github.com/bltavares/colmeia/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249004610,"owners_count":21196905,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dat","hypercore","hypercore-protocol","p2p"],"created_at":"2024-10-16T14:16:53.923Z","updated_at":"2025-04-15T04:22:58.430Z","avatar_url":"https://github.com/bltavares.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# colmeia\n\n\u003ca href=\"https://github.com/bltavares/colmeia/actions?query=workflow%3AQuickstart+branch%3Amaster\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/workflow/status/bltavares/colmeia/Quickstart/master?label=main%20ci\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"https://github.com/bltavares/colmeia/actions?query=workflow%3ACross-compile+branch%3Amaster\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/workflow/status/bltavares/colmeia/Cross-compile/master?label=cross%20ci\" /\u003e\n\u003c/a\u003e\n\n----\nHive (in portuguese). Attempt to make an interop layer to connect to [dat](https://github.com/datrs/) on [hyperswarm](https://github.com/hyperswarm) and legacy infra as well. Vaporware and might never be finished. Contributions welcome.\n\n## Goals\n\nWrite a pure rust network stack compatible with dat, to be able to run it on desktop (Win/Mac/Linux), mobile (and maybe wasm).\n\n## Milestones\n\n- [x] Generate a binary that finds and talk to a LAN `dat` node: handshake and disconnect\n- [x] Compile to Android\n- [ ] **next** Create a connection pool that tracks dat peers, and track hypercore version to allow crossing bridges\n- [ ] Bundle the static library into a Flutter app (**validated as viable already with dat (legacy)**) that displays the connection pool of dat peers for a given dat url (needs work)\n- [x] **(strech goal)** Support routers (MIPS)\n- [ ] *(stretch goal)* **validated as viable already with dat (legacy)**  Write a Flutter app (micro-app as in a  micro-service) that syncs and share files, allowing to build other local-first apps without needing to bundle the network stack logic (like [dat-desktop](https://github.com/dat-land/dat-desktop) but for mobile and desktop without Node)\n- [ ] *(stretch-er goal)* WASM and websocket integration\n- [ ] *(stretch-er goal)* **validated as viable already with dat (legacy)** FFI bindings to be able to link into an iOS app and other languages ([neon](https://neon-bindings.com/)?/[helix](https://usehelix.com/)?/[dart ffi](https://users.rust-lang.org/t/ffi-support-in-dart/32375)?).\n\n## References\n\n### [hyperswarm](https://github.com/hyperswarm) \u0026 [hypercore-protocol](https://hypercore-protocol.org/) compatibility\n\nUses NOISE protocol and a different handshake.\n\n#### Modules\n\n#### Discovery\n\n- [x] `colmeia-hyperswarm-mdns`: Support to hyperswarm mdns infrastructure: based on [hyperswarm/discovery](https://github.com/hyperswarm/discovery/)\n  - [x] `Locator`: stream to find dat members in the network\n  - [x] `Announcer`: stream that announces a dat in the network\n  - [x] `Mdns`: announces and find dat in the network\n  - [ ] Tests\n  - [ ] All protocol 1:1\n- [ ] `colmeia-dht`: Interop with hypwerswarm dht infrastructure (:eyes: \u003chttps://github.com/mattsse/hyperswarm-dht\u003e)\n  - [ ] Tests\n  - [ ] All protocol 1:1\n\n#### Protocol\n\n- [x] **wip** `colmeia-hypercore`: Networked module for hypercore storage\n  - [x] Hypercore replicate logic\n  - [ ] Tests\n  - [ ] All protocol 1:1\n- [x] **wip** `colmeia-hyperdrive`: Networked implementation of content and metadata hypercore's\n  - [x] `Hyperdrive`: Use metadata to clone content hypercore feed ([first block is content public key](https://github.com/hypercore-protocol/hyperdrive/blob/v10.13.0/index.js#L186))\n  - [x] In memory\n  - [ ] Write to disk\n  - [x] Impl remaining replicate logic for Hyperdrives\n  - [ ] Tests\n  - [ ] All protocol 1:1\n- [x] **wip** `colmeia-hyperstack`: Discovery integration of hyperdrives\n  - [x] Create a `Hypercore` struct that discovers and creates peeredfeed interactions\n  - [ ] Tests\n\n\n#### Reference tools\n\n- [hyperdrive-daemon](https://github.com/andrewosh/hyperdrive-daemon)\n- [hyperdrive v10](https://github.com/mafintosh/hyperdrive)\n- [snow noise protocol](https://snow.rs/)\n- [hypercore-protocol-rust-experiments](https://github.com/Frando/hypercore-protocol-rust-experiments)\n- [hypercore-protocol-rs](https://github.com/Frando/hypercore-protocol-rs)\n- [Beaker Browser](https://beakerbrowser.com/)\n\n#### Versions for reference\n\n- [hypercore v10.13.0](https://github.com/hypercore-protocol/hyperdrive/blob/v10.13.0/index.js)\n- [hypercore-protocol spec](https://hypercore-protocol.org/)\n- **TDB**\n\n## CLI binaries\n\n```txt\nLocal dat2: be41e3d43d054982e14dfc60281d9d4425ab5d4b0b280a355b7927869ca08fc5\n```\n\n### Find a local peer\n\n[colmeia-mdns](./colmeia-bins/src/bin/colmeia-hyperswarm-mdns.rs)\n\n```sh\nRUST_LOG=debug cargo run --bin colmeia-hyperswarm-mdns -- 7e5998407b3d9dbb94db21ff50ad6f1b1d2c79e476fbaf9856c342eb4382e7f5 8000\n```\n\n### Connect and troubleshoot a peer\n\n[colmeia-nc](./colmeia-bins/src/bin/colmeia-nc.rs)\n\n```sh\nRUST_LOG=debug cargo run --bin colmeia-nc -- 7e5998407b3d9dbb94db21ff50ad6f1b1d2c79e476fbaf9856c342eb4382e7f5 127.0.0.1:3282\n```\n\n### Clone from a single host\n\n```sh\nRUST_LOG=debug cargo run --bin colmeia-clone -- 192.168.15.173:3282 dat://6268b99fbacacea49c6bc3d4776b606db2aeadb3fa831342ba9f70d55c98929f\n```\n\n## Platforms\n\n:warning: **TODO**: redo support as part of dat -\u003e hypercore migration\n\n### Android Direct compilation\n\n- **No root required**\n\nInside termux, install `git` and `rust`:\n\n```sh\napk install git\napk install rust\ngit clone https://github.com/bltavares/colmeia\ncd colmeia\n\ncargo run # your desired bin\n```\n\n### Cross Compilation\n\n[It works!](https://twitter.com/bltavares/status/1221587189668163584)\n\n- **No root required**\n- Windows: Using WSL2 (windows insider) allows you to use Docker from WSL. [Follow this guide to enable docker](https://docs.docker.com/docker-for-windows/wsl-tech-preview/)\n- Mac/Linux: It should just need to have docker configured properly\n\n- Install [cross](https://github.com/rust-embedded/cross#installation)\n\n- Cross compile to your target:\n\n```sh\ncross build --target armv7-linux-androideabi\n```\n\n- To find the correct target for your device: `adb shell uname -m`\n\n| arch             | target                  |\n|------------------|-------------------------|\n| armv7l           | armv7-linux-androideabi |\n| aarch64 or arm64 | aarch64-linux-android   |\n| arm*             | arm-linux-androideabi   |\n| mips-musl        | mips-unknown-linux-musl |\n\n*arm: ([seems to be broken](https://internals.rust-lang.org/t/what-is-the-current-status-of-arm-linux-androideabi/4507/7))\n\n- Push to Android on a writeable location. Android does not allow binaries to be executed from the `/sdcard`:\n\n```sh\n# Once\nadb shell mkdir -p /data/local/tmp/colmeia\nadb shell chmod 777 /data/local/tmp/colmeia # Make it readable to let Termux read it as well\n\nadb push ./target/armv7-linux-androideabi/debug/colmeia-mdns /data/local/tmp/colmeia\n```\n\n### Android ADB shell\n\n```sh\nadb shell # open shell in android\nchmod +x /data/local/tmp/colmeia/colmeia-server\n\nRUST_LOG=debug  /data/local/tmp/colmeia/colmeia-server 0.0.0.0:8787 dat://460f04cf12c3b9833e5a0d3dd8eea05eab59dd8c1438a7454afe9630b9b4f8bd\n```\n\n### [Termux](https://termux.com/) from inside Android\n\nAndroid don't allow execution from outside the app data dir, and you can't push data to other apps dir from `adb push`. You need to copy the binary to your root folder, which should be executable.\n\nOpen the app and copy the content to the home folder:\n\n```sh\n# Inside termux shell\ncp /data/local/tmp/colmeia/colmeia-server ~\n\nRUST_LOG=debug ~/colmeia-server 0.0.0.0:8787 dat://460f04cf12c3b9833e5a0d3dd8eea05eab59dd8c1438a7454afe9630b9b4f8bd\n```\n\n### OpenWRT\n\nIt compiles with OpenWRT. Tested with 2 different routers on `18.06.1` and `19.07.2`.\n\nRun and `scp` the binaries on `target/mips-unknown-linux-musl`:\n\n```sh\n cross build --target mips-unknown-linux-musl\n ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbltavares%2Fcolmeia","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbltavares%2Fcolmeia","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbltavares%2Fcolmeia/lists"}