{"id":20975249,"url":"https://github.com/functionland/fula-ota","last_synced_at":"2026-04-15T03:01:59.215Z","repository":{"id":142948694,"uuid":"583744892","full_name":"functionland/fula-ota","owner":"functionland","description":"This repository contains the fula dockers that runs on Linux and is responsible for managing the over the air update, blockchain and protocol services","archived":false,"fork":false,"pushed_at":"2026-04-10T03:32:55.000Z","size":283347,"stargazers_count":8,"open_issues_count":15,"forks_count":5,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-04-10T04:06:53.576Z","etag":null,"topics":["armbian","docker","linux","overtheair","raspberry-pi","watchtowerrr"],"latest_commit_sha":null,"homepage":"https://fx.land","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/functionland.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-12-30T19:18:51.000Z","updated_at":"2026-04-10T03:33:00.000Z","dependencies_parsed_at":"2025-12-07T00:05:36.493Z","dependency_job_id":null,"html_url":"https://github.com/functionland/fula-ota","commit_stats":null,"previous_names":[],"tags_count":158,"template":false,"template_full_name":null,"purl":"pkg:github/functionland/fula-ota","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/functionland%2Ffula-ota","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/functionland%2Ffula-ota/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/functionland%2Ffula-ota/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/functionland%2Ffula-ota/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/functionland","download_url":"https://codeload.github.com/functionland/fula-ota/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/functionland%2Ffula-ota/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31824118,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"online","status_checked_at":"2026-04-15T02:00:06.175Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["armbian","docker","linux","overtheair","raspberry-pi","watchtowerrr"],"created_at":"2024-11-19T04:41:29.547Z","updated_at":"2026-04-15T03:01:59.200Z","avatar_url":"https://github.com/functionland.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fula OTA\n\nOver-the-air update system for FxBlox devices and PCs. Manages Docker-based services for IPFS storage, cluster pinning, auto-pinning, and the Fula protocol on ARM64 hardware (RK3588/RK1) and x86_64 desktops (Windows/Linux/macOS).\n\n## Architecture Overview\n\nEach Fula node runs nine Docker containers orchestrated by `docker-compose`:\n\n```\n+---------------------+     +-------------------+     +---------------------+\n|     fxsupport       |     |       kubo        |     |    ipfs-cluster     |\n| (alpine, scripts)   |     | (ipfs/kubo:release|     | (functionland/      |\n|                     |     |  bridge network)  |     |  ipfs-cluster)      |\n| Carries all config  |     |                   |     |  host network       |\n| and scripts in      |     | Ports: 4001, 5001 |     |  Ports: 9094-9096   |\n| /linux/ directory   |     |  8080, 8081       |     |                     |\n+---------------------+     +---+---------------+     +---------------------+\n         |                       |           ^\n         | docker cp to         | IPFS       | pin/add, pin/ls\n         | /usr/bin/fula/       | block      |\n         v                      | storage  +-+-----------------+   +-------------------+\n+----------------------------+  |          |   fula-pinning    |   |   fula-gateway     |\n| /uniondrive (merged)      |  |          | (auto-pin daemon) |   | (S3-compatible     |\n| union-drive.sh (mergerfs) |  |          | host network      |   |  storage gateway)  |\n+----------------------------+  |          | Port: 3501        |   | host network       |\n         |                      |          +---+---+-----------+   | Port: 9000         |\n+--------+--------+     +------+------+        |   |               +--------+-----------+\n|    go-fula       |     |  kubo-local |   Syncs |   | registry.cid        |\n| (functionland/   |     | (ipfs/kubo  |   pins  |   +----\u003e shared file --+\n|  go-fula)        |     |  :release)  |   from  |         /internal/fula-gateway/\n|  host network    |     |  bridge     |  remote |\n|  Port: 40001     |     | Port: 5002  | pinning |\n+------------------+     +-------------+         |\n                         +---------------------+ |\n                         |  watchtower          | |\n                         |  auto-update         | |\n                         |  every 3600s         | |\n                         +---------------------+ |\n                         +---------------------+ |\n                         |  fula.service        | |\n                         |  (systemd)           | |\n                         |  runs fula.sh        | |\n                         +---------------------+ |\n```\n\n### Container Details\n\n| Container | Image | Network | Purpose |\n|-----------|-------|---------|---------|\n| `fula_fxsupport` | `functionland/fxsupport:release` | default | Carries scripts and config files. Acts as a delivery mechanism — `fula.sh` copies `/linux/` from this container to `/usr/bin/fula/` on the host. |\n| `ipfs_host` | `ipfs/kubo:release` | bridge | Main IPFS node. Stores blocks on `/uniondrive`. Config template at `kubo/config`, runtime config at `/home/pi/.internal/ipfs_data/config`. |\n| `ipfs_local` | `ipfs/kubo:release` | bridge | Local-only IPFS node for fula-gateway and fula-pinning. Separate IPFS repo at `/internal/ipfs_data_local`. Port 5002 (localhost only). |\n| `ipfs_cluster` | `functionland/ipfs-cluster:release` | host | IPFS Cluster follower node. Manages pin replication across the network. Config at `/uniondrive/ipfs-cluster/service.json`. |\n| `fula_go` | `functionland/go-fula:release` | host | Fula protocol: blockchain proxy (ports 4020/4021), libp2p blox (port 40001), WAP server (port 3500). |\n| `fula_pinning` | `functionland/fula-pinning:release` | host | Auto-pinning daemon. Syncs pins from a remote IPFS Pinning Service to local kubo. Writes `registry.cid` for fula-gateway. See [docker/fula-pinning/README.md](docker/fula-pinning/README.md). |\n| `fula_gateway` | `functionland/fula-gateway:release` | host | Local S3-compatible gateway. Serves the paired user's files from local kubo on port 9000 (LAN only). Auth via pairing secret from `box_props.json`. |\n| `fula_updater` | `containrrr/watchtower:latest` | default | Polls Docker Hub hourly for updated images, auto-restarts containers. |\n\n### Networking\n\n- **kubo** and **kubo-local** run in Docker bridge networking (ports mapped: `4001:4001`, `5001:5001`, `5002:5002`)\n- **go-fula**, **ipfs-cluster**, **fula-pinning**, and **fula-gateway** run with `network_mode: \"host\"` (direct host networking)\n- go-fula cannot reach kubo at `127.0.0.1` from host network — it uses the `docker0` bridge IP (typically `172.17.0.1`), auto-detected in `go-fula/blox/kubo_proxy.go`\n- **Docker Compose bridge caveat**: Compose creates its own bridge interfaces (`br-\u003chash\u003e`), not `docker0`. Firewall rules must match `-i br-+` (iptables wildcard) in addition to `-i docker0`.\n- **fula-gateway** reaches kubo at `127.0.0.1:5001` via host network (kubo's port mapping), serves S3 API on `0.0.0.0:9000` (LAN-only via firewall)\n- **PC installer networking**: All services run in bridge mode (`fula-net`) since Docker Desktop doesn't support host networking on Windows/macOS. kubo's LAN and public IP are injected into `Addresses.AppendAnnounce` so it advertises reachable addresses to the DHT.\n\n### Storage Layout\n\n```\n/uniondrive/                    # mergerfs mount (union of all attached drives)\n  ipfs_datastore/blocks/        # IPFS block storage (flatfs)\n  ipfs_datastore/datastore/     # IPFS metadata (pebble)\n  ipfs-cluster/                 # Cluster state, service.json, identity.json\n  ipfs_staging/                 # IPFS staging directory\n\n/home/pi/.internal/             # Device-internal state (Armbian)\n  ipfs_data/config              # Deployed kubo config (runtime)\n  ipfs_data_local/config        # Local kubo config (runtime)\n  ipfs_config                   # Template copy (used by initipfs)\n  config.yaml                   # Fula device config (pool, authorizer, etc.)\n  box_props.json                # Pairing credentials (JWT, secret, endpoint)\n  fula-gateway/registry.cid     # Bucket registry CID (written by fula-pinning)\n\n/usr/bin/fula/                  # On-host scripts and configs (copied from fxsupport)\n  fula.sh                       # Main orchestrator script\n  docker-compose.yml            # Container definitions\n  .env                          # Image tags (GO_FULA, FX_SUPPROT, IPFS_CLUSTER)\n  .env.cluster                  # Cluster env var overrides\n  kubo/config                   # Kubo config template\n  kubo/kubo-container-init.d.sh # Kubo init script\n  kubo-local/config-local       # Local kubo config template\n  kubo-local/kubo-local-container-init.d.sh  # Local kubo init script\n  ipfs-cluster/ipfs-cluster-container-init.d.sh  # Cluster init script\n  update_kubo_config.py         # Selective kubo config merger\n  union-drive.sh                # UnionDrive mount management\n  bluetooth.py                  # BLE command handler\n  local_command_server.py       # Local TCP command server\n  control_led.py                # LED control\n  readiness-check.py            # Health monitoring and auto-recovery\n  commands.sh                   # File-based command handler (reboot, LED, partition)\n  firewall.sh                   # iptables firewall rules\n  plugins/                      # Plugin system\n  ...\n```\n\n### Identity System\n\nEach Fula node has two peer IDs derived from a single private key:\n\n- **Cluster Peer ID**: The original `peer.IDFromPrivateKey(identity)` — used for on-chain pool membership and IPFS Cluster identity\n- **Kubo Peer ID**: HMAC-SHA256 derived with domain `\"fula-kubo-identity-v1\"` — used for IPFS kubo identity (prevents collision with cluster)\n\ngo-fula uses the cluster peer ID for `discoverPoolAndChain()` membership checks, while kubo uses the derived peer ID.\n\n### Auto-Pinning (fula-pinning)\n\nThe `fula-pinning` daemon replicates data from the remote Fula Storage API to this device's local IPFS node. The local blox is a **sync consumer only** — uploads go to the remote Fula Gateway S3 server, not to this device:\n\n```\nUser uploads via S3 API -\u003e remote Fula Gateway -\u003e stores in Gateway's Kubo + pins on Remote Pinning Service\n                                                                                       |\nfula-pinning daemon (this device) \u003c-- fetches pin list every 3 min (user's JWT) -------+\n        |\n        +---\u003e pins missing CIDs on local Kubo (fetches data via IPFS P2P network)\n```\n\n**User isolation**: The daemon uses the paired user's JWT (`auto_pin_token`) to query the pinning service. The service only returns pins belonging to that token, so the local kubo only pins the paired user's data.\n\n**Configuration**: Pairing credentials are stored in `/home/pi/.internal/box_props.json`:\n```json\n{\n  \"auto_pin_token\": \"user-jwt-token\",\n  \"auto_pin_endpoint\": \"https://api.pinata.cloud/psa\",\n  \"auto_pin_pairing_secret\": \"local-api-secret\"\n}\n```\n\nThe daemon monitors this file every 30 seconds — pair/unpair/rotate tokens without restarting the container. When unpaired (empty token/endpoint), the daemon idles.\n\n**Local HTTP API** (port 3501, requires `Bearer {pairing_secret}`):\n- `GET /api/v1/auto-pin/status` — pinned count, last/next sync times\n- `POST /api/v1/auto-pin/report-missing` — request immediate pinning of specific CIDs\n\nSee [docker/fula-pinning/README.md](docker/fula-pinning/README.md) for full documentation.\n\n### Local S3 Gateway (fula-gateway)\n\nThe `fula-gateway` container is a standalone Rust binary (`fula-local-gateway`) that provides an S3-compatible API for local file access. It uses `fula-core` and `fula-blockstore` crates from [fula-api](https://github.com/functionland/fula-api) as shared libraries, but contains no cloud code.\n\n```\nFxFiles (client-side encryption/decryption)\n    |\n    +-- Remote: S3 API -\u003e s3.cloud.fx.land:443 -\u003e remote fula-cli -\u003e remote kubo\n    |\n    +-- Local (LAN): S3 API -\u003e blox-ip:9000 -\u003e local fula-gateway -\u003e local kubo\n                                                     ^\n                                              registry.cid written by\n                                              fula-pinning daemon\n```\n\n**Key features**:\n- **Bearer-only auth**: Uses `auto_pin_pairing_secret` from `box_props.json`. When unpaired, auth is disabled (safe — port 9000 is LAN-only via firewall).\n- **User scoping**: Derives owner ID from BLAKE3 hash of JWT `sub` claim. All bucket operations are scoped to the paired user.\n- **Multipart uploads**: Full lifecycle support (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`). Creates unified DAG via `put_ipld()` for multi-part files.\n- **CID watcher**: Polls `/internal/fula-gateway/registry.cid` every 30 seconds, reloads bucket registry when CID changes.\n- **Content CID header**: Always returns `X-Fula-Content-Cid` header on object responses.\n- **Registry persistence**: Uses `persist_registry()` (no token) for local operation.\n- **mDNS**: go-fula advertises `s3Port=9000` in mDNS TXT records so FxFiles discovers the local gateway automatically.\n\n**Firewall**: Port 9000 is restricted to RFC1918 private addresses only (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12).\n\n### Health Monitoring (Armbian)\n\nThe `readiness-check.py` script runs as a systemd service and provides comprehensive health monitoring:\n\n- **Container health**: Monitors all 8 containers, detects crashes and error logs\n- **Config validation**: Detects YAML invalid control characters, auto-restores from backup if corrupted\n- **PeerID collision detection**: Detects and fixes kubo/cluster PeerID collisions by regenerating cluster identity\n- **Proxy health**: Checks go-fula proxy ports (4020, 40001) reachability\n- **Disk space**: Triggers `docker system prune` if \u003c1GB free\n- **Kubo config**: Strips deprecated Provider/Reprovider fields (kubo 0.40+)\n- **LED status**: Green=healthy, Yellow=restarting, Blue=restarted, Red=critical failure\n- **Auto-recovery**: Up to 4 restart attempts, then activates WireGuard fallback and triggers re-partition after 12+ hours of failure\n\n### Plugin System\n\nOptional plugins extend device functionality. Each plugin includes `install.sh`, `start.sh`, `stop.sh`, `uninstall.sh`, a `docker-compose.yml`, and an `info.json` manifest.\n\n| Plugin | Purpose | Requirements |\n|--------|---------|-------------|\n| **streamr-node** | Runs a Streamr node to earn $DATA token rewards | Port 32200 forwarding |\n| **loyal-agent** | Local AI agent using NPU (deepseek-llm-7b-chat model) | 32GB RAM, 10GB storage, ARM64 only |\n\nActive plugins are tracked in `/home/pi/active-plugins.txt`. The PC installer excludes hardware-specific plugins (e.g. `loyal-agent`).\n\n### Command Handler\n\nThe `commands.sh` script watches `/home/pi/commands/` via `inotifywait` for file-based commands:\n\n| Command File | Action |\n|-------------|--------|\n| `.command_partition` | Runs `resize.sh` for disk repartitioning |\n| `.command_repairfs` | Runs `repairfs.sh` for external storage filesystem repair |\n| `.command_led` | Sets LED color/duration (file content: `color duration`) |\n| `.command_reboot` | Triggers system reboot |\n\n## PC Installer\n\nThe `pc-installer/` directory contains an Electron desktop application that brings the full Fula node stack to Windows, Linux, and macOS PCs via Docker Desktop.\n\n### Key Differences from Armbian\n\n| Aspect | Armbian (fula.sh) | PC Installer (Electron) |\n|--------|-------------------|------------------------|\n| Runtime | Bash scripts on bare metal | Electron GUI + Node.js managers |\n| Docker networking | Mixed (host for go-fula, bridge for kubo) | All bridge (`fula-net`) |\n| Firewall | `iptables` rules | Windows Firewall rules via Squirrel hooks |\n| Storage paths | Fixed: `/home/pi`, `/media/pi` | User-chosen `dataDir` + `storageDir` |\n| Health monitoring | `readiness-check.py` (systemd) | Real-time `HealthMonitor` + tray icon colors |\n| mDNS | Python `advertisement.py` | Node.js `bonjour-service` with interface detection |\n| Hardware ID | ARM cpuinfo hash | SHA-256 of first non-internal MAC address |\n| UI | CLI only | Setup wizard (6 steps) + dashboard |\n\n### Architecture\n\n```\nsrc/main/\n  index.js              # Main entry, IPC handlers, lifecycle orchestration\n  constants.js          # Ports, container names, bootstrap peers\n  config-store.js       # Electron-store config persistence\n  docker-manager.js     # Docker Compose lifecycle, waitForDocker(), ghost container cleanup\n  health-monitor.js     # Continuous health checks, auto-recovery, Docker recovery\n  tray-manager.js       # System tray icon + context menu (color = health status)\n  storage-manager.js    # Directory tree init, template copying\n  mdns-advertiser.js    # mDNS advertisement via bonjour-service\n  update-manager.js     # Stale image detection\n  plugin-manager.js     # Plugin extraction from fxsupport container\n  logger.js             # Winston-based logging (file + console)\n\nsrc/renderer/\n  wizard/               # Setup wizard UI (terms, Docker check, storage, ports, pull, launch)\n  dashboard/            # Dashboard UI (containers, logs, health, plugins)\n```\n\n### Features\n\n- **Setup wizard**: 6-step guided setup (terms, Docker check, storage selection, port availability, image pull, launch)\n- **System tray**: Color-coded health status (green/yellow/red/blue/cyan/grey)\n- **Health monitor**: Continuous checks for container health, kubo API, relay connection, bootstrap peers, proxy ports, cluster errors, config validity, disk space, PeerID collision\n- **Docker Desktop recovery**: Detects unresponsive Docker daemon, kills and relaunches Docker Desktop, waits for recovery. Health monitor triggers recovery after 3 consecutive failures.\n- **mDNS**: Advertises `_fulatower._tcp` with device properties (bloxPeerIdString, authorizer, hardwareID, ipfsClusterID, s3Port). Polls go-fula `/properties` every 60s to update.\n- **kubo network hardening**: Injects LAN + public IP into `AppendAnnounce`, forces `Libp2pForceReachability=private` for relay-v2 (AutoNAT can't work in Docker bridge)\n- **Template sync**: `scripts/sync-shared-templates.js` copies kubo/cluster configs from Armbian sources, transforming bridge DNS names where needed (`127.0.0.1:5001` -\u003e `kubo:5001`)\n- **Windows install hooks**: Squirrel installer creates Start Menu shortcuts, adds Windows Firewall rules (mDNS UDP 5353, services TCP 3500/4001/9000/9094) via UAC-elevated PowerShell\n\n### Data Directory Layout (PC)\n\n```\n{dataDir}/                       # e.g. E:\\.fula or %LOCALAPPDATA%\\FulaData\n  config/\n    docker-compose.pc.yml        # Container orchestration\n    .env.pc                      # Image tags and paths\n    .env.cluster                 # Cluster identity and bootstrap\n    .env.gofula                  # go-fula env (HARDWARE_ID injected at runtime)\n    kubo/config                  # Kubo config template\n    kubo-local/config-local      # Local kubo config template\n    ipfs-cluster/                # Cluster init script\n  internal/\n    config.yaml                  # Fula device config\n    box_props.json               # Pairing credentials\n    ipfs_data/config             # Kubo runtime config\n    ipfs_data_local/config       # Local kubo runtime config\n    fula-gateway/registry.cid    # Bucket registry CID\n    plugins/                     # Extracted plugins\n  storage/                       # IPFS block storage (or separate storageDir)\n  logs/\n    fula-node.log                # Application logs\n```\n\n## Update Propagation Flow\n\nHow code changes in this repo reach devices:\n\n```\n1. Push to GitHub repo\n2. GitHub Actions builds Docker images (on release) OR manual build\n3. Images pushed to Docker Hub (functionland/fxsupport:release, etc.)\n4. Watchtower on device detects updated images (polls hourly)\n5. fula.service triggers: fula.sh start\n6. First restart() — runs with CURRENT on-device files\n7. docker cp fula_fxsupport:/linux/. /usr/bin/fula/ — copies NEW files from updated fxsupport image\n8. If fula.sh itself changed -\u003e second restart() — runs with NEW files\n```\n\n### Config Merge Flow (kubo)\n\nOn every `fula.sh start` (before `docker-compose up`):\n\n1. `update_kubo_config.py` (or inline fallback in `fula.sh`) runs\n2. Reads the **template** (`/usr/bin/fula/kubo/config`) and the **deployed** config (`/home/pi/.internal/ipfs_data/config`)\n3. Merges only **managed fields** (Bootstrap, Peering, Swarm, Experimental, Routing, etc.) from template into deployed config\n4. **Preserved fields** (Identity, Datastore paths, API/Gateway addresses) are never touched\n5. Dynamic `StorageMax` is calculated as 80% of `/uniondrive` total space (minimum 800GB floor)\n6. Writes updated deployed config, then runs `docker-compose up`\n\n### Config Flow (ipfs-cluster)\n\n1. `.env.cluster` is loaded by docker-compose as `env_file` (injects env vars into container)\n2. `.env.cluster` is also bind-mounted at `/.env.cluster` inside the container\n3. `ipfs-cluster-container-init.d.sh` runs as entrypoint:\n   - Waits for pool name from `config.yaml`\n   - Generates cluster secret from pool name\n   - Runs `jq` to patch `service.json` with connection_manager, pubsub, batching, timeouts, informer settings\n   - Starts `ipfs-cluster-service daemon` with bootstrap address\n\n## Repository Structure\n\n```\nfula-ota/\n  docker/\n    build_and_push_images.sh    # Builds all images and pushes to Docker Hub\n    env_release.sh              # ARM64 release env vars (image names, tags)\n    env_release_amd64.sh        # AMD64 release env vars\n    env_test.sh                 # ARM64 test env vars\n    env_test_amd64.sh           # AMD64 test env vars\n    run.sh                      # Local dev docker-compose runner\n    fxsupport/\n      Dockerfile                # alpine + COPY ./linux -\u003e /linux\n      build.sh                  # buildx build + push\n      linux/                    # All on-device scripts and configs\n        docker-compose.yml      # Container orchestration (9 services)\n        .env                    # Docker image tags\n        .env.cluster            # Cluster env var overrides\n        fula.sh                 # Main orchestrator (start/stop/restart/rebuild)\n        union-drive.sh          # mergerfs mount management\n        update_kubo_config.py   # Selective kubo config merger\n        readiness-check.py      # Health monitoring and auto-recovery (1500+ lines)\n        commands.sh             # File-based command handler (reboot, LED, partition)\n        firewall.sh             # iptables firewall rules\n        kubo/\n          config                # Kubo config template\n          kubo-container-init.d.sh\n        kubo-local/\n          config-local          # Local kubo config template\n          kubo-local-container-init.d.sh\n        ipfs-cluster/\n          ipfs-cluster-container-init.d.sh\n        bluetooth.py            # BLE setup and command handling\n        local_command_server.py # TCP command server\n        control_led.py          # LED control for device status\n        plugins/                # Plugin system (loyal-agent, streamr-node)\n        ...\n    fula-pinning/\n      Dockerfile                # Go build stage + alpine runtime\n      build.sh                  # buildx build + push\n      *.go                      # Daemon source: config, sync loop, HTTP server, kubo/pinning clients\n      README.md                 # Full service documentation\n    fula-gateway/\n      build.sh                  # buildx build from fula-local-gateway/\n      download_image.sh         # Pull pre-built image from Docker Hub\n      fula-local-gateway/       # Standalone Rust binary\n        Cargo.toml              # Uses fula-core + fula-blockstore as git deps\n        Dockerfile              # Rust build stage + debian runtime\n        src/\n          main.rs               # CLI, config loading, kubo wait loop\n          server.rs             # Axum router, CID watcher spawn\n          auth.rs               # Bearer-token middleware\n          box_props.rs          # JWT sub extraction, BLAKE3 owner ID\n          state.rs              # AppState, IPFS connection, CID watcher\n          handlers/             # S3 API handlers (bucket, object, multipart, service)\n          ...\n    go-fula/\n      Dockerfile                # Go build stage + alpine runtime\n      build.sh                  # Clones go-fula repo, buildx build + push\n      go-fula.sh                # Container entrypoint\n    ipfs-cluster/\n      Dockerfile                # Go build stage + alpine runtime\n      build.sh                  # Clones ipfs-cluster repo, buildx build + push\n  pc-installer/                 # Electron desktop app for Windows/Linux/macOS\n    package.json                # Electron 40, electron-forge\n    forge.config.js             # Build config (Squirrel/DEB/ZIP)\n    scripts/\n      sync-shared-templates.js  # Copies Armbian configs with bridge transforms\n    templates/                  # PC-specific Docker compose, env files\n    src/\n      main/                     # Electron main process (10 manager modules)\n      renderer/                 # Wizard + Dashboard UI\n      preload.js                # IPC bridge\n  .github/workflows/\n    docker-image.yml            # Release CI: ARM64 images + tar upload\n    docker-image-test.yml       # ARM64 test build (all services)\n    docker-image-selective-test.yml  # Selective service test build\n    docker-image-amd64-test.yml     # AMD64 test build\n    docker-image-amd64-release.yml  # AMD64 release build\n    pc-installer-release.yml    # PC installer build (Windows .exe + Linux .deb)\n  tests/                        # Shell-based test scripts\n    README.md                   # Test documentation\n    test-config-validation.sh   # Scan for stale config references\n    test-docker-setup.sh        # Validate Docker Compose syntax and env vars\n    test-container-dependencies.sh  # Analyze service dependencies and startup order\n    test-fula-sh-functions.sh   # Test fula.sh install/update/run operations\n    test-docker-services.sh     # Test container runtime behavior\n    test-uniondrive-readiness.sh    # Test uniondrive and readiness-check\n    test-go-fula-system-integration.sh  # WiFi, hotspot, storage, network tests\n    test-device-hardening.sh    # Full production update flow on real RK3588\n  install-ubuntu.sh             # Ubuntu/Debian automated installer\n```\n\n## Prerequisites\n\n### Docker Engine (Armbian/Linux)\n\n```bash\ncurl -fsSL https://get.docker.com -o get-docker.sh\nsudo sh get-docker.sh\n```\n\nOptionally manage Docker as a non-root user ([docs](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user)):\n\n```bash\nsudo groupadd docker\nsudo usermod -aG docker $USER\nnewgrp docker\n```\n\n### Docker Desktop (PC Installer)\n\nThe PC installer requires [Docker Desktop](https://www.docker.com/products/docker-desktop/) on Windows and macOS. The setup wizard checks for Docker Desktop and provides install instructions if missing.\n\n### Docker Compose\n\n```bash\nsudo curl -L \"https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-$(uname -s)-$(uname -m)\" -o /usr/local/bin/docker-compose\nsudo chmod +x /usr/local/bin/docker-compose\nsudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose\n```\n\n### NetworkManager\n\n```bash\nsudo systemctl start NetworkManager\nsudo systemctl enable NetworkManager\n```\n\n### Dependencies\n\n```bash\nsudo apt-get install gcc python3-dev python-is-python3 python3-pip\nsudo apt-get install python3-gi python3-gi-cairo gir1.2-gtk-3.0\nsudo apt install net-tools dnsmasq-base rfkill lshw\n```\n\n### Automount (Armbian only)\n\nRaspberry Pi OS handles auto-mounting natively. On Armbian, set up automount:\n\n#### 1. Install dependencies\n\n```bash\nsudo apt install net-tools dnsmasq-base rfkill git\n```\n\n#### 2. Create automount script\n\n```bash\nsudo nano /usr/local/bin/automount.sh\n```\n\n```bash\n#!/bin/bash\n\nMOUNTPOINT=\"/media/pi\"\nDEVICE=\"/dev/$1\"\nMOUNTNAME=$(echo $1 | sed 's/[^a-zA-Z0-9]//g')\nmkdir -p ${MOUNTPOINT}/${MOUNTNAME}\n\nFSTYPE=$(blkid -o value -s TYPE ${DEVICE})\n\nif [ ${FSTYPE} = \"ntfs\" ]; then\n    mount -t ntfs -o uid=pi,gid=pi,dmask=0000,fmask=0000 ${DEVICE} ${MOUNTPOINT}/${MOUNTNAME}\nelif [ ${FSTYPE} = \"vfat\" ]; then\n    mount -t vfat -o uid=pi,gid=pi,dmask=0000,fmask=0000 ${DEVICE} ${MOUNTPOINT}/${MOUNTNAME}\nelse\n    mount ${DEVICE} ${MOUNTPOINT}/${MOUNTNAME}\n    chown pi:pi ${MOUNTPOINT}/${MOUNTNAME}\nfi\n```\n\n```bash\nsudo chmod +x /usr/local/bin/automount.sh\n```\n\n#### 3. Create udev rules and service\n\n```bash\nsudo nano /etc/udev/rules.d/99-automount.rules\n```\n\n```\nACTION==\"add\", KERNEL==\"sd[a-z][0-9]\", TAG+=\"systemd\", ENV{SYSTEMD_WANTS}=\"automount@%k.service\"\nACTION==\"add\", KERNEL==\"nvme[0-9]n[0-9]p[0-9]\", TAG+=\"systemd\", ENV{SYSTEMD_WANTS}=\"automount@%k.service\"\n\nACTION==\"remove\", KERNEL==\"sd[a-z][0-9]\", RUN+=\"/bin/systemctl stop automount@%k.service\"\nACTION==\"remove\", KERNEL==\"nvme[0-9]n[0-9]p[0-9]\", RUN+=\"/bin/systemctl stop automount@%k.service\"\n```\n\n```bash\nsudo nano /etc/systemd/system/automount@.service\n```\n\n```ini\n[Unit]\nDescription=Automount disks\nBindsTo=dev-%i.device\nAfter=dev-%i.device\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=/usr/local/bin/automount.sh %I\nExecStop=/usr/bin/sh -c '/bin/umount /media/pi/$(echo %I | sed 's/[^a-zA-Z0-9]//g'); /bin/rmdir /media/pi/$(echo %I | sed 's/[^a-zA-Z0-9]//g')'\n```\n\n```bash\nsudo udevadm control --reload-rules\nsudo systemctl daemon-reload\n```\n\n## Device Installation\n\n### Armbian (ARM64)\n\n```bash\ngit clone https://github.com/functionland/fula-ota\ncd fula-ota/docker/fxsupport/linux\nsudo bash ./fula.sh rebuild\nsudo bash ./fula.sh start\n```\n\n### Ubuntu/Debian (automated)\n\n```bash\ncurl -fsSL https://raw.githubusercontent.com/functionland/fula-ota/main/install-ubuntu.sh | sudo bash\n```\n\nThe `install-ubuntu.sh` script handles: OS detection, dependency installation, Docker setup, repo cloning, automount configuration, and systemd service enablement. Supports Ubuntu 22.04+ (Jammy, Lunar, Mantic, Noble).\n\n### Windows/macOS/Linux PC\n\nDownload the installer from [GitHub Releases](https://github.com/functionland/fula-ota/releases):\n- Windows: `.exe` (Squirrel installer)\n- Linux: `.deb` package\n\n**GUI mode** (default): The setup wizard guides through Docker Desktop verification, storage selection, port checks, image pulling, and service launch.\n\n**CLI mode** (headless): Pass `--wallet` to skip the GUI entirely. Useful for Linux servers, scripted deployments, and headless machines.\n\n```bash\n# Basic setup (identity + services, no pool):\nfula-node --wallet 0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 \\\n          --password \"mypassword\" \\\n          --authorizer 12D3KooWAazUofWiZPiTovPDTqk8pucGw5k9ruSfbwncaktWi7DN\n\n# Full setup with pool joining:\nfula-node --wallet 0xac09...f80 --password \"mypass\" --authorizer 12D3KooWAaz... \\\n          --chain skale --poolId 1\n\n# Custom data directory:\nfula-node --wallet 0xac09...f80 --password \"mypass\" --authorizer 12D3KooWAaz... \\\n          --data-dir /opt/fula-data\n```\n\n| Parameter | Required | Description |\n|-----------|----------|-------------|\n| `--wallet \u003chex\u003e` | Yes | Ethereum private key (64 hex chars, with or without `0x` prefix) |\n| `--password \u003cstring\u003e` | Yes | Password for identity derivation (same as in mobile app) |\n| `--authorizer \u003cpeerID\u003e` | Yes | Peer ID of the authorizing mobile device (starts with `12D3KooW`) |\n| `--chain \u003cskale\\|base\u003e` | No | Blockchain for pool joining (default: `skale`) |\n| `--poolId \u003cnumber\u003e` | No | Pool ID to join (requires `--chain`) |\n| `--data-dir \u003cpath\u003e` | No | Data directory (default: `%LOCALAPPDATA%\\FulaData` on Windows, `~/.fula` on Linux) |\n| `--help`, `-h` | No | Show help message and exit |\n\nThe CLI mode performs the full setup sequence: input validation, identity derivation from wallet+password, config.yaml and box_props.json generation, Docker image pull (with per-image progress output), service startup, and optional pool joining. On success it prints a summary and exits with code 0.\n\n## Building and Pushing Docker Images\n\n### Production Release (pushes to Docker Hub)\n\n```bash\ncd docker\nsource env_release.sh        # ARM64\n# or: source env_release_amd64.sh   # AMD64\nbash ./build_and_push_images.sh\n```\n\nThis builds and pushes all images:\n- `functionland/fxsupport:release`\n- `functionland/go-fula:release`\n- `functionland/ipfs-cluster:release`\n- `functionland/fula-pinning:release`\n- `functionland/fula-gateway:release`\n\n### Test Build and Deploy Workflow\n\nUse test-tagged images (e.g. `test147`) to validate changes on a device before promoting to `release`.\n\n#### Step 1: Set the test tag\n\n`env_test.sh` defaults to `test147`. Override at runtime without editing the file:\n\n```bash\ncd docker\n\n# Use default tag (test147):\nsource env_test.sh\n\n# Or override:\nTEST_TAG=test148 source env_test.sh\n```\n\nThis exports all image tags with your tag and writes a matching `.env` file.\n\n#### Step 2: Build test images\n\n**Option A: GitHub Actions (recommended)**\n\nTrigger one of these workflows manually from the Actions tab:\n- `docker-image-test.yml` — builds all ARM64 images with the test tag\n- `docker-image-amd64-test.yml` — builds all AMD64 images\n- `docker-image-selective-test.yml` — selectively build individual services (checkbox per service, custom tag override)\n\n**Option B: Local build + push to Docker Hub**\n\n```bash\ncd docker\nsource env_test.sh          # or: TEST_TAG=test148 source env_test.sh\nbash ./build_and_push_images.sh\n```\n\n**Option C: On-device build (no Docker Hub push)**\n\n```bash\n# Build fxsupport (seconds — just file copies)\ncd /tmp/fula-ota/docker/fxsupport\nsudo docker build --load -t functionland/fxsupport:test147 .\n\n# Build go-fula (requires Go compilation)\ncd /tmp/fula-ota/docker/go-fula\ngit clone -b main https://github.com/functionland/go-fula\nsudo docker build --load -t functionland/go-fula:test147 .\n\n# Build ipfs-cluster (requires Go compilation)\ncd /tmp/fula-ota/docker/ipfs-cluster\ngit clone -b master https://github.com/ipfs-cluster/ipfs-cluster\nsudo docker build --load -t functionland/ipfs-cluster:test147 .\n```\n\n#### Step 3: Deploy to device\n\nIf you built with Options A or B (images pushed to Docker Hub), just update `.env` and restart:\n\n```bash\n# 1. Update .env to use test tags\nsudo tee /usr/bin/fula/.env \u003c\u003c 'EOF'\nGO_FULA=functionland/go-fula:test147\nFX_SUPPROT=functionland/fxsupport:test147\nIPFS_CLUSTER=functionland/ipfs-cluster:test147\nFULA_PINNING=functionland/fula-pinning:test147\nFULA_GATEWAY=functionland/fula-gateway:test147\nWPA_SUPLICANT_PATH=/etc\nCURRENT_USER=pi\nEOF\n\n# 2. Restart fula — it will pull test147 images from Docker Hub and start them\nsudo systemctl restart fula\n```\n\nThat's it. `fula.sh` runs `docker-compose pull --env-file .env` which pulls whatever tags are in `.env`. Watchtower also respects the running container's tag — it will check for updates to `:test147`, not `:release`.\n\nThe `docker cp` step (which copies files from `fula_fxsupport` to `/usr/bin/fula/`) is also safe: the `fxsupport:test147` image was built with `env_test.sh`, so the `.env` baked inside it already has test tags.\n\n\u003e **Note**: `FULA_PINNING` and `FULA_GATEWAY` are optional in `.env`. The `docker-compose.yml` uses default fallbacks (e.g. `${FULA_GATEWAY:-functionland/fula-gateway:release}`), so older `.env` files without these variables will still work.\n\n**If you built on-device (Option C)**, you must also block Docker Hub so `fula.sh` doesn't pull release images over your local builds:\n\n```bash\n# Block Docker Hub BEFORE restarting\nsudo bash -c 'echo \"127.0.0.1 index.docker.io registry-1.docker.io\" \u003e\u003e /etc/hosts'\nsudo systemctl restart fula\n```\n\n#### Step 4: Verify\n\n```bash\n# Check running images have correct tags\nsudo docker ps --format '{{.Names}}\\t{{.Image}}'\n# Expected: fula_go -\u003e functionland/go-fula:test147, etc.\n\n# Check logs\nsudo docker logs fula_go --tail 50\nsudo docker logs ipfs_host --tail 50\nsudo docker logs ipfs_cluster --tail 50\n```\n\n#### Step 5: Revert to production\n\n```bash\n# 1. If Docker Hub was blocked, unblock it\nsudo sed -i '/index.docker.io/d' /etc/hosts\n\n# 2. Restore release .env\nsudo tee /usr/bin/fula/.env \u003c\u003c 'EOF'\nGO_FULA=functionland/go-fula:release\nFX_SUPPROT=functionland/fxsupport:release\nIPFS_CLUSTER=functionland/ipfs-cluster:release\nFULA_PINNING=functionland/fula-pinning:release\nFULA_GATEWAY=functionland/fula-gateway:release\nWPA_SUPLICANT_PATH=/etc\nCURRENT_USER=pi\nEOF\n\n# 3. Restart fula (will pull release images)\nsudo systemctl restart fula\n```\n\n#### Gotchas\n\n- **On-device builds need Docker Hub blocked**: `fula.sh` runs `docker-compose pull` on every restart if internet is available. This pulls from Docker Hub using tags from `.env`. For remote-built test images (Options A/B), this is fine — it pulls your `:test147` images. But for on-device builds (Option C), the pull would overwrite your local images with Docker Hub versions. Block Docker Hub in `/etc/hosts` for on-device builds.\n- **Watchtower tag matching**: Watchtower checks for updates to the exact tag the container is running. Containers running `:test147` won't be replaced with `:release`.\n- **`docker cp` and `.env`**: On every restart, `fula.sh` copies files from `fula_fxsupport` container to `/usr/bin/fula/`, including `.env`. This is safe when using test images built through `env_test.sh` (the `.env` inside the image matches your test tags). Use `stop_docker_copy.txt` only if you manually edited `.env` on-device without rebuilding fxsupport.\n- **`stop_docker_copy.txt` expires after 24 hours**: If needed, this file in `/home/pi/` blocks the `docker cp` step. Expires after 24h — touch it again to extend.\n- **Docker Compose bridge != docker0**: Docker Compose creates its own bridge interfaces (`br-\u003chash\u003e`), not `docker0`. Firewall rules matching `-i docker0` don't cover Compose bridges. Must also match `-i br-+` (iptables wildcard). This can cause kubo-\u003ego-fula proxy traffic to be silently dropped.\n- **`((var++))` with `set -e`**: In bash, post-increment `((0++))` evaluates to 0 (falsy), returning exit code 1, which kills the script under `set -e`. Use pre-increment `((++var))` instead.\n\n## Testing Changes on a Live Device\n\n### Method 1: Build fxsupport locally on device (recommended for script/config changes)\n\nIf you only changed files under `docker/fxsupport/linux/` (scripts, configs, env files):\n\n```bash\n# On the device:\ncd /tmp \u0026\u0026 git clone --depth 1 https://github.com/functionland/fula-ota.git\ncd /tmp/fula-ota/docker/fxsupport\nsudo docker build --load -t functionland/fxsupport:release .\nsudo docker stop fula_updater\nsudo systemctl restart fula\n```\n\n### Method 2: Build all images locally on device\n\n```bash\n# Build fxsupport (seconds)\ncd /tmp/fula-ota/docker/fxsupport\nsudo docker build --load -t functionland/fxsupport:release .\n\n# Build ipfs-cluster (requires Go compilation)\ncd /tmp/fula-ota/docker/ipfs-cluster\ngit clone -b master https://github.com/ipfs-cluster/ipfs-cluster\nsudo docker build --load -t functionland/ipfs-cluster:release .\n\n# Build go-fula (requires Go compilation)\ncd /tmp/fula-ota/docker/go-fula\ngit clone -b main https://github.com/functionland/go-fula\nsudo docker build --load -t functionland/go-fula:release .\n\n# Stop watchtower, restart\nsudo docker stop fula_updater\nsudo systemctl restart fula\n```\n\n### Method 3: Copy files directly (skip Docker build entirely)\n\nFor rapid iteration, copy files directly and prevent `fula.sh` from overwriting them via `docker cp`:\n\n```bash\n# 1. Copy changed files\nsudo cp /tmp/fula-ota/docker/fxsupport/linux/.env.cluster /usr/bin/fula/.env.cluster\nsudo cp /tmp/fula-ota/docker/fxsupport/linux/ipfs-cluster/ipfs-cluster-container-init.d.sh /usr/bin/fula/ipfs-cluster/ipfs-cluster-container-init.d.sh\nsudo cp /tmp/fula-ota/docker/fxsupport/linux/kubo/config /usr/bin/fula/kubo/config\nsudo cp /tmp/fula-ota/docker/fxsupport/linux/fula.sh /usr/bin/fula/fula.sh\nsudo cp /tmp/fula-ota/docker/fxsupport/linux/update_kubo_config.py /usr/bin/fula/update_kubo_config.py\n\n# 2. Block docker cp from overwriting your files (valid for 24 hours)\ntouch /home/pi/stop_docker_copy.txt\n\n# 3. Restart using your files\nsudo /usr/bin/fula/fula.sh restart\n\n# 4. When done testing, remove the block so normal OTA updates resume\nrm /home/pi/stop_docker_copy.txt\n```\n\n## Testing\n\nThe `tests/` directory contains shell-based test scripts organized into three categories:\n\n### Validation Tests\n- `test-config-validation.sh` — Scans scripts and configs for stale references\n- `test-docker-setup.sh` — Validates Docker Compose syntax and environment variables\n- `test-container-dependencies.sh` — Analyzes service dependencies and startup order\n\n### Core Component Tests\n- `test-fula-sh-functions.sh` — Tests fula.sh install/update/run operations\n- `test-docker-services.sh` — Tests container runtime behavior\n- `test-uniondrive-readiness.sh` — Tests uniondrive service and readiness-check.py\n\n### System Integration Tests\n- `test-go-fula-system-integration.sh` — WiFi, hotspot, storage, network tests\n- `test-device-hardening.sh` — Full production update flow on real RK3588 hardware\n\nRun tests from the project root:\n```bash\n./tests/test-config-validation.sh\n./tests/test-docker-setup.sh\n# Device tests (require hardware):\nsudo bash ./tests/test-device-hardening.sh --build --deploy --verify --finish\n```\n\nSee [tests/README.md](tests/README.md) for full documentation.\n\n## CI/CD Workflows\n\n| Workflow | Trigger | Purpose |\n|----------|---------|---------|\n| `docker-image.yml` | GitHub release | ARM64 release build, uploads watchtower tar |\n| `docker-image-test.yml` | Manual | ARM64 test build (all services) |\n| `docker-image-selective-test.yml` | Manual | Selective service build (checkbox per service) |\n| `docker-image-amd64-test.yml` | Manual | AMD64 test build |\n| `docker-image-amd64-release.yml` | GitHub release | AMD64 release build |\n| `pc-installer-release.yml` | GitHub release | Windows .exe + Linux .deb installer build |\n\n### Verification Commands\n\n```bash\n# Kubo StorageMax (should be ~80% of drive size, min 800GB)\ncat /home/pi/.internal/ipfs_data/config | jq '.Datastore.StorageMax'\n\n# Kubo AcceleratedDHTClient\ncat /home/pi/.internal/ipfs_data/config | jq '.Routing.AcceleratedDHTClient'\n# Expected: true\n\n# Cluster connection_manager\ncat /uniondrive/ipfs-cluster/service.json | jq '.cluster.connection_manager'\n# Expected: high_water: 400, low_water: 100\n\n# Cluster concurrent_pins\ncat /uniondrive/ipfs-cluster/service.json | jq '.pin_tracker.stateless.concurrent_pins'\n# Expected: 5\n\n# Cluster batching\ncat /uniondrive/ipfs-cluster/service.json | jq '.consensus.crdt.batching'\n# Expected: max_batch_size: 100, max_batch_age: \"1m\"\n\n# Fula-pinning status (requires pairing secret)\ncurl -s -H \"Authorization: Bearer YOUR_PAIRING_SECRET\" http://127.0.0.1:3501/api/v1/auto-pin/status | jq .\n# Expected: {\"paired\":true,\"total_pinned\":N,...}\n\n# Fula-pinning logs\ndocker logs fula_pinning --tail 20\n\n# Fula-gateway health (no auth)\ncurl -s http://127.0.0.1:9000/healthz\n# Expected: 200 OK\n\n# Fula-gateway bucket listing (requires pairing secret)\ncurl -s -H \"Authorization: Bearer YOUR_PAIRING_SECRET\" http://127.0.0.1:9000/ | head -20\n# Expected: XML bucket listing\n\n# Fula-gateway logs\ndocker logs fula_gateway --tail 20\n\n# Registry CID (written by fula-pinning, read by fula-gateway)\ncat /home/pi/.internal/fula-gateway/registry.cid\n\n# IPFS repo stats\ndocker exec ipfs_host ipfs repo stat\n\n# IPFS DHT status (should show full routing table with AcceleratedDHTClient)\ndocker exec ipfs_host ipfs stats dht\n\n# Cluster peers\ndocker exec ipfs_cluster ipfs-cluster-ctl peers ls | wc -l\n```\n\n## fula.sh Commands\n\nCommand | Description\n--- | ---\n`start` | Start all containers (runs config merge, docker-compose up).\n`restart` | Same as start.\n`stop` | Stop all containers (docker-compose down).\n`rebuild` | Full rebuild: install dependencies, copy files, docker-compose build.\n`update` | Pull latest docker images.\n`install` | Run the initial installer.\n`help` | List all commands.\n\n## Key Configuration Files\n\n### `.env.cluster` - Cluster Environment Overrides\n\nEnv vars injected into the ipfs-cluster container. Key settings:\n\n| Variable | Value | Purpose |\n|----------|-------|---------|\n| `CLUSTER_CONNMGR_HIGHWATER` | 400 | Max peer connections (prevents 60-node ceiling) |\n| `CLUSTER_CONNMGR_LOWWATER` | 100 | Connection pruning target |\n| `CLUSTER_MONITORPINGINTERVAL` | 60s | How often peers ping the monitor |\n| `CLUSTER_STATELESS_CONCURRENTPINS` | 5 | Parallel pin operations (lower for ARM I/O) |\n| `CLUSTER_IPFSHTTP_PINTIMEOUT` | 15m0s | Timeout for individual pin operations |\n| `CLUSTER_PINRECOVERINTERVAL` | 8m0s | How often to retry failed pins |\n\n### `kubo/config` - Kubo Config Template\n\nTemplate for IPFS node configuration. Managed fields are merged into the deployed config on every restart. Key settings:\n\n- `AcceleratedDHTClient: true` - Full Amino DHT routing table with parallel lookups\n- `StorageMax: \"800GB\"` - Static fallback; dynamically set to 80% of drive on startup\n- `ConnMgr.HighWater: 200` - IPFS swarm connection limit\n- `Libp2pStreamMounting: true` - Required for go-fula p2p protocol forwarding\n\n### `update_kubo_config.py` - Config Merger\n\nSelectively merges managed fields from template into deployed config while preserving device-specific settings (Identity, Datastore). Runs on every `fula.sh start`.\n\n## Notes\n\n- **Watchtower** polls Docker Hub every 3600 seconds (1 hour). After pushing new images, devices update within the next polling cycle.\n- **`stop_docker_copy.txt`** — when this file exists in `/home/pi/` and was modified within the last 24 hours, `fula.sh` skips the `docker cp` step. Useful for testing local file changes without them being overwritten.\n- **Kubo** uses the upstream `ipfs/kubo:release` image (not custom-built). Only the config template and init script are customized.\n- **go-fula**, **ipfs-cluster**, and **fula-pinning** are custom-built from source in their respective Dockerfiles using Go 1.25.\n- **fula-gateway** is a standalone Rust binary built from `docker/fula-gateway/fula-local-gateway/`. It uses `fula-core` and `fula-blockstore` crates from [fula-api](https://github.com/functionland/fula-api) as shared libraries but contains no cloud code.\n- **fula-pinning** and **fula-gateway** run with `no-new-privileges:true` (no elevated privileges needed).\n- All other containers except kubo run with `privileged: true` and `CAP_ADD: ALL`.\n- **Dead containers**: Docker Compose can leave Dead containers with project labels. These prevent `--no-recreate` from creating new containers. `fula.sh` and the PC installer purge ghost containers via `docker ps -a --filter \"label=com.docker.compose.project=fula\" -q | xargs -r docker rm -f` before starting.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffunctionland%2Ffula-ota","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffunctionland%2Ffula-ota","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffunctionland%2Ffula-ota/lists"}