https://github.com/subsquid/network-static-server
A cache layer for serving assignment files (NET-237)
https://github.com/subsquid/network-static-server
Last synced: 4 days ago
JSON representation
A cache layer for serving assignment files (NET-237)
- Host: GitHub
- URL: https://github.com/subsquid/network-static-server
- Owner: subsquid
- Created: 2026-03-18T15:17:34.000Z (4 months ago)
- Default Branch: master
- Last Pushed: 2026-05-22T09:36:58.000Z (about 2 months ago)
- Last Synced: 2026-05-22T16:15:59.046Z (about 2 months ago)
- Language: Go
- Homepage:
- Size: 26.4 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Static data server for SQD Network
Downloads assignment files from S3 and re-serves them on the localhost.
It allows N Workers/Portals to download it over the LAN instead of N direct
downloads from the S3 upstream.
## How it works
For each configured network the service:
1. Polls the upstream `network-state-.json` every `poll-interval`.
2. When `assignment.id` changes, downloads the file at `assignment.fb_url_v1`
into `//.fb.1.gz` and deletes the previous file.
3. Serves the metadata at `/network-state-.json` with `fb_url_v1`
rewritten to `http:///data//.fb.1.gz`.
4. Streams the cached data file at `/data//.fb.1.gz` via
`http.ServeFile`.
If a client requests metadata for a network whose data file has not finished
downloading yet, the request is transparently proxied to the upstream so the
client still gets a usable response.
`GET /ready` returns 200 only once every configured network has its initial
data file on disk; otherwise 503. Use it as the Kubernetes readiness probe.
## Building
Install Go and run
```bash
go build
```
## Running
```bash
static-server --networks mainnet,tethys --cache-dir /var/cache/static-server
```
Or with docker:
```bash
docker run -d --name static-server \
-p 8080:8080 \
-v static-server-cache:/tmp/cache \
subsquid/network-static-server:latest \
--networks mainnet,tethys
```
All flags can also be set via environment variables:
| Flag | Env | Default | Description |
| ---- | --- | ------- | ----------- |
| `--networks` | `NETWORKS` | *(required)* | Comma-separated network names. |
| `--poll-interval` | `POLL_INTERVAL` | `60s` | Interval of checking for upstream updates. |
| `--cache-dir` | `CACHE_DIR` | `/tmp/cache` | Directory for downloaded files. |
| `--listen-addr` | `LISTEN_ADDR` | `:8080` | HTTP listen address. |
A sample Docker image is built from the included `Dockerfile`. Mount a
persistent volume at `cache-dir` to avoid re-downloading on pod restart.
## Using from the Portal
In the Portal config (`portal/src/config.rs`), `assignments_url` is the **base
URL** — the portal appends `/network-state-.json` itself. Point it at
the static-server's cluster-local address:
```yaml
hostname: ...
sqd_network:
datasets: ...
assignments_url: http://static-server:8080
...
```
## Using from the Worker
In the worker, `--assignment-url` / `ASSIGNMENT_URL` is the **full URL** including
the file name. Set it per network:
```yaml
# mainnet
ASSIGNMENT_URL: http://static-server:8080/network-state-mainnet.json
# tethys
ASSIGNMENT_URL: http://static-server:8080/network-state-tethys.json
```