{"id":47948829,"url":"https://github.com/slaclab/react-squirrel-backend","last_synced_at":"2026-04-04T08:55:09.911Z","repository":{"id":343572880,"uuid":"1115324848","full_name":"slaclab/react-squirrel-backend","owner":"slaclab","description":"A FastAPI backend with Postgres database for the react version of Squirrel ","archived":false,"fork":false,"pushed_at":"2026-04-03T18:17:11.000Z","size":1302,"stargazers_count":2,"open_issues_count":19,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-04T08:55:06.100Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://slaclab.github.io/react-squirrel-backend/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/slaclab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-12T17:21:18.000Z","updated_at":"2026-04-03T18:16:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/slaclab/react-squirrel-backend","commit_stats":null,"previous_names":["slaclab/react-squirrel-backend"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/slaclab/react-squirrel-backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Freact-squirrel-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Freact-squirrel-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Freact-squirrel-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Freact-squirrel-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/slaclab","download_url":"https://codeload.github.com/slaclab/react-squirrel-backend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Freact-squirrel-backend/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31393781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T04:26:24.776Z","status":"ssl_error","status_checked_at":"2026-04-04T04:23:34.147Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-04T08:55:09.296Z","updated_at":"2026-04-04T08:55:09.894Z","avatar_url":"https://github.com/slaclab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Squirrel Backend\n\nHigh-performance Python FastAPI backend for EPICS control system snapshot/restore operations, designed to handle 40-50K PVs efficiently.\n\n## Features\n\n- **Distributed Architecture**: Separate processes for API, PV monitoring, and background tasks\n- **Fast Snapshot Creation**: Parallel EPICS reads or instant Redis cache reads (\u003c5s for 40K PVs)\n- **Efficient Restore Operations**: Parallel EPICS writes for quick machine state restoration\n- **Real-Time Updates**: WebSocket streaming with diff-based updates and multi-instance support\n- **Tag-based Organization**: Group and categorize PVs using hierarchical tags\n- **Snapshot Comparison**: Compare two snapshots with tolerance-based diff\n- **Persistent Job Queue**: Background tasks survive restarts with automatic retries\n- **Circuit Breaker**: Fail-fast protection against unresponsive IOCs\n- **PostgreSQL Storage**: Reliable relational database with async support\n\n## Technology Stack\n\n| Component | Technology |\n|-----------|------------|\n| Language | Python 3.11+ |\n| Framework | FastAPI |\n| Database | PostgreSQL 16+ |\n| ORM | SQLAlchemy 2.0 (async) |\n| Cache/Queue | Redis 7+ |\n| Task Queue | Arq |\n| EPICS | aioca (async Channel Access) |\n| Migrations | Alembic |\n| Validation | Pydantic v2 |\n\n---\n\n## Quick Start\n\n**New here?** See [QUICKSTART.md](QUICKSTART.md) for a 2-minute setup guide!\n\n### Option 1: Docker Compose (Recommended)\n\nThe easiest way to get started with the full distributed architecture:\n\n```bash\n# Clone the repository\ngit clone \u003crepository-url\u003e\ncd react-squirrel-backend\n\n# Start the full stack\ncd docker\ncp .env.example .env\n# Note: If needing to make EPICS connections outside of your machine's localhost, edit\n# the .env file to add the IP addresses or host names to EPICS_CA_ADDR_LIST/EPICS_PVA_ADDR_LIST\n# as necessary. For example:\n# EPICS_CA_ADDR_LIST=lcls-prod01:5068 lcls-prod01:5063\ndocker-compose up -d --build\n\n# Configure the database\ndocker exec squirrel-api alembic upgrade head\n```\n\nThis starts:\n- **PostgreSQL** on port `5432`\n- **Redis** on port `6379`\n- **API Server** on port `8080` (REST/WebSocket)\n- **PV Monitor** (1 replica) - EPICS monitoring process\n- **Workers** (2 replicas) - Background task processors\n\nThe Docker Compose project is named **`squirrel`**, so containers are:\n- `squirrel-api`, `squirrel-db`, `squirrel-redis`, `squirrel-monitor`, `squirrel-worker-1`, `squirrel-worker-2`\n\nThe API will be available at:\n- **API**: http://localhost:8080\n- **Swagger Docs**: http://localhost:8080/docs\n- **Health Check**: http://localhost:8080/v1/health/summary\n\nTo stop the services:\n```bash\ndocker compose down\n```\n\nTo reset the database (delete all data):\n```bash\ndocker compose down -v\n```\n\n### Option 2: Legacy Mode (Single Process)\n\nFor simpler deployments with embedded PV monitoring:\n\n```bash\ncd docker\ncp .env.example .env\ndocker compose --profile legacy up backend db redis\n```\n\nThis runs the API with embedded PV monitor on port `8001`.\n\n**Note**: Workers are still required for snapshot creation. Start them separately:\n```bash\ndocker compose up -d worker\n```\n\n### Option 3: Local Development\n\nRun infrastructure in Docker, services locally for faster development:\n\n```bash\n# 1. Start PostgreSQL and Redis\ncd docker\ndocker compose up -d db redis\n\n# 2. Set up Python environment (or run ./setup.sh)\ncd ..\npython -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\npip install -e \".[dev]\"\n\n# 3. Configure environment\ncp .env.example .env\n# Edit .env if needed (defaults work with docker compose)\n\n# 4. Run database migrations\nalembic upgrade head\n\n# 5. (Optional) Load test data\npython -m scripts.seed_pvs --count 100\n\n# 6. Start services (in separate terminals)\nuvicorn app.main:app --reload --port 8000      # API Server\npython -m app.monitor_main                      # PV Monitor\narq app.worker.WorkerSettings                   # Worker (REQUIRED for snapshots)\n```\n\n**Important**: All three services must be running for full functionality:\n- **API**: Handles HTTP/WebSocket requests\n- **Monitor**: Maintains Redis cache of live PV values\n- **Worker**: Processes background jobs (snapshot creation/restore)\n\n---\n\n## Architecture Overview\n\n```\n┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐\n│   API Server    │     │   PV Monitor    │     │   Arq Worker    │\n│  (squirrel-api) │     │(squirrel-monitor)│     │(squirrel-worker)│\n│  REST/WebSocket │     │  EPICS → Redis  │     │  Snapshot jobs  │\n└────────┬────────┘     └────────┬────────┘     └────────┬────────┘\n         │                       │                       │\n         └───────────────────────┼───────────────────────┘\n                                 │\n                    ┌────────────┴────────────┐\n                    ▼                         ▼\n            ┌─────────────┐           ┌─────────────┐\n            │    Redis    │           │  PostgreSQL │\n            │ Cache/Queue │           │   Storage   │\n            └─────────────┘           └─────────────┘\n                    │\n                    ▼\n            ┌─────────────┐\n            │  EPICS IOCs │\n            │  40-50K PVs │\n            └─────────────┘\n```\n\nFor detailed architecture documentation, see [ARCHITECTURE.md](ARCHITECTURE.md).\n\n---\n\n## Loading Data\n\n### Upload PVs from CSV\n\nThe expected format:\n```csv\nSetpoint,Readback,Region,Area,Subsystem\nFBCK:LNG6:1:BC2ELTOL,,\"Feedback-All\",\"LIMITS\",\"FBCK\"\nQUAD:LI21:201:BDES,QUAD:LI21:201:BACT,\"Cu Linac\",\"LI21\",\"Magnet\"\n...\n```\n\n#### Using the UI\n* navigate to the \"Browser PVs\" page\n* click the \"Import PVs\" button\n* select the consolidated CSV\n\n#### Using a bash script\nIn addition to importing PVs, upload_csv.py also creates tag groups for the tags found in the CSV.  However, it must be run from within the docker service.\n\n```bash\n# Copying script and data into docker service\ndocker cp /path/to/local/upload_csv.py /path/to/local/consolidated.py squirrel-api:/tmp/\n\n# Dry run (see what would be uploaded)\ndocker exec squirrel-api python /tmp/upload_csv.py /tmp/consolidated.csv --dry-run\n\n# Full upload (~36K PVs)\ndocker exec squirrel-api python /tmp/upload_csv.py /tmp/consolidated.csv\n\n# With custom batch size\ndocker exec squirrel-api python /tmp/upload_csv.py /tmp/consolidated.csv --batch-size 1000\n```\n\n### Seed Test Data\n\nFor development/testing with sample data:\n\n```bash\n# Create 1000 test PVs with tags\npython -m scripts.seed_pvs --count 1000\n\n# Create 50K PVs for performance testing\npython -m scripts.seed_pvs --count 50000 --batch-size 5000\n\n# Clear existing data first\npython -m scripts.seed_pvs --count 1000 --clear\n```\n\n---\n\n## Development\n\n### Project Structure\n\n```\nsquirrel-backend/\n├── app/\n│   ├── main.py              # API entry point\n│   ├── monitor_main.py      # PV Monitor entry point\n│   ├── worker.py            # Arq worker configuration\n│   ├── config.py            # Configuration settings\n│   ├── api/v1/              # API endpoints\n│   ├── models/              # SQLAlchemy models\n│   ├── schemas/             # Pydantic schemas (DTOs)\n│   ├── services/            # Business logic layer\n│   ├── repositories/        # Data access layer\n│   ├── tasks/               # Arq task definitions\n│   └── db/                  # Database session management\n├── alembic/                 # Database migrations\n├── tests/                   # Test suite\n├── docker/                  # Docker configuration\n└── scripts/                 # Utility scripts\n```\n\n### Running Tests\n\n```bash\n# Run all tests\npytest\n\n# Run with verbose output\npytest -v\n\n# Run specific test file\npytest tests/test_api/test_pvs.py\n\n# Run with coverage report\npytest --cov=app --cov-report=html\n```\n\n**Note**: Tests use a separate test database (`squirrel_test`). Create it first:\n```bash\ncreatedb squirrel_test\n# Or via Docker:\ndocker exec -it squirrel-db createdb -U squirrel squirrel_test\n```\n\n### Database Migrations\n\n```bash\n# Apply all migrations\nalembic upgrade head\n\n# Create new migration after model changes\nalembic revision --autogenerate -m \"description of changes\"\n\n# Rollback one migration\nalembic downgrade -1\n\n# Show current migration status\nalembic current\n```\n\n### Code Quality\n\n```bash\n# Format code\nruff format .\n\n# Lint code\nruff check .\n\n# Fix auto-fixable lint issues\nruff check . --fix\n\n# Type checking\nmypy app/\n```\n\n---\n\n## API Endpoints\n\n### PV Endpoints (`/v1/pvs`)\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/v1/pvs` | Search PVs (simple) |\n| `GET` | `/v1/pvs/paged` | Search PVs with pagination |\n| `POST` | `/v1/pvs` | Create single PV |\n| `POST` | `/v1/pvs/multi` | Bulk create PVs |\n| `PUT` | `/v1/pvs/{id}` | Update PV |\n| `DELETE` | `/v1/pvs/{id}` | Delete PV |\n\n### Snapshot Endpoints (`/v1/snapshots`)\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/v1/snapshots` | List snapshots |\n| `POST` | `/v1/snapshots` | Create snapshot (async, returns job ID) |\n| `GET` | `/v1/snapshots/{id}` | Get snapshot with all values |\n| `DELETE` | `/v1/snapshots/{id}` | Delete snapshot |\n| `POST` | `/v1/snapshots/{id}/restore` | Restore values to EPICS |\n| `GET` | `/v1/snapshots/{id}/compare/{id2}` | Compare two snapshots |\n\n### Tag Endpoints (`/v1/tags`)\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/v1/tags` | List tag groups |\n| `POST` | `/v1/tags` | Create tag group |\n| `GET` | `/v1/tags/{id}` | Get tag group with tags |\n| `PUT` | `/v1/tags/{id}` | Update tag group |\n| `DELETE` | `/v1/tags/{id}` | Delete tag group |\n\n### Job Endpoints (`/v1/jobs`)\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/v1/jobs/{id}` | Get job status and progress |\n\n### Health Endpoints (`/v1/health`)\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/v1/health` | Overall health |\n| `GET` | `/v1/health/db` | Database connectivity |\n| `GET` | `/v1/health/redis` | Redis connectivity |\n| `GET` | `/v1/health/monitor/status` | PV monitor process health |\n| `GET` | `/v1/health/circuits` | Circuit breaker status |\n\n### WebSocket (`/ws`)\n\nReal-time PV value streaming with diff-based updates:\n\n```javascript\nconst ws = new WebSocket('ws://localhost:8000/ws');\n\n// Subscribe to PVs\nws.send(JSON.stringify({\n  action: 'subscribe',\n  pv_names: ['PV:NAME:1', 'PV:NAME:2']\n}));\n\n// Receive updates\nws.onmessage = (event) =\u003e {\n  const data = JSON.parse(event.data);\n  // { pv_name: 'PV:NAME:1', value: 42.0, timestamp: '...' }\n};\n```\n\n---\n\n## Configuration\n\nAll configuration is via environment variables (with `SQUIRREL_` prefix):\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `SQUIRREL_DATABASE_URL` | `postgresql+asyncpg://...` | Database connection |\n| `SQUIRREL_DATABASE_POOL_SIZE` | `30` | Connection pool size |\n| `SQUIRREL_REDIS_URL` | `redis://localhost:6379/0` | Redis connection |\n| `SQUIRREL_EPICS_CA_TIMEOUT` | `10.0` | Operation timeout (seconds) |\n| `SQUIRREL_EPICS_CHUNK_SIZE` | `1000` | PVs per batch in parallel ops |\n| `SQUIRREL_PV_MONITOR_BATCH_SIZE` | `500` | PVs per subscription batch |\n| `SQUIRREL_WATCHDOG_ENABLED` | `true` | Enable health monitoring |\n| `SQUIRREL_EMBEDDED_MONITOR` | `false` | Run monitor in API process |\n| `SQUIRREL_DEBUG` | `false` | Enable debug logging |\n\nSee `.env.example` for a complete template.\n\n### Docker-Specific Configuration\n\nIf needing to create EPICS connections to specific host names, configure EPICS server DNS mappings:\n\n```bash\n# Copy the example file\ncp docker/.env.example docker/.env\n\n# Edit with your EPICS server hostnames and IPs\n# Get IPs with: host \u003chostname\u003e\n```\n\nExample `docker/.env`:\n```bash\nCOMPOSE_PROJECT_NAME=squirrel\n\nEPICS_HOST_PROD=your-epics-server:xxx.xxx.xxx.xxx\nEPICS_HOST_DMZ=your-dmz-server:xxx.xxx.xxx.xxx\n```\n\n**Note**: `docker/.env` is gitignored and should contain your site-specific configuration.\n\n---\n\n## Docker Commands Reference\n\n```bash\n# Start all services\ncd docker\ndocker compose up\n\n# Start in background (detached)\ndocker compose up -d\n\n# Rebuild images after code changes\ndocker compose up --build\n\n# View logs\ndocker compose logs -f api\ndocker compose logs -f monitor\ndocker compose logs -f worker\n\n# Stop services\ndocker compose down\n\n# Stop and remove volumes (reset database)\ndocker compose down -v\n\n# Scale workers (for high load)\ndocker compose up -d --scale worker=4\n\n# Execute command in running container\ndocker exec -it squirrel-api bash\ndocker exec -it squirrel-db psql -U squirrel\n\n# Run migrations in Docker\ndocker exec -it squirrel-api alembic upgrade head\n\n# Load test data in Docker\ndocker compose exec api python -m scripts.seed_pvs --count 100\n```\n\n---\n\n## Troubleshooting\n\n### Database connection refused\n```bash\n# Check if PostgreSQL is running\ndocker compose ps db\n\n# Check database health\ndocker compose logs db\n\n# Test connection\ndocker exec -it squirrel-db pg_isready -U squirrel\n# Or for local: pg_isready -h localhost -p 5432\n```\n\n### Migrations fail\n```bash\n# Ensure database exists\ndocker exec -it squirrel-db createdb -U squirrel squirrel\n\n# Check migration status\nalembic current\n```\n\n### EPICS connection issues\n```bash\n# Verify EPICS environment\necho $EPICS_CA_ADDR_LIST\n\n# Test PV connectivity\ncaget \u003cpv_name\u003e\n```\n\n### PV Monitor not updating\n```bash\n# Check monitor health via API\ncurl http://localhost:8000/v1/health/monitor/status\n\n# Check Redis for heartbeat\ndocker exec -it squirrel-redis redis-cli GET squirrel:monitor:heartbeat\n```\n\n### Snapshots hanging or have no data\n```bash\n# Check if worker is running\ndocker compose ps worker\n\n# If not running, start it\ndocker compose up -d worker\n\n# Check worker logs\ndocker compose logs -f worker\n\n# Verify worker is processing jobs\ndocker exec -it squirrel-redis redis-cli LLEN arq:queue\n```\n\n**Note**: Snapshots will be empty if:\n- Test PVs don't exist on EPICS network (expected for development)\n- Monitor can't connect to PVs (check EPICS_CA_ADDR_LIST)\n- Redis cache is empty and direct EPICS reads fail\n\n### Port already in use\n```bash\n# Find process using port 8080 (Docker) or 8000 (local)\nlsof -i :8080\n\n# Change Docker port in docker-compose.yml:\n# ports:\n#   - \"8081:8000\"  # Change 8080 to 8081\n\n# Or use different port locally\nuvicorn app.main:app --reload --port 8001\n```\n\n---\n\n## API Key Management\n\nThese scripts manage API keys for authenticating requests to the backend. Run them from the project root with `python -m scripts.\u003cscript_name\u003e`.\n\n### Create a key\n\n```bash\npython -m scripts.create_key \u003capp_name\u003e [--read] [--write]\n```\n\nAt least one of `--read` / `--write` is required.\n\n```bash\n# Read-only key for the frontend app\npython -m scripts.create_key my-app --read\n\n# Read/write key\npython -m scripts.create_key my-app --read --write\n```\n\nOutput includes the app name, key ID, access level, creation timestamp, and the token (only shown at creation time).\n\n### List keys\n\n```bash\npython -m scripts.list_keys [--active-only]\n```\n\nPrints a table of all stored API keys. Use `--active-only` (`-a`) to filter out deactivated keys.\n\n### Deactivate a key\n\n```bash\npython -m scripts.deactivate_key \u003cid\u003e\n```\n\nDeactivates the key with the given ID. The key is retained in the database but can no longer be used for authentication.\n\n---\n\n## Performance Benchmarking\n\n```bash\n# Start the backend first, then run:\npython -m scripts.benchmark\n\n# With more iterations\npython -m scripts.benchmark --iterations 10\n\n# Skip restore benchmark (no EPICS writes)\npython -m scripts.benchmark --skip-restore\n```\n\n---\n\n## Frontend\n\nThe Squirrel React frontend is available at:\n- Repository: `squirrel` (separate repo)\n- Default API URL: `http://localhost:8000`\n\nConfigure the frontend to point to this backend by setting the API base URL.\n\n---\n\n## License\n\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslaclab%2Freact-squirrel-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fslaclab%2Freact-squirrel-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslaclab%2Freact-squirrel-backend/lists"}