{"id":31243505,"url":"https://github.com/system76/cosmic-sync-server","last_synced_at":"2025-10-08T19:33:46.147Z","repository":{"id":313291130,"uuid":"1044507302","full_name":"system76/cosmic-sync-server","owner":"system76","description":"Cosmic file synchronization service with server and client components","archived":false,"fork":false,"pushed_at":"2025-09-20T00:34:19.000Z","size":1514,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"staging","last_synced_at":"2025-09-20T01:31:59.355Z","etag":null,"topics":["cosmic-desktop","file-sync","grpc","rust","synchronization"],"latest_commit_sha":null,"homepage":"https://sync.pop-os.org","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/system76.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-25T19:40:15.000Z","updated_at":"2025-09-19T23:20:09.000Z","dependencies_parsed_at":"2025-09-05T04:26:25.384Z","dependency_job_id":"aac7f5df-72b5-4ac5-8b4a-8e54221dc83b","html_url":"https://github.com/system76/cosmic-sync-server","commit_stats":null,"previous_names":["system76/cosmic-sync-server"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/system76/cosmic-sync-server","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/system76%2Fcosmic-sync-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/system76%2Fcosmic-sync-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/system76%2Fcosmic-sync-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/system76%2Fcosmic-sync-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/system76","download_url":"https://codeload.github.com/system76/cosmic-sync-server/tar.gz/refs/heads/staging","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/system76%2Fcosmic-sync-server/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276502210,"owners_count":25653728,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-22T02:00:08.972Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cosmic-desktop","file-sync","grpc","rust","synchronization"],"created_at":"2025-09-23T01:21:31.705Z","updated_at":"2025-09-23T01:21:36.547Z","avatar_url":"https://github.com/system76.png","language":"Rust","readme":"# COSMIC Sync Server\n\nCOSMIC Sync Server is a server for synchronizing user settings and files in System76's COSMIC desktop environment.\n\n## Features\n\n- OAuth-based authentication\n- Device registration and management\n- File synchronization\n- Encryption key management\n- Watcher group management\n\n## Installation and Running\n\n### Requirements\n\n- Rust 1.70.0 or higher\n- MySQL or MariaDB\n- Protobuf compiler (protoc)\n- (Optional) RabbitMQ broker if enabling message bus\n\n### Environment Configuration\n\nCreate a `.env` file in the project root or copy the provided `.env.sample`:\n\n```bash\ncp .env.sample .env\n```\n\nThen edit the `.env` file to configure the following settings (keys unified):\n\n```\n# Environment\nENVIRONMENT=development\n\n# Server\nSERVER_HOST=0.0.0.0\nGRPC_PORT=50051\nWORKER_THREADS=4\nAUTH_TOKEN_EXPIRY_HOURS=24\nMAX_CONCURRENT_REQUESTS=100\nMAX_FILE_SIZE=52428800\nHEARTBEAT_INTERVAL_SECS=10\n\n# Database (MySQL)\nDB_HOST=localhost\nDB_PORT=3306\nDB_NAME=cosmic_sync\nDB_USER=username\nDB_PASS=password\nDB_POOL=5\nDATABASE_CONNECTION_TIMEOUT=30\nDATABASE_LOG_QUERIES=false\n\n# Storage\nSTORAGE_TYPE=database  # or s3\n# STORAGE_PATH=/tmp/cosmic-sync\n\n# S3 (if STORAGE_TYPE=s3)\nAWS_REGION=us-east-2\nAWS_S3_BUCKET=cosmic-sync-files\nS3_KEY_PREFIX=files/\n# S3_ENDPOINT_URL=http://localhost:9000\nS3_FORCE_PATH_STYLE=true\nS3_TIMEOUT_SECONDS=30\nS3_MAX_RETRIES=3\n# AWS_ACCESS_KEY_ID=...\n# AWS_SECRET_ACCESS_KEY=...\n# AWS_SESSION_TOKEN=...\n\n# Logging\nLOG_LEVEL=info\nLOG_TO_FILE=true\nLOG_FILE=logs/cosmic-sync-server.log\nLOG_MAX_FILE_SIZE=10485760\nLOG_MAX_BACKUPS=5\nLOG_FORMAT=text  # json for production\n\n# OAuth\nOAUTH_CLIENT_ID=your_client_id\nOAUTH_CLIENT_SECRET=your_client_secret\nOAUTH_REDIRECT_URI=http://localhost:50051/oauth/callback\nOAUTH_AUTH_URL=https://oauth-provider.com/auth\nOAUTH_TOKEN_URL=https://oauth-provider.com/token\nOAUTH_USER_INFO_URL=https://oauth-provider.com/userinfo\n\n# Feature flags\nCOSMIC_SYNC_DEV_MODE=false\nCOSMIC_SYNC_TEST_MODE=false\nCOSMIC_SYNC_DEBUG_MODE=false\nENABLE_METRICS=false\nSTORAGE_ENCRYPTION=true\nREQUEST_VALIDATION=true\n\n# RabbitMQ\nRABBITMQ_ENABLED=false\nRABBITMQ_URL=amqp://guest:guest@127.0.0.1:5672/%2f\nRABBITMQ_EXCHANGE=cosmic.sync\nRABBITMQ_QUEUE_PREFIX=cosmic.sync\nRABBITMQ_PREFETCH=200\nRABBITMQ_DURABLE=true\n# RETRY_TTL_MS=5000\n# MAX_RETRIES=3\n```\n\n### Database Preparation\n\nCreate the MySQL database:\n\n```bash\n# Create the database\nmysql -u root -p -e \"CREATE DATABASE cosmic_sync;\"\n```\n\nThe server will initialize the necessary tables when first started.\n\n### Running the Server\n\n```bash\n# Build (prefer filtered lbuild)\ncargo lbuild\n\n# Run in development mode\n./target/debug/cosmic-sync-server\n\n# Build and run in release mode\ncargo build --release\n./target/release/cosmic-sync-server\n```\n\n### Advanced Execution Options\n\nTo run the server with specific environment variables and debug options:\n\n```bash\n# Run with development mode, debug mode, and debug logging\nLOG_LEVEL=debug LOG_FORMAT=text cargo run\n\n# Run the compiled binary directly with root privileges\nLOG_LEVEL=debug LOG_FORMAT=text sudo -E ./target/debug/cosmic-sync-server\n```\n\n## Project Structure\n\nThe COSMIC Sync Server is organized into several modules:\n\n- **Server**: Contains HTTP and gRPC server implementations \n- **Handlers**: Request handlers for different API endpoints\n- **Services**: Business logic for handling authentication, devices, files, etc.\n- **Storage**: Data storage implementations (MySQL, in-memory)\n- **Models**: Data models for accounts, devices, files, etc.\n- **Auth**: Authentication and OAuth implementations\n- **Utils**: Utility functions for crypto, time, etc.\n- **Config**: Centralized constants and settings (including message broker)\n- **Event Bus**: RabbitMQ integration behind trait abstraction\n- **Tests**: Integration tests under top-level `tests/`\n\n### Storage Module Architecture\n\nThe storage system is designed with a modular approach using traits:\n\n- `Storage`: Main trait defining all storage operations\n- Implementation options:\n  - `MySqlStorage`: Production storage using MySQL\n  - `MemoryStorage`: In-memory storage for testing\n\nThe MySQL implementation is split into multiple files for better maintainability:\n- `mysql.rs`: Core MySQL implementation and connection management\n- `mysql_account.rs`: Account-related operations\n- `mysql_auth.rs`: Authentication token operations\n- `mysql_device.rs`: Device management operations\n- `mysql_file.rs`: File storage and retrieval operations\n- `mysql_watcher.rs`: Watcher and synchronization operations\n\n## Event Bus (RabbitMQ)\n\n- Abstraction: `src/server/event_bus.rs`\n  - `EventBus` trait with `publish`/`subscribe`\n  - `RabbitMqEventBus` (lapin) and `NoopEventBus` (disabled)\n- Routing keys (topic):\n  - Files: `file.uploaded.{account_hash}.{group_id}.{watcher_id}`, `file.deleted.{account_hash}`\n  - Versions: `version.created.{account_hash}.{file_id}`, `version.deleted.{account_hash}.{file_id}`, `version.restored.{account_hash}.{file_id}`\n  - Devices/Watchers as applicable\n- Emission sites: handlers layer (`src/handlers/...`), not services\n- Consumer example: `src/bin/rabbit_consumer.rs`\n  - Declares DLX (`\u003cexchange\u003e.dlx`), main/retry/DLQ queues, basic idempotency\n  - Run: `sudo -E /home/yongjinchong/.cargo/bin/cargo run --bin rabbit_consumer`\n\n## Tests\n\n- Integration tests live in `tests/` with shared utilities in `tests/common/mod.rs`\n- Quick compile: `cargo test --no-run`\n- Run: `cargo test`\n\n## Development\n\n### Protocol Buffer Generation\n\nAfter modifying the Protocol Buffer definitions, run the following command to generate Rust code:\n\n```bash\ncargo build\n```\n\nThe `build.rs` script will automatically compile the protobuf files.\n\n### Authentication Flow\n\nThe server uses OAuth for authentication:\n\n1. Client requests auth URL from server\n2. User completes OAuth flow in browser\n3. OAuth provider redirects to callback URL with auth code\n4. Server exchanges code for tokens\n5. Server creates an account for the user if needed\n6. Client registers the device\n7. Authentication complete\n\n### Device Registration\n\nDevice registration is a separate process from authentication:\n\n1. Client completes authentication flow and receives `auth.json`\n2. Client sends device information to register\n3. Server creates or updates device record\n4. Server returns success/failure status\n\n### File Synchronization \n\nFiles are synchronized based on watcher groups:\n\n1. Client registers watcher groups for specific directories\n2. When files change, client uploads to server\n3. Server stores files in database with metadata (or S3, depending on config)\n4. Other devices can download changed files\n5. Encryption is supported for secure file storage\n\n## Troubleshooting\n\n### Common Issues\n\n#### Database Connection\n\nIf you encounter database connection issues:\n- Verify MySQL/MariaDB is running\n- Check credentials in .env file\n- Ensure database exists and is accessible\n\n#### Authentication Failures\n\nOAuth authentication issues:\n- Verify OAuth provider settings\n- Check client ID and secret\n- Ensure redirect URI is properly configured\n- Check logs for detailed error messages\n\n#### File Synchronization Problems\n\nIf file synchronization isn't working:\n- Check device registration status\n- Verify watcher groups are properly configured\n- Check file permissions on client\n- Increase log level for detailed diagnostics\n\n## License\n\nThis project is licensed under the terms of the GNU General Public License v3.0.\n\n## Cloud DB/Cache configuration\n\n- MySQL (required)\n  - `DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASS`, `DB_NAME`\n  - Optional TLS: `DB_SSL_MODE` (DISABLED|PREFERRED|REQUIRED|VERIFY_CA|VERIFY_IDENTITY), `DB_SSL_CA` (path)\n- Redis (optional; enable with build feature `redis-cache` and/or env)\n  - `REDIS_ENABLED=true`, `REDIS_HOST=\u003chost\u003e`, `REDIS_PORT=6379`\n  - Optional: `REDIS_KEY_PREFIX` (default: `cosmic.sync`)","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsystem76%2Fcosmic-sync-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsystem76%2Fcosmic-sync-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsystem76%2Fcosmic-sync-server/lists"}