{"id":32662615,"url":"https://github.com/vitorpy/noreposts-atproto-feed","last_synced_at":"2026-04-15T20:03:49.882Z","repository":{"id":317419958,"uuid":"1067296091","full_name":"vitorpy/noreposts-atproto-feed","owner":"vitorpy","description":null,"archived":false,"fork":false,"pushed_at":"2025-12-19T20:28:28.000Z","size":272,"stargazers_count":0,"open_issues_count":8,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-22T08:45:04.178Z","etag":null,"topics":["atproto","bluesky","feed-generator","jetstream","rust"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vitorpy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-30T16:48:33.000Z","updated_at":"2025-12-19T20:28:32.000Z","dependencies_parsed_at":"2025-12-20T02:05:01.787Z","dependency_job_id":null,"html_url":"https://github.com/vitorpy/noreposts-atproto-feed","commit_stats":null,"previous_names":["vitorpy/noreposts-atproto-feed"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vitorpy/noreposts-atproto-feed","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitorpy%2Fnoreposts-atproto-feed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitorpy%2Fnoreposts-atproto-feed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitorpy%2Fnoreposts-atproto-feed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitorpy%2Fnoreposts-atproto-feed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vitorpy","download_url":"https://codeload.github.com/vitorpy/noreposts-atproto-feed/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitorpy%2Fnoreposts-atproto-feed/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31857625,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"ssl_error","status_checked_at":"2026-04-15T15:24:39.138Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atproto","bluesky","feed-generator","jetstream","rust"],"created_at":"2025-10-31T20:00:57.626Z","updated_at":"2026-04-15T20:03:49.877Z","avatar_url":"https://github.com/vitorpy.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Following No Reposts Feed Generator\n\nA production-ready Bluesky feed generator written in Rust that shows posts from people you follow, excluding all reposts. Built using Jetstream for efficient real-time data consumption with full JWT signature verification.\n\n## Table of Contents\n\n- [Features](#features)\n- [How It Works](#how-it-works)\n- [Prerequisites](#prerequisites)\n- [Installation](#installation)\n- [Configuration](#configuration)\n- [Running Locally](#running-locally)\n- [Deployment](#deployment)\n- [Publishing Your Feed](#publishing-your-feed)\n- [Architecture](#architecture)\n- [API Endpoints](#api-endpoints)\n- [Performance](#performance)\n- [Development](#development)\n- [Troubleshooting](#troubleshooting)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Features\n\n- **🚫 No Reposts**: Automatically filters out all reposts, showing only original content\n- **👥 Personalized**: Shows only posts from accounts you follow\n- **⚡ Real-time**: Updates in real-time as new posts are created\n- **🔒 Secure**: Full ES256K JWT signature verification with DID resolution\n- **📡 Efficient**: Uses Jetstream for lightweight event consumption (~850 MB/day vs 200+ GB/day)\n- **🗄️ Smart Caching**: Automatic cleanup of posts older than 48 hours\n- **🔄 Auto-recovery**: Automatic reconnection on Jetstream disconnects\n- **📊 Observable**: Structured logging with configurable verbosity\n- **🏗️ Production Ready**: Battle-tested error handling and recovery mechanisms\n\n## How It Works\n\nThis feed generator:\n\n1. **Consumes Events**: Connects to Bluesky's Jetstream to receive real-time events for posts and follows\n2. **Filters Content**: Only subscribes to `app.bsky.feed.post` and `app.bsky.graph.follow` collections\n3. **Stores Data**: Maintains a local SQLite database of recent posts and follow relationships\n4. **Serves Feeds**: Provides personalized feeds via AT Protocol's `app.bsky.feed.getFeedSkeleton` endpoint\n5. **Authenticates Users**: Validates JWT tokens by resolving user DIDs and verifying signatures\n\n## Prerequisites\n\n- **Rust** 1.70+ (install via [rustup](https://rustup.rs))\n- **SQLite** 3.35+ (usually pre-installed on modern systems)\n- **Domain** with HTTPS (required for production deployment)\n- **Bluesky Account** (for publishing the feed)\n\n## Installation\n\n### 1. Clone the Repository\n\n```bash\ngit clone https://github.com/vitorpy/noreposts-atproto-feed.git\ncd noreposts-atproto-feed\n```\n\n### 2. Build the Project\n\n```bash\ncargo build --release\n```\n\nThe compiled binary will be at `target/release/following-no-reposts-feed`.\n\n## Configuration\n\n### Environment Variables\n\nCreate a `.env` file in the project root (see `.env.example` for reference):\n\n```bash\n# Required: Database location\nDATABASE_URL=sqlite:./feed.db\n\n# Required: Server port\nPORT=3000\n\n# Required: Your domain name\nFEEDGEN_HOSTNAME=your-domain.com\n\n# Required: Your service DID\nFEEDGEN_SERVICE_DID=did:web:your-domain.com\n\n# Optional: Jetstream server (defaults to jetstream1.us-east.bsky.network)\nJETSTREAM_HOSTNAME=jetstream1.us-east.bsky.network\n```\n\n### Service DID Setup\n\nYour `FEEDGEN_SERVICE_DID` should match your domain. For `did:web`, it's typically:\n- Domain: `feed.example.com` → DID: `did:web:feed.example.com`\n- Domain: `example.com` → DID: `did:web:example.com`\n\n## Running Locally\n\n### Quick Start\n\n```bash\n# Set up environment\ncp .env.example .env\n# Edit .env with your configuration\nnano .env\n\n# Run the server\ncargo run --release\n```\n\nThe server will:\n1. Automatically run database migrations\n2. Connect to Jetstream and start consuming events\n3. Start the HTTP server on the configured port\n4. Serve the DID document at `/.well-known/did.json`\n\n### Testing Locally\n\n```bash\n# Test DID document endpoint\ncurl http://localhost:3000/.well-known/did.json\n\n# Test feed endpoint (requires authentication in production)\ncurl \"http://localhost:3000/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:web:your-domain.com/app.bsky.feed.generator/following-no-reposts\u0026limit=10\"\n```\n\n### Command-Line Options\n\n```bash\n# Override environment variables\n./following-no-reposts-feed --port 8080 --hostname feed.example.com\n\n# Run database migrations only\n./following-no-reposts-feed migrate\n\n# Publish feed to your Bluesky account\n./following-no-reposts-feed publish \\\n  --handle your-handle.bsky.social \\\n  --password your-app-password \\\n  --record-name following-no-reposts \\\n  --display-name \"Following (No Reposts)\" \\\n  --description \"See posts from people you follow, without any reposts\"\n\n# Backfill posts from firehose (optional)\n./following-no-reposts-feed backfill --cursor \u003ccursor-value\u003e\n```\n\n## Deployment\n\n### 1. Build for Production\n\n```bash\ncargo build --release --locked\n```\n\n### 2. Set Up Your Server\n\nTransfer the binary to your server:\n\n```bash\nscp target/release/following-no-reposts-feed user@your-server:/opt/feed-generator/\n```\n\n### 3. Create a Systemd Service\n\nCreate `/etc/systemd/system/feed-generator.service`:\n\n```ini\n[Unit]\nDescription=Bluesky Feed Generator - Following No Reposts\nAfter=network.target\n\n[Service]\nType=simple\nUser=feedgen\nWorkingDirectory=/opt/feed-generator\nEnvironment=\"DATABASE_URL=sqlite:/opt/feed-generator/feed.db\"\nEnvironment=\"PORT=3000\"\nEnvironment=\"FEEDGEN_HOSTNAME=your-domain.com\"\nEnvironment=\"FEEDGEN_SERVICE_DID=did:web:your-domain.com\"\nExecStart=/opt/feed-generator/following-no-reposts-feed\nRestart=always\nRestartSec=10\n\n[Install]\nWantedBy=multi-user.target\n```\n\nEnable and start the service:\n\n```bash\nsudo systemctl daemon-reload\nsudo systemctl enable feed-generator\nsudo systemctl start feed-generator\nsudo systemctl status feed-generator\n```\n\n### 4. Configure Reverse Proxy\n\nThe feed generator must be accessible via HTTPS. Configure your reverse proxy:\n\n#### Nginx\n\n```nginx\nserver {\n    listen 443 ssl http2;\n    server_name your-domain.com;\n\n    ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;\n    ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;\n\n    location / {\n        proxy_pass http://127.0.0.1:3000;\n        proxy_set_header Host $host;\n        proxy_set_header X-Real-IP $remote_addr;\n        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n        proxy_set_header X-Forwarded-Proto $scheme;\n\n        # WebSocket support for Jetstream\n        proxy_http_version 1.1;\n        proxy_set_header Upgrade $http_upgrade;\n        proxy_set_header Connection \"upgrade\";\n    }\n}\n```\n\n#### Caddy\n\n```caddyfile\nyour-domain.com {\n    reverse_proxy localhost:3000\n}\n```\n\n### 5. Verify Deployment\n\n```bash\n# Test DID document\ncurl https://your-domain.com/.well-known/did.json\n\n# Check if feed endpoint is accessible\ncurl https://your-domain.com/xrpc/app.bsky.feed.getFeedSkeleton\n```\n\n## Publishing Your Feed\n\nOnce your feed generator is deployed and accessible via HTTPS:\n\n### Method 1: Using the Built-in Publish Command\n\n```bash\n./following-no-reposts-feed publish \\\n  --handle your-handle.bsky.social \\\n  --password your-app-password \\\n  --record-name following-no-reposts \\\n  --display-name \"Following (No Reposts)\" \\\n  --description \"See posts from people you follow, without any reposts\" \\\n  --avatar ./avatar.png\n```\n\n**Note**: Use an [App Password](https://bsky.app/settings/app-passwords), not your main account password!\n\n### Method 2: Manual Publishing\n\n1. **Get your DID**:\n   ```bash\n   curl \"https://bsky.social/xrpc/com.atproto.identity.resolveHandle?handle=yourhandle.bsky.social\"\n   ```\n\n2. **Create a session**:\n   ```bash\n   curl -X POST https://bsky.social/xrpc/com.atproto.server.createSession \\\n     -H \"Content-Type: application/json\" \\\n     -d '{\"identifier\": \"yourhandle.bsky.social\", \"password\": \"your-app-password\"}'\n   ```\n\n3. **Publish the feed generator record**:\n   ```bash\n   curl -X POST https://bsky.social/xrpc/com.atproto.repo.putRecord \\\n     -H \"Authorization: Bearer YOUR_ACCESS_JWT\" \\\n     -H \"Content-Type: application/json\" \\\n     -d '{\n       \"repo\": \"your.did\",\n       \"collection\": \"app.bsky.feed.generator\",\n       \"rkey\": \"following-no-reposts\",\n       \"record\": {\n         \"$type\": \"app.bsky.feed.generator\",\n         \"did\": \"did:web:your-domain.com\",\n         \"displayName\": \"Following (No Reposts)\",\n         \"description\": \"See posts from people you follow, without any reposts\",\n         \"createdAt\": \"2025-01-01T00:00:00.000Z\"\n       }\n     }'\n   ```\n\n### Finding Your Feed\n\nAfter publishing, your feed will be available at:\n\n```\nhttps://bsky.app/profile/yourhandle.bsky.social/feed/following-no-reposts\n```\n\n## Architecture\n\n### System Components\n\n```\n┌──────────────┐         ┌──────────────┐         ┌──────────────┐\n│   Jetstream  │────────▶│   Consumer   │────────▶│   Database   │\n│  (WebSocket) │  Events │   (Async)    │  Store  │   (SQLite)   │\n└──────────────┘         └──────────────┘         └──────────────┘\n                                │\n                                │ Read\n                                ▼\n┌──────────────┐         ┌──────────────┐         ┌──────────────┐\n│   Bluesky    │────────▶│  HTTP Server │────────▶│     Feed     │\n│     App      │   JWT   │    (Axum)    │  Query  │  Algorithm   │\n└──────────────┘         └──────────────┘         └──────────────┘\n```\n\n### Code Structure\n\n- **`main.rs`**: Application entry point, HTTP server setup, routing\n- **`jetstream_consumer.rs`**: WebSocket client for Jetstream events\n- **`database.rs`**: SQLite abstraction layer, queries, and migrations\n- **`feed_algorithm.rs`**: Feed generation logic (filtering by follows, excluding reposts)\n- **`auth.rs`**: JWT validation with ES256K signature verification\n- **`backfill.rs`**: Optional historical data backfilling from firehose\n- **`publish.rs`**: Feed generator publishing utilities\n- **`admin_socket.rs`**: Unix socket for admin commands\n- **`types.rs`**: Shared data structures\n\n### Data Flow\n\n1. **Event Ingestion**: Jetstream sends `commit` events when posts are created or follows happen\n2. **Event Processing**: Consumer parses events and extracts relevant data\n3. **Database Storage**: Posts and follows are stored in SQLite with TTL\n4. **Feed Requests**: Bluesky app requests feed via `getFeedSkeleton`\n5. **Authentication**: JWT is validated by resolving DID and verifying signature\n6. **Feed Generation**: Algorithm queries posts from followed users, excludes reposts\n7. **Response**: Ordered list of post URIs returned with pagination cursor\n\n## API Endpoints\n\n### `GET /.well-known/did.json`\n\nReturns the DID document for the feed generator service.\n\n**Response**:\n```json\n{\n  \"@context\": [\"https://www.w3.org/ns/did/v1\"],\n  \"id\": \"did:web:your-domain.com\",\n  \"service\": [{\n    \"id\": \"#bsky_fg\",\n    \"type\": \"BskyFeedGenerator\",\n    \"serviceEndpoint\": \"https://your-domain.com\"\n  }]\n}\n```\n\n### `GET /xrpc/app.bsky.feed.getFeedSkeleton`\n\nReturns a personalized feed skeleton for the authenticated user.\n\n**Query Parameters**:\n- `feed` (required): Feed AT-URI (e.g., `at://did:web:your-domain.com/app.bsky.feed.generator/following-no-reposts`)\n- `limit` (optional): Number of posts (1-100, default: 50)\n- `cursor` (optional): Pagination cursor\n\n**Headers**:\n- `Authorization`: Bearer JWT token from Bluesky app\n\n**Response**:\n```json\n{\n  \"feed\": [\n    {\"post\": \"at://did:plc:xxx/app.bsky.feed.post/abc123\"},\n    {\"post\": \"at://did:plc:yyy/app.bsky.feed.post/def456\"}\n  ],\n  \"cursor\": \"1234567890\"\n}\n```\n\n## Performance\n\n### Resource Usage\n\n- **Memory**: ~50-100 MB (depends on database size)\n- **CPU**: Minimal (\u003c1% on modern hardware)\n- **Bandwidth**: ~850 MB/day (Jetstream with compression)\n- **Storage**: ~100-500 MB (48-hour post retention)\n\n### Scalability\n\nThe feed generator can handle:\n- **10,000+** active users\n- **1M+** posts/day ingestion\n- **100+** requests/second\n\n### Database Optimization\n\n```sql\n-- Efficient indexes\nCREATE INDEX idx_posts_author_did ON posts(author_did);\nCREATE INDEX idx_posts_indexed_at ON posts(indexed_at);\nCREATE INDEX idx_follows_follower ON follows(follower_did);\n\n-- Automatic cleanup (posts \u003e 48 hours old)\nDELETE FROM posts WHERE indexed_at \u003c datetime('now', '-2 days');\n```\n\n## Development\n\n### Running Tests\n\n```bash\ncargo test\n```\n\n### Database Migrations\n\nCreate a new migration:\n\n```bash\nsqlx migrate add your_migration_name\n```\n\nRun migrations manually:\n\n```bash\ncargo install sqlx-cli --no-default-features --features sqlite\nsqlx migrate run\n```\n\n### Logging\n\nSet the `RUST_LOG` environment variable:\n\n```bash\n# Debug level (verbose)\nRUST_LOG=debug cargo run\n\n# Info level (default)\nRUST_LOG=info cargo run\n\n# Specific module logging\nRUST_LOG=following_no_reposts_feed::jetstream_consumer=debug cargo run\n```\n\n### Code Quality\n\n```bash\n# Format code\ncargo fmt\n\n# Run linter\ncargo clippy --all-targets\n\n# Check for security vulnerabilities\ncargo audit\n```\n\n## Troubleshooting\n\n### Jetstream Connection Issues\n\n**Problem**: Cannot connect to Jetstream\n\n**Solutions**:\n- Verify network connectivity: `ping jetstream1.us-east.bsky.network`\n- Check firewall rules (port 443 outbound)\n- Try alternative Jetstream servers\n- Monitor logs for specific error messages\n\n### Database Locked Errors\n\n**Problem**: `database is locked` errors\n\n**Solutions**:\n- Ensure only one instance is running\n- Check for long-running transactions\n- Increase `busy_timeout` in database configuration\n- Consider using WAL mode: `PRAGMA journal_mode=WAL;`\n\n### Authentication Failures\n\n**Problem**: JWT validation errors\n\n**Solutions**:\n- Verify `FEEDGEN_SERVICE_DID` matches your domain\n- Check network access to `plc.directory` for DID resolution\n- Enable debug logging: `RUST_LOG=following_no_reposts_feed::auth=debug`\n- Verify your domain's HTTPS certificate is valid\n\n### Feed Not Updating\n\n**Problem**: Feed shows stale content\n\n**Solutions**:\n- Check Jetstream connection: look for \"Connected to Jetstream\" in logs\n- Verify database is being updated: `sqlite3 feed.db \"SELECT COUNT(*) FROM posts;\"`\n- Check for errors in logs: `journalctl -u feed-generator -n 100`\n- Restart the service: `systemctl restart feed-generator`\n\n### High Memory Usage\n\n**Problem**: Memory usage growing over time\n\n**Solutions**:\n- Verify automatic cleanup is working: check `posts` table size\n- Manually trigger cleanup: `DELETE FROM posts WHERE indexed_at \u003c datetime('now', '-2 days');`\n- Reduce retention period in code if needed\n- Monitor with: `ps aux | grep following-no-reposts-feed`\n\n### Debug Mode\n\nEnable comprehensive debugging:\n\n```bash\nRUST_LOG=debug,hyper=info,tokio=info cargo run\n```\n\nTest endpoints directly:\n\n```bash\n# Test DID endpoint\ncurl -v https://your-domain.com/.well-known/did.json\n\n# Test feed endpoint with authentication\ncurl -v -H \"Authorization: Bearer YOUR_JWT\" \\\n  \"https://your-domain.com/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:web:your-domain.com/app.bsky.feed.generator/following-no-reposts\u0026limit=5\"\n```\n\n## Contributing\n\nContributions are welcome! Please:\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Make your changes\n4. Add tests if applicable\n5. Run `cargo fmt` and `cargo clippy`\n6. Commit your changes (`git commit -m 'Add amazing feature'`)\n7. Push to the branch (`git push origin feature/amazing-feature`)\n8. Open a Pull Request\n\n### Development Guidelines\n\n- Follow Rust best practices and idioms\n- Add tests for new functionality\n- Update documentation for user-facing changes\n- Keep commits atomic and well-described\n- Ensure CI passes before submitting PR\n\n## License\n\nThis project is licensed under the GNU General Public License v3.0 - see the [LICENSE](LICENSE) file for details.\n\nThis ensures the code remains free and open source. If you modify and distribute this software, you must:\n- Disclose your source code\n- License your modifications under GPLv3\n- State significant changes made\n- Include the original copyright notice\n\n## Resources\n\n- [AT Protocol Documentation](https://atproto.com)\n- [Bluesky API Reference](https://docs.bsky.app)\n- [Jetstream Documentation](https://github.com/bluesky-social/jetstream)\n- [ATrium Rust Library](https://github.com/sugyan/atrium)\n- [Feed Generator Guide](https://docs.bsky.app/docs/starter-templates/custom-feeds)\n\n## Acknowledgments\n\n- Built with [ATrium](https://github.com/sugyan/atrium) - Rust libraries for AT Protocol\n- Uses [Jetstream](https://github.com/bluesky-social/jetstream) for efficient event streaming\n- Inspired by the Bluesky community's work on custom feeds\n\n---\n\n**Made with ❤️ for the Bluesky community**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitorpy%2Fnoreposts-atproto-feed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvitorpy%2Fnoreposts-atproto-feed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitorpy%2Fnoreposts-atproto-feed/lists"}