{"id":29248508,"url":"https://github.com/polareth/nightwatch","last_synced_at":"2026-01-20T16:30:22.731Z","repository":{"id":288041458,"uuid":"965603762","full_name":"polareth/nightwatch","owner":"polareth","description":"A public archive of investigations into crypto scams and bad actors.","archived":false,"fork":false,"pushed_at":"2025-05-30T07:06:33.000Z","size":2440,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-10T03:37:11.455Z","etag":null,"topics":["archive","blockchain","investigation","scams","sleuth"],"latest_commit_sha":null,"homepage":"https://nightwatch.polareth.org","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/polareth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-13T14:27:13.000Z","updated_at":"2025-05-30T07:06:37.000Z","dependencies_parsed_at":"2025-07-22T08:35:16.135Z","dependency_job_id":"879dfa3d-fd8e-4e82-a995-2aa4f51d3d9e","html_url":"https://github.com/polareth/nightwatch","commit_stats":null,"previous_names":["polareth/nightwatch"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/polareth/nightwatch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polareth%2Fnightwatch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polareth%2Fnightwatch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polareth%2Fnightwatch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polareth%2Fnightwatch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/polareth","download_url":"https://codeload.github.com/polareth/nightwatch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polareth%2Fnightwatch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28607159,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T16:10:39.856Z","status":"ssl_error","status_checked_at":"2026-01-20T16:10:39.493Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archive","blockchain","investigation","scams","sleuth"],"created_at":"2025-07-04T00:08:00.171Z","updated_at":"2026-01-20T16:30:22.708Z","avatar_url":"https://github.com/polareth.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nightwatch\n\n**A public archive of investigations into crypto scams and bad actors.**\n\n![Nightwatch](./public/logo-white.png)\n\nNightwatch collects and preserves tweets and Telegram messages from trusted blockchain sleuths, acting as a curated and convenient searchable record of their work.\n\nA ledger of exposure. A watchful memory. A stain that won't fade.\n\n## Table of Contents\n\n- [Introduction](#introduction)\n  - [Overview](#overview)\n  - [Key Features](#key-features)\n- [Getting Started](#getting-started)\n  - [Prerequisites](#prerequisites)\n  - [Installation](#installation)\n  - [Development](#development)\n  - [Deployment](#deployment)\n- [Architecture](#architecture)\n  - [Data Flow](#data-flow)\n  - [API Endpoints](#api-endpoints)\n  - [Database Schema](#database-schema)\n- [Technical Details](#technical-details)\n  - [API Integration](#api-integration)\n  - [Caching Strategy](#caching-strategy)\n  - [Scheduled Jobs](#scheduled-jobs)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Introduction\n\n### Overview\n\nNightwatch serves as a permanent archive for investigations conducted by trusted blockchain investigators like @zachxbt, sourced from both Twitter and Telegram.\n\nThe application indexes tweets and messages from selected accounts/channels, along with media attachments and metadata. It reconstructs Twitter conversation threads and Telegram reply chains to provide better context. This creates a reliable reference point to research potential scams and bad actors, with an easily searchable interface.\n\n### Key Features\n\n- **Permanent Archive**: Tweets and Telegram messages are stored in a database, ensuring they remain accessible even if deleted from the source platforms.\n- **Full-Text Search**: Search through the entire archive using keywords across both platforms.\n- **Conversation Context**: View entire Twitter threads and Telegram reply chains from relevant accounts/channels.\n- **Media Preservation**: Images and videos attached to tweets are preserved and viewable..\n- **Regular Updates**: Automatic synchronization with Twitter and Telegram to capture new content.\n\n## Getting Started\n\n### Prerequisites\n\n- [Deno](https://deno.com/) (v1.37 or higher)\n- [Node.js](https://nodejs.org/) (v20 or higher)\n- [pnpm](https://pnpm.io/) (v8 or higher)\n- A [Neon](https://neon.tech/) PostgreSQL database\n- A [TwitterAPI.io](https://twitterapi.io/) API key\n- Telegram API Credentials (`TELEGRAM_API_ID`, `TELEGRAM_API_HASH`)\n- A Telegram User Session String (`TELEGRAM_SESSION`) generated with `pnpm telegram-login`\n\n### Installation\n\n1. Clone the repository:\n\n   ```bash\n   git clone https://github.com/polareth/nightwatch.git\n   cd nightwatch\n   ```\n\n2. Install dependencies:\n\n   ```bash\n   pnpm install\n   ```\n\n3. Set up environment variables:\n\n   ```\n   NEON_DATABASE_URL=your_neon_postgres_connection_string\n   TWITTERAPI_API_KEY=your_twitterapi_io_key\n   CRON_SECRET=your_secret_for_cron_jobs\n   TELEGRAM_API_ID=your_telegram_api_id\n   TELEGRAM_API_HASH=your_telegram_api_hash\n   TELEGRAM_SESSION=your_telegram_session_string # See Telegram section below\n   ```\n\n4. Generate a Telegram session string (if you don't have one):\n   ```bash\n   pnpm telegram-login\n   ```\n   Follow the prompts to log in with your Telegram account. The session string will be printed to the console. Add it to your environment variables (`TELEGRAM_SESSION`).\n\n### Development\n\nRun the development server:\n\n```bash\npnpm dev\n```\n\nThe application will be available at http://localhost:5173.\n\n### Deployment\n\nNightwatch is designed to be deployed on [Deno Deploy](https://deno.com/deploy). The repository includes a GitHub Actions workflow for automatic deployment.\n\n1. Build the application:\n\n   ```bash\n   pnpm build\n   ```\n\n2. Deploy manually (if not using GitHub Actions):\n   ```bash\n   pnpm run deploy\n   ```\n\n## Architecture\n\n### Data Flow\n\n1.  **Data Collection**: Tweets from specified accounts are fetched from [TwitterAPI.io](https://twitterapi.io/). Messages from specified channels are fetched using [the Telegram API](https://core.telegram.org/api).\n2.  **Data Processing**: Content is parsed and normalized, extracting mentions, URLs, media (Twitter), and reply structures.\n3.  **Data Storage**: Processed content is stored in a [Neon](https://neon.tech/) database.\n4.  **Data Retrieval**: Users query the database through the search interface via the `/api/search` endpoint.\n5.  **Data Presentation**: Results are displayed with highlighting, conversation context, and media previews (Twitter) or indicators (Telegram).\n\n### API Endpoints\n\n- **`/api/search`**: Search for tweets and Telegram messages matching a query.\n- **`/api/periodic-sync`**: Trigger a synchronization with Twitter and Telegram (protected by auth). Fetches new content since the last sync.\n- **`/api/initial-sync`**: Perform initial backfill for specific Twitter users or Telegram channels (protected by auth).\n- **`/api/health`**: Check the health of the application and its dependencies.\n\n### Database Schema\n\nThe database uses four main tables:\n\n1.  **`tw_users`**: Stores information about Twitter authors.\n\n    - `id`: bigint (Twitter user ID)\n    - `username`: text\n    - `display_name`: text\n    - `profile_picture_url`: text\n    - `followers`: integer\n    - `following`: integer\n    - `profile_bio`: jsonb (bio, mentions, urls)\n\n2.  **`tw_posts`**: Stores the tweets.\n\n    - `id`: bigint (Tweet ID)\n    - `url`: text\n    - `text`: text\n    - `user_id`: bigint (FK to `tw_users.id`)\n    - `conversation_id`: bigint\n    - `created_at`: timestamptz\n    - `user_mentions`: jsonb (array of `DbMentionType`)\n    - `urls`: jsonb (array of `DbUrlType`)\n    - `medias`: jsonb (array of `DbMediaType`)\n    - `fts_tokens`: tsvector (for full-text search)\n\n3.  **`tg_channels`**: Stores information about Telegram channels.\n\n    - `id`: bigint (Telegram channel ID)\n    - `title`: text\n    - `about`: text\n    - `channel_username`: text\n    - `admin_usernames`: text[]\n\n4.  **`tg_messages`**: Stores Telegram messages.\n    - `id`: text (Composite: `channel_id-message_id`)\n    - `message_id`: bigint\n    - `message`: text\n    - `url`: text\n    - `channel_id`: bigint (FK to `tg_channels.id`)\n    - `reply_to_message_id`: bigint (Original message ID it replies to)\n    - `created_at`: timestamptz\n    - `urls`: jsonb (array of `DbUrlType`)\n    - `has_media`: boolean\n    - `thread_id`: text (ID of the root message in the reply chain)\n    - `fts_tokens`: tsvector (for full-text search)\n\nYou can directly use [the reference SQL schema](./resources/init.sql) to create the database.\n\n## Technical Details\n\n### API Integration\n\n- **Twitter**: Uses [TwitterAPI.io](https://twitterapi.io/) advanced search. Implements batch processing, cursor-based pagination, and differential updates (fetching only new tweets). See [`app/lib/sync.server.ts`](./app/lib/sync.server.ts#L69-L144).\n- **Telegram**: Uses [`telejs`](https://github.com/gram-js/telejs) to connect directly to the Telegram API as a user. Fetches channel info and messages, performing differential updates based on the last stored message ID. Requires API credentials and a user session string. See [`app/lib/sync.server.ts`](./app/lib/sync.server.ts#L147-L233).\n\nYou can manually trigger syncs via the API endpoints:\n\n```bash\n# Periodic sync (fetches new content for all configured sources)\ncurl -X POST \"http://localhost:5173/api/periodic-sync\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $CRON_SECRET\"\n\n# Initial sync (backfills specific sources)\n# Twitter user:\ncurl -X POST \"http://localhost:5173/api/initial-sync?twitter=zachxbt\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $CRON_SECRET\"\n\n# Telegram channel:\ncurl -X POST \"http://localhost:5173/api/initial-sync?telegram=some_channel_username\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $CRON_SECRET\"\n\n# Multiple sources:\ncurl -X POST \"http://localhost:5173/api/initial-sync?twitter=userA,userB\u0026telegram=channelA,channelB\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $CRON_SECRET\"\n```\n\nYou can customize the relevant users and channels in [`app/lib/constants.server.ts`](./app/lib/constants.server.ts) at `RELEVANT_SOURCES`. Same for the `BATCH_SIZE`.\n\n### Caching Strategy\n\nThe application implements caching for search results to improve performance and reduce database load:\n\n- **Search Results Caching**: `/api/search` results are cached for 1 hour.\n\nYou can customize the cache TTL in [`app/lib/constants.server.ts`](./app/lib/constants.server.ts) at `CACHE_TTL`.\n\n### Scheduled Jobs\n\nNightwatch uses Deno's built-in cron functionality (`Deno.cron`) for regular updates:\n\n- **Content Synchronization**: Runs `/api/periodic-sync` every 6 hours to fetch new tweets and messages.\n- **Authentication**: Jobs are protected by a secret token (`CRON_SECRET`) to prevent unauthorized access.\n\nYou can customize the cron schedule directly in [`server.production.ts`](./server.production.ts).\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1.  Fork the repository\n2.  Create your feature branch (`git checkout -b feature/amazing-feature`)\n3.  Commit your changes (`git commit -m 'Add some amazing feature'`)\n4.  Push to the branch (`git push origin feature/amazing-feature`)\n5.  Open a Pull Request\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE file](./LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpolareth%2Fnightwatch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpolareth%2Fnightwatch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpolareth%2Fnightwatch/lists"}