{"id":48193062,"url":"https://github.com/devrupt-io/ethos","last_synced_at":"2026-04-04T17:56:33.675Z","repository":{"id":338918232,"uuid":"1155182553","full_name":"devrupt-io/ethos","owner":"devrupt-io","description":"Exploring what HN is really thinking","archived":false,"fork":false,"pushed_at":"2026-02-17T03:49:24.000Z","size":94,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-17T09:44:44.327Z","etag":null,"topics":["ai","hackernews","typescript","vector-database"],"latest_commit_sha":null,"homepage":"https://ethos.devrupt.io","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devrupt-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-02-11T08:08:38.000Z","updated_at":"2026-02-17T03:59:25.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/devrupt-io/ethos","commit_stats":null,"previous_names":["devrupt-io/ethos"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/devrupt-io/ethos","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devrupt-io%2Fethos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devrupt-io%2Fethos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devrupt-io%2Fethos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devrupt-io%2Fethos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devrupt-io","download_url":"https://codeload.github.com/devrupt-io/ethos/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devrupt-io%2Fethos/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31407654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","hackernews","typescript","vector-database"],"created_at":"2026-04-04T17:56:33.142Z","updated_at":"2026-04-04T17:56:33.588Z","avatar_url":"https://github.com/devrupt-io.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ethos\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![TypeScript](https://img.shields.io/badge/TypeScript-5.x-blue.svg)](https://www.typescriptlang.org/)\n[![Docker](https://img.shields.io/badge/Docker-Compose-2496ED.svg)](https://docs.docker.com/compose/)\n\n\u003e Explore what Hacker News is *really* thinking — concepts, sentiment, and discourse patterns extracted through LLM-powered semantic analysis and vector embeddings.\n\n`ethos` goes beyond surface-level HN browsing. It automatically ingests stories\nand comments, uses LLM structured output to extract deep concepts (like\n\"technological determinism\" or \"open source sustainability\"), tracks entities\n(companies, products, and OSS projects like \"OpenAI\" or \"SQLite\"), identifies\nspecific technologies, embeds them as vectors in ChromaDB, and presents insights\nabout what ideas are trending, how the community feels about specific companies\nand technologies, and what kinds of arguments people are making.\n\n**Not a proxy.** ethos doesn't just reformat HN's homepage — it analyzes the\nunderlying ideas, clusters them semantically, and surfaces patterns that aren't\nvisible from reading individual stories.\n\n## Features\n\n- 🧠 **Concept Explorer** — See what abstract ideas HN is engaging with, sized by frequency, colored by sentiment\n- 🏢 **Entity Tracker** — Track companies, products, services, and open-source projects being discussed, with community sentiment toward each\n- 📊 **Sentiment Analysis** — Community emotional temperature, controversy levels, intellectual depth\n- 💬 **Discourse Patterns** — What types of arguments people make (technical insights, counterarguments, personal experience, etc.)\n- 🔍 **Semantic Search** — Search by concept, not keywords (\"fear of AI replacing jobs\" finds relevant discussions)\n- ⚡ **Background Ingestion** — Automatic polling and processing, no manual triggers\n- 🗄️ **Smart Caching** — Already-seen stories and comments are skipped to save time and API costs\n- 🔧 **Admin Dashboard** — Monitor worker progress, analysis versions, and trigger re-analysis\n\n## Development\n\n### Built With\n\n- **OpenRouter** for LLM inference (structured output + reasoning token exclusion) and vector embeddings\n- **ChromaDB** for similarity search across embedded concepts\n- **TypeScript** as the programming language for both frontend and backend\n- **Next.js** with Tailwind CSS for the frontend\n- **Express** for the RESTful HTTP backend\n- **Sequelize** ORM with PostgreSQL for persistent storage\n- **Docker Compose** for development, testing, and deployment\n\n### Quick Start\n\n```\ngit clone https://github.com/devrupt-io/ethos.git\ncd ethos\ncp example.env .env  # then edit .env with your OpenRouter API key\ndocker compose --profile dev up -d\n```\n\nThis will bring up a frontend on `http://localhost:23100` and a backend running\non `http://localhost:23101` in development mode supporting Hot Module Reload\n(HMR) allowing for rapid development. Under the hood Next.js redirects all of\nthe `/api/*` URLs to the backend.\n\nThe background worker starts automatically on boot and begins ingesting HN\nstories and comments. Concepts, sentiment, and discourse data will appear in the\nUI within a few minutes.\n\n### Architecture\n\n```\nFrontend (Next.js + Tailwind)\n  ├── Concept Explorer (trending ideas + sentiment)\n  ├── Entity Tracker (companies, products, OSS projects + sentiment)\n  ├── Sentiment Dashboard (controversy, depth)\n  ├── Discourse View (argument types, strong opinions)\n  └── Semantic Search (vector similarity)\n            ↓ /api/* proxy\nBackend (Express + TypeScript)\n  ├── Background Worker (auto-polls HN every 5min)\n  │     ├── Fetches top stories + comments\n  │     ├── LLM Analysis (structured output via OpenRouter)\n  │     │     ├── Concepts (abstract ideas, philosophies)\n  │     │     ├── Entities (companies, products, services)\n  │     │     ├── Technologies (languages, frameworks, tools)\n  │     │     └── Sentiment + controversy scoring\n  │     └── Vector Embedding (stored in ChromaDB)\n  ├── PostgreSQL (stories, comments, analysis metadata)\n  ├── ChromaDB (vector similarity search)\n  └── OpenRouter (Qwen models: chat + embeddings)\n```\n\n### Analysis Model\n\nethos extracts three complementary dimensions from every HN story and comment:\n\n- **Concepts** — Abstract ideas, philosophies, and themes (e.g. \"open source sustainability\", \"surveillance capitalism\", \"right to repair\")\n- **Entities** — Companies, brands, products, services, and notable OSS projects (e.g. \"OpenAI\", \"Hetzner\", \"SQLite\", \"ChatGPT\")\n- **Technologies** — Programming languages, frameworks, tools, and platforms (e.g. \"Rust\", \"PostgreSQL\", \"Kubernetes\")\n\nThis separation ensures users can track both the philosophical discourse *and* the concrete products/companies the community is discussing. Sentiment is scored independently for each item, so you can see that HN loves SQLite but is skeptical of certain SaaS pricing models.\n\n### Testing\n\nThere is a `run-tests.sh` script which uses an ephemeral testing container to\nrun all of the tests in a clean environment with a separate database than\nproduction that is wiped before test runs.\n\n```\n./run-tests.sh\n```\n\nYou may also run `./run-tests.sh --last` to see the output from the last test\nrun without re-running the tests, which is useful for grepping for different\nthings or reviewing test results.\n\n### API Endpoints\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| GET | `/api/health` | Health check with data counts |\n| GET | `/api/stories` | List analyzed stories with concepts (paginated) |\n| GET | `/api/stories/:hnId` | Get a story by HN ID |\n| GET | `/api/comments` | List comments (paginated, filterable by storyHnId) |\n| POST | `/api/search` | Semantic search by concept across stories or comments |\n| GET | `/api/insights/concepts` | Trending concepts with sentiment and story connections |\n| GET | `/api/insights/concepts/:name` | Detailed view of a specific concept with stories and comments |\n| GET | `/api/insights/entities` | Trending companies, products, and OSS projects with sentiment |\n| GET | `/api/insights/sentiment` | Sentiment distribution, controversy, and depth metrics |\n| GET | `/api/insights/discourse` | Comment type distribution and strongest arguments |\n| GET | `/api/insights/timeline` | Time-series data for dashboard charts |\n| POST | `/api/admin/login` | Admin authentication |\n| GET | `/api/admin/status` | Combined health, worker, and analysis status (auth required) |\n| POST | `/api/admin/regenerate` | Re-analyze items with outdated analysis versions (auth required) |\n\n### Deployment\n\nA Caddyfile is provided that is used to serve the docker containers in\nproduction. In this configuration the frontend is served on `localhost:23110`\nand the backend on `localhost:23111`, with Caddy being used to serve both under\na single domain such as `ethos.devrupt.io` without HMR.\n\nAll configurations are available in the `.env` file at the top level of this\nrepository and a `example.env` file is provided to help you get started.\n\n_(Note: Postgres is intentionally never exposed outside of the container stack\nand you NEED to set a strong password if you expose it or your container will\nget popped in seconds)_\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is open source under the MIT License.\n\n## FAQ\n\n### What is Hacker News?\n\nHacker News (HN) is a community where technology and enthusiasts share and\ncomment on stories. It was created by [Y Combinator](https://ycombinator.com)\n\n### What value does HN provide?\n\nThe community is often ahead of Reddit or Facebook with interesting or impactful\nevents or insights because many work for large companies or the government.\n\n### What is HN lacking?\n\nHN is intentionally designed to be a simple website without many features. For\nexample, the website uses very minimal javascript and offers very limited\ntheming.\n\nThese lack of features lead the community to instead fill the gaps as HN is very\nopen with their data.\n\n### How is HN open with their data?\n\nHN provides a free and easy to use [API](https://github.com/HackerNews/API)\nallowing anyone access to resources such as stories, comments, users with\nsupport for filtering. For example, you can easily request all of the stories\nabout Google within the last week.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevrupt-io%2Fethos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevrupt-io%2Fethos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevrupt-io%2Fethos/lists"}