{"id":27710467,"url":"https://github.com/outpoot/vyntr","last_synced_at":"2025-05-15T12:11:41.164Z","repository":{"id":289103116,"uuid":"941590449","full_name":"outpoot/vyntr","owner":"outpoot","description":"Independent search engine. Includes web crawling, search indexing, dictionary API, and more. https://vyntr.com","archived":false,"fork":false,"pushed_at":"2025-05-01T11:11:20.000Z","size":9099,"stargazers_count":255,"open_issues_count":6,"forks_count":20,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-05-01T12:23:46.177Z","etag":null,"topics":["crawler","duckduckgo","engine","google","python","rust","search","tantivy","web"],"latest_commit_sha":null,"homepage":"https://vyntr.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/outpoot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-02T16:47:19.000Z","updated_at":"2025-05-01T11:55:41.000Z","dependencies_parsed_at":"2025-04-21T15:47:40.199Z","dependency_job_id":null,"html_url":"https://github.com/outpoot/vyntr","commit_stats":null,"previous_names":["outpoot/vyntr"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outpoot%2Fvyntr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outpoot%2Fvyntr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outpoot%2Fvyntr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outpoot%2Fvyntr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/outpoot","download_url":"https://codeload.github.com/outpoot/vyntr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254337539,"owners_count":22054254,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","duckduckgo","engine","google","python","rust","search","tantivy","web"],"created_at":"2025-04-26T16:03:11.792Z","updated_at":"2025-05-15T12:11:41.134Z","avatar_url":"https://github.com/outpoot.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"\u003cimg style=\"width: 128px; height: 128px\" src=\"website/static/favicon_white.svg\" /\u003e\u003ch1 style=\"font-size: 48px\"\u003e\u003ca href=\"https://vyntr.com\"\u003eVyntr.com\u003c/a\u003e - the independent search engine.\u003c/h1\u003e\n[Privacy Policy](https://vyntr.com/legal/privacy) | [Terms of Service](https://vyntr.com/legal/terms) | [License](LICENSE) | [YouTube video](https://youtu.be/-DzzCA1mGow)\n\nVyntr is a search engine project with multiple components:\n\n## Components\n\n- [Genesis](genesis/README.md) - Web crawler and content analyzer\n- [Pulse](pulse/README.md) - Search indexing system using Tantivy\n- [Lexicon](lexicon/README.md) - WordNet-based dictionary lookup service\n- [Website](website/README.md) - Frontend interface at vyntr.com\n\n## Setup\n\n1. Create a `.env` file in the root directory:\n\n```bash\n# Database\nPRIVATE_DB_URL=\"postgresql://postgres:your_password@serverip:port/postgres\"\n\n# AWS S3/Compatible Storage\nS3_ENDPOINT=\"https://s3.eu-central-1.amazonaws.com\"\nS3_REGION=\"eu-central-1\"\nS3_BUCKET=\"vyntr\"\nAWS_ACCESS_KEY_ID=\"your-key-id\"\nAWS_SECRET_ACCESS_KEY=\"your-secret-key\"\n```\n\n2. Set up the database:\n```bash\ncd genesis/tools/database\ndocker compose up -d\n```\n\n3. Set up individual components:\n- Genesis crawler: Follow [genesis setup](genesis/README.md)\n- Lexicon service: Follow [lexicon setup](lexicon/README.md)\n- Website: Follow [website setup](website/README.md)\n\n## Pipeline\n\n1. `Genesis` crawler collects and analyzes web pages\n2. Data is stored in partitioned JSONL files in `S3`\n3. Content is cleaned through `dataset`.\n3. Content is processed through `embedding` tools (vector), or `Pulse` (full-text).\n4. Website frontend provides search interface.\n\n## Requirements\n\n- Python with [uv](https://github.com/astral-sh/uv) package manager\n- Node.js\n- PostgreSQL with pgvector\n- Docker\n- Bun runtime (for Lexicon service)\n- Rust toolchain\n\n## Dataset\nThe Vyntr dataset is not publicly available. For licensing inquiries, please contact contact@outpoot.com.\n\nYou may also use the official API provided at https://vyntr.com/api.\n## License\n\nThis project is licensed under the **Creative Commons Attribution-NonCommercial 4.0 International** License (**CC BY-NC 4.0)**. See the [LICENSE](LICENSE) file for details.\n\nIndividual components may have additional licensing requirements. See their respective directories for specific licensing information.\n\nWordNet data used in Lexicon is subject to the [WordNet License](https://creativecommons.org/licenses/by/4.0/).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foutpoot%2Fvyntr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foutpoot%2Fvyntr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foutpoot%2Fvyntr/lists"}