{"id":50723784,"url":"https://github.com/udaykumar-dhokia/trawl","last_synced_at":"2026-06-10T02:04:17.271Z","repository":{"id":344160696,"uuid":"1179869295","full_name":"udaykumar-dhokia/trawl","owner":"udaykumar-dhokia","description":"On-Premise AI Powered Research Assistant Right In Your Terminal","archived":false,"fork":false,"pushed_at":"2026-03-18T11:37:06.000Z","size":1525,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-19T02:25:53.628Z","etag":null,"topics":["ai","ai-assistant","assistant","langchain","llms","open-source","perplexity-alternative","postgresql","python","research","search-interface","searxng","self-hosted","terminal-based","tui"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/udaykumar-dhokia.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-12T13:16:45.000Z","updated_at":"2026-03-18T14:38:59.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/udaykumar-dhokia/trawl","commit_stats":null,"previous_names":["udaykumar-dhokia/searchx","udaykumar-dhokia/trawl"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/udaykumar-dhokia/trawl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/udaykumar-dhokia%2Ftrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/udaykumar-dhokia%2Ftrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/udaykumar-dhokia%2Ftrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/udaykumar-dhokia%2Ftrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/udaykumar-dhokia","download_url":"https://codeload.github.com/udaykumar-dhokia/trawl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/udaykumar-dhokia%2Ftrawl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34133408,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-assistant","assistant","langchain","llms","open-source","perplexity-alternative","postgresql","python","research","search-interface","searxng","self-hosted","terminal-based","tui"],"created_at":"2026-06-10T02:04:16.611Z","updated_at":"2026-06-10T02:04:17.263Z","avatar_url":"https://github.com/udaykumar-dhokia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# trawl - On-Premise AI Powered Research Assistant\n\n[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)\n[![FastAPI](https://img.shields.io/badge/FastAPI-005571?style=flat\u0026logo=fastapi)](https://fastapi.tiangolo.com/)\n[![Textual](https://img.shields.io/badge/Built%20with-Textual-ff00ff.svg)](https://textual.textualize.io/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**trawl** is a high-performance, On-Premise AI Powered Research Assistant with terminal interface. Inspired by modern search assistants like Perplexity, it brings deep-dive research capabilities directly to your command line.\n\n![Demo](demo.gif)\n\n---\n\n## Features\n\n- **Streaming Responses**: Real-time markdown streaming with live spinners and status updates in the terminal.\n- **DeepSearch Mode**: Analyze complex queries by automatically splitting them into 3 verticals, researching 45+ sources simultaneously, and synthesizing them into a comprehensive long-form response.\n- **Stable Research UI**: A structured terminal layout using `Rich` to show research progress, sources, and media links without flickering.\n- **Visual Research Insights**: Automatic detection and display of relevant images and videos discovered during research.\n- **Integrated Configuration**: Manage your provider (Gemini/Ollama), API keys, and models directly from the CLI.\n- **Source Citations**: Clean display of research sources with clickable links (where supported by the terminal).\n- **Persistent Threads**: Full chat history support powered by PostgreSQL and SQLAlchemy.\n- **Premium TUI**: A sleek, customizable terminal interface with both Light and Dark themes.\n- **Fast-API Backend**: Robust, asynchronous backend architecture for multi-step research.\n- **Vector Search**: pgvector-powered semantic search for efficient document retrieval.\n\n---\n\n## Roadmap\n\nHere's what's coming next to make trawl even more powerful. Contributions towards any of these are especially welcome!\n\n| Status | Feature                            | Description                                                                                                |\n| :----- | :--------------------------------- | :--------------------------------------------------------------------------------------------------------- |\n| [ ]    | **More LLM Providers**             | Support for Anthropic, OpenAI, Mistral, and other popular providers beyond.                                |\n| [x]    | **Research Modes**                 | Multiple depth levels (General search and DeepSearch) for comprehensive research.                          |\n| [ ]    | **Academic Search**                | Specialized research mode preferring academic sources like arXiv, PubMed, and Scholar.                     |\n| [ ]    | **Follow-up Question Suggestions** | After each answer, trawl will surface related questions you can instantly continue the thread with.        |\n| [ ]    | **File Upload \u0026 Analysis**         | Drop in a PDF, DOCX, or folder and ask questions against your local files — combined with live web search. |\n\n---\n\n## Technology Stack\n\n- **Frontend**: [Textual](https://textual.textualize.io/) (Rich TUI Framework)\n- **Backend**: [FastAPI](https://fastapi.tiangolo.com/) (Asynchronous Python Web Framework)\n- **Database**: [PostgreSQL](https://www.postgresql.org/) with [pgvector](https://github.com/pgvector/pgvector) and [SQLAlchemy](https://www.sqlalchemy.org/) ORM\n- **LLM Engine**: Integration with Google Gemini / Ollama\n- **Search Engine**: [SearxNG](https://github.com/searxng/searxng)\n- **Embeddings**: [Sentence Transformers](https://www.sbert.net/)\n\n---\n\n### Docker Quick Start (Recommended)\n\nThe fastest way to get `trawl` up and running is using Docker. This will start the backend, database, and search engine automatically.\n\n```bash\n# 1. Clone and enter the repository\ngit clone https://github.com/udaykumar-dhokia/trawl.git\ncd trawl\n\n# 2. Start the stack (this includes Ollama, PostgreSQL, and SearXNG)\ndocker compose up -d\n```\n\n### Interact with Docker\n\nOnce the containers are running, you can use the `trawl` CLI to perform research or manage configuration.\n\n```bash\n# Run research through the Docker backend\ndocker exec -it trawl_backend uv run trawl research \"How can we create AI Agents with LangChain?\"\n\n# View configuration inside Docker\ndocker exec -it trawl_backend uv run trawl config view\n```\n\n---\n\n## Manual Installation\n\n### Prerequisites\n\n- [Python 3.10+](https://www.python.org/downloads/)\n- [uv](https://github.com/astral-sh/uv) package manager\n- PostgreSQL database\n- SearxNG instance (optional, for web search)\n\n### Installation\n\n```bash\ngit clone https://github.com/udaykumar-dhokia/trawl.git\ncd trawl\nuv sync\n```\n\n### Configuration\n\n1. Copy the environment template:\n\n   ```bash\n   cp .env.example .env\n   ```\n\n2. Edit `.env` with your database and API keys.\n\n---\n\n## Usage\n\n### Research Mode\n\n```bash\n# Basic research query\ntrawl research \"Your query here\"\n\n# Short flag\ntrawl research --q \"Your query here\"\n\n# DeepSearch mode (More sources, 3 vertical analysis, comprehensive answer)\ntrawl research \"Your query here\" --deep\n```\n\n### Configuration Management\n\n```bash\n# View current settings\ntrawl config view\n\n# Update a setting\ntrawl config set provider google\ntrawl config set google_api_key YOUR_API_KEY\n```\n\n### Terminal User Interface (TUI)\n\n```bash\n# Start the interactive dashboard\ntrawl tui\n```\n\n### Programmatic Usage\n\n```python\nimport asyncio\nfrom trawl.services.invoke_chat import invoke_chat\n\nasync def main():\n    async for chunk in invoke_chat(query=\"How do bees fly?\"):\n        print(chunk)\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n---\n\n## Development\n\n### Setup Development Environment\n\n```bash\n# Run the development setup script\n./scripts/setup-dev.sh\n\n# Or manually\nuv sync --dev\ncp .env.example .env\n```\n\n### Running Tests\n\n```bash\n# Run all tests\nuv run pytest\n\n# With coverage\nuv run pytest --cov=trawl\n\n# Run specific test\nuv run pytest tests/test_specific.py\n```\n\n### Code Quality\n\n```bash\n# Lint code\nuv run ruff check .\n\n# Format code\nuv run ruff format .\n\n# Type checking\nuv run mypy .\n```\n\n### Building\n\n```bash\n# Build package\nuv build\n\n# Or using make\nmake build\n```\n\n---\n\n## Project Structure\n\n```\ntrawl/\n├── cli.py              # Command-line interface\n├── main.py             # FastAPI application\n├── tui_app.py          # Textual TUI interface\n├── core/               # Core configuration and utilities\n├── db/                 # Database models and connections\n├── models/             # SQLAlchemy models\n├── schemas/            # Pydantic schemas\n├── services/           # Business logic services\n└── utils/              # Utility functions\n\ntests/                  # Test suite\ndocs/                   # Documentation\nexamples/               # Usage examples\nscripts/                # Development scripts\n.github/                # GitHub Actions and templates\n```\n\n---\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests\n5. Submit a pull request\n\n---\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\n## Changelog\n\nSee [CHANGELOG.md](CHANGELOG.md) for version history.\n\n---\n\n## Support\n\n- 📖 [Documentation](https://github.com/udaykumar-dhokia/trawl#readme)\n- 🐛 [Issues](https://github.com/udaykumar-dhokia/trawl/issues)\n- 💬 [Discussions](https://github.com/udaykumar-dhokia/trawl/discussions)\n\n---\n\n## Keyboard Shortcuts (TUI)\n\n| Shortcut   | Action            |\n| :--------- | :---------------- |\n| `Ctrl + N` | New Chat          |\n| `Ctrl + R` | Refresh Chat List |\n| `Escape`   | Blur / Exit Input |\n| `Ctrl + C` | Quit Application  |\n\n---\n\n\u003cp align=\"center\"\u003eBuilt with ❤️ by udthedeveloper\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fudaykumar-dhokia%2Ftrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fudaykumar-dhokia%2Ftrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fudaykumar-dhokia%2Ftrawl/lists"}