{"id":21012612,"url":"https://github.com/developersdigest/internet-enabled-llms","last_synced_at":"2025-08-21T12:42:20.871Z","repository":{"id":260313262,"uuid":"880942135","full_name":"developersdigest/internet-enabled-llms","owner":"developersdigest","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-30T23:59:11.000Z","size":258,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-12T09:08:40.532Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/developersdigest.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-30T16:26:50.000Z","updated_at":"2025-01-16T18:01:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"54d75f26-4b7a-4aae-bfb1-003160f9eca6","html_url":"https://github.com/developersdigest/internet-enabled-llms","commit_stats":null,"previous_names":["developersdigest/internet-enabled-llms"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developersdigest%2Finternet-enabled-llms","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developersdigest%2Finternet-enabled-llms/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developersdigest%2Finternet-enabled-llms/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developersdigest%2Finternet-enabled-llms/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/developersdigest","download_url":"https://codeload.github.com/developersdigest/internet-enabled-llms/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248543850,"owners_count":21121838,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T09:37:51.561Z","updated_at":"2025-04-12T09:12:17.626Z","avatar_url":"https://github.com/developersdigest.png","language":"TypeScript","readme":"# Internet-Enabled LLMs\n\nThis API provides a powerful and flexible solution for web data extraction and LLM-powered question answering. It uses multiple integrations, including Firecrawl, OpenAI, Not Diamond, and Serper, to collect, process, and provide relevant information from web pages, enhancing user queries with contextual data. \n\n## Getting Started\n\n### Installation\n1. Clone the repository:\n   ```sh\n   git clone https://github.com/developersdigest/internet-enabled-llms.git\n   cd internet-enabled-llms\n   ```\n2. Install dependencies:\n   ```sh\n   bun install\n   ```\n3. Create a `.env` file in the root directory and add your API keys:\n   ```env\n   FIRECRAWL_API_KEY=your_firecrawl_api_key\n   NOTDIAMOND_API_KEY=your_notdiamond_api_key\n   OPENAI_API_KEY=your_openai_api_key\n   ANTHROPIC_API_KEY=your_anthropic_api_key\n   GOOGLE_API_KEY=your_google_api_key\n   SERPER_API_KEY=your_serper_api_key\n   ```\n\n### Running the Server\nTo start the server, run:\n```sh\nbun dev\n```\nThis will start the server in development mode using Next.js.\n\n## Environment Variables\nThe API requires the following environment variables:\n- `FIRECRAWL_API_KEY` - Firecrawl API key for web scraping.\n- `NOTDIAMOND_API_KEY` - NotDiamond API key for LLM handling.\n- `OPENAI_API_KEY` - OpenAI API key for embeddings and LLMs.\n- `ANTHROPIC_API_KEY` - Anthropic API key for LLMs.\n- `GOOGLE_API_KEY` - Google API key for LLMs.\n- `SERPER_API_KEY` - Serper API key for web search.\n\n## Endpoints\n\n### POST Request\n- **URL**: `/api/llm/`\n- **Method**: `POST`\n- **Description**: Processes a user message, searches for relevant pages, scrapes the content, optionally uses embeddings, and then provides an LLM-powered response.\n\n#### Request Body\nThe request body should be in JSON format with the following fields:\n- `message` (string, required): The user query.\n- `pagesToCrawl` (number, optional): Number of pages to crawl for context (default is 3, max is 10).\n- `skipEmbeddings` (boolean, optional): If `true`, embeddings are skipped and raw content is processed instead.\n\nExample:\n```json\n{\n  \"message\": \"When did chatgpt canvas come out?\",\n  \"pagesToCrawl\": 5,\n  \"skipEmbeddings\": false\n}\n```\n\n#### Response\nThe response is in JSON format and includes:\n- `answer` (string): The generated response from the LLM.\n- `selectedModel` (object): Details about the model used.\n- `crawlInfo` (object): Information about the crawling and embedding process.\n- `sources` (array): A list of sources that were used to generate the response.\n\nExample Response:\n```json\n{\n  \"answer\": \"ChatGPT Canvas came out on October 3, 2024.\",\n  \"selectedModel\": {\n    \"provider\": \"openai\",\n    \"model\": \"gpt-4o-mini\"\n  },\n  \"crawlInfo\": {\n    \"requestedPages\": 5,\n    \"actualPagesCrawled\": 4,\n    \"embeddingsUsed\": true\n  },\n  \"sources\": [\n    {\n      \"title\": \"ChatGPT Canvas Release Date\",\n      \"url\": \"https://example.com/chatgpt-canvas-release\"\n    }\n  ]\n}\n```\n\n## Detailed Flow\n1. **Request Parsing**: Extract the user query and validate the request.\n2. **Web Search \u0026 Filtering**: Search using Serper and filter out unsupported domains (e.g., social media).\n3. **Scraping \u0026 Content Processing**: Scrape relevant content from filtered URLs and optionally create embeddings.\n4. **LLM Query Handling**: Generate an answer using one of the LLMs (OpenAI, Anthropic, or Gemini).\n5. **Return the Response**: Send the answer along with source and model information.\n\n## License\nThis project is licensed under the MIT License.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevelopersdigest%2Finternet-enabled-llms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevelopersdigest%2Finternet-enabled-llms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevelopersdigest%2Finternet-enabled-llms/lists"}