{"id":14064883,"url":"https://github.com/different-ai/embedbase-internet-search","last_synced_at":"2025-04-25T20:11:02.253Z","repository":{"id":169523734,"uuid":"643828356","full_name":"different-ai/embedbase-internet-search","owner":"different-ai","description":"An internet search extension for Embedbase","archived":false,"fork":false,"pushed_at":"2023-05-26T14:06:17.000Z","size":211,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-22T14:57:21.298Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/different-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-22T08:39:58.000Z","updated_at":"2024-04-19T19:51:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"94696eae-08b7-4f03-9547-8daae4689086","html_url":"https://github.com/different-ai/embedbase-internet-search","commit_stats":null,"previous_names":["different-ai/embedbase-internet-search"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/different-ai%2Fembedbase-internet-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/different-ai%2Fembedbase-internet-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/different-ai%2Fembedbase-internet-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/different-ai%2Fembedbase-internet-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/different-ai","download_url":"https://codeload.github.com/different-ai/embedbase-internet-search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250887507,"owners_count":21503033,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-13T07:04:09.257Z","updated_at":"2025-04-25T20:11:02.237Z","avatar_url":"https://github.com/different-ai.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\n# embedbase-internet-search\n\n\u003cdiv align=\"center\"\u003e\n\nembedbase-internet-search - internet search extension for [Embedbase](https://github.com/different-ai/embedbase)\n\u003cbr\u003e\n\u003cbr\u003e\n⚠️ Status: Alpha release ⚠️\n\u003cbr\u003e\n\u003cbr\u003e\n\u003ca href=\"https://discord.gg/pMNeuGrDky\"\u003e\u003cimg alt=\"Discord\" src=\"https://img.shields.io/discord/1066022656845025310?color=black\u0026style=for-the-badge\"\u003e\u003c/a\u003e\n\u003ca href=\"https://badge.fury.io/py/embedbase-internet-search\"\u003e\u003cimg alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/embedbase-internet-search?color=black\u0026style=for-the-badge\"\u003e\u003c/a\u003e\n\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![License](https://img.shields.io/github/license/different-ai/embedbase)](https://github.com/different-ai/embedbase-internet-search/blob/main/LICENSE)\n![Coverage Report](assets/images/coverage.svg)\n\n\u003c/div\u003e\n\nThe point of internet search in embedbase is to combine your private information with latest public information.\n\nAlso remember that AIs like ChatGPT have limited knowledge to a certain date, for example try to ask ChatGPT about GPT4 or about Sam Altman talk with the senate (which happened few days ago), it will not know about it.\n\n\nhttps://github.com/different-ai/embedbase-internet-search/assets/25003283/3131450f-93e2-46b1-8dcb-30673e5abe70\n\n\nPlease check [examples](./examples/answer-question/README.md) for usage or keep reading.\n\n## Quick tour\n\nHere's an example to answer general purpose questions using embedbase hosted:\n\nThe recommended workflow is like this:\n1. search your question using internet endpoint\n2. (optional) add results to embedbase\n3. (optional) search embedbase with the question\n4. use `.generate()` to get your question answered\n\n```ts\n// npm i embedbase-js\nimport { createClient } from 'embedbase-js'\n\nconst formatInternetResultsInPrompt = (internetResult: any) =\u003e\n    `Name: ${internetResult.title}\nSnippet: ${internetResult.snippet}\nUrl: ${internetResult.url}`\n\n\nconst system = `You are an AI assistant that can answer questions.\nWhen a user send a question, we will answer its question following these steps:\n1. we will search the internet with the user's question.\n2. we will ask you to answer the question based on the internet results.`\n\nconst fn = async () =\u003e {\n    const embedbase = createClient('https://api.embedbase.xyz', process.env.EMBEDBASE_API_KEY)\n\n    // get question from process.argv\n    const question = process.argv[2]\n\n    if (!question) {\n        console.log('Please provide a question as argument, for example \"What is GPT4?\"')\n        return\n    }\n\n    const results = await embedbase.internetSearch(question)\n\n    const answerQuestionPrompt = `Based on the following internet search results:\n${results.map(formatInternetResultsInPrompt).join('\\n')}\n\\n\nPlease answer the question: ${question}`\n\n    for await (const result of embedbase.generate(answerQuestionPrompt, {\n        history: [{\n            role: 'system',\n            content: system\n        }],\n    })) {\n        console.log(result)\n    }\n}\n\nfn()\n```\n\n```bash\nEMBEDBASE_API_KEY=\"\u003cget me here https://app.embedbase.xyz\u003e\" npx tsx answer.ts \"What did Sam Altman say to to US Senate lately?\"\n```\n\n## Self-hosted usage\n\nJust add two lines to your original embedbase entrypoint:\n\n```py\nimport os\nimport uvicorn\nfrom embedbase import get_app\nfrom embedbase.database.memory_db import MemoryDatabase\nfrom embedbase.embedding.openai import OpenAI\n# import this\nfrom embedbase_internet_search import internet_search\n\napp = get_app().use_db(MemoryDatabase()).use_embedder(OpenAI(os.environ[\"OPENAI_API_KEY\"])).run()\n# add the new endpoint\napp.add_api_route(\"/internet-search\", internet_search, methods=[\"POST\"])\n\nif __name__ == \"__main__\":\n    uvicorn.run(app)\n```\n\n[Check how it is used in Embedbase hosted](https://github.com/different-ai/embedbase/blob/4e1181eb8e162e1a9544bf72461a721d10f458fc/hosted/main.py#L38)\n\nIf you have any feedback or issues, please let us know by opening an issue or contacting us on [discord](https://discord.gg/pMNeuGrDky).\n\nRegarding the SDK, please refer to the [documentation](https://docs.embedbase.xyz/sdk).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdifferent-ai%2Fembedbase-internet-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdifferent-ai%2Fembedbase-internet-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdifferent-ai%2Fembedbase-internet-search/lists"}