{"id":14070240,"url":"https://github.com/langchain-ai/weblangchain","last_synced_at":"2025-04-04T14:08:19.351Z","repository":{"id":198134924,"uuid":"700142988","full_name":"langchain-ai/weblangchain","owner":"langchain-ai","description":"LangChain-powered web researcher chatbot. Searches for sources on the web and cites them in generated answers.","archived":false,"fork":false,"pushed_at":"2024-03-21T14:26:02.000Z","size":195,"stargazers_count":540,"open_issues_count":4,"forks_count":110,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-03-28T13:11:11.199Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://weblangchain.vercel.app","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/langchain-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-04T03:02:21.000Z","updated_at":"2025-03-27T03:47:53.000Z","dependencies_parsed_at":"2023-10-20T05:24:12.299Z","dependency_job_id":"302d67cc-2aa2-4352-8f66-f8805626e6a9","html_url":"https://github.com/langchain-ai/weblangchain","commit_stats":null,"previous_names":["langchain-ai/weblangchain"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Fweblangchain","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Fweblangchain/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Fweblangchain/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Fweblangchain/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/langchain-ai","download_url":"https://codeload.github.com/langchain-ai/weblangchain/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247190252,"owners_count":20898702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-13T07:07:35.661Z","updated_at":"2025-04-04T14:08:19.321Z","avatar_url":"https://github.com/langchain-ai.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# 🦜️🌐 WebLangChain\n\nThis repo is an example of performing retrieval using the entire internet as a document store.\n\n**Try it live:** [weblangchain.vercel.app](https://weblangchain.vercel.app)\n\n## ✅ Running locally\n\nBy default, WebLangChain uses [Tavily](https://tavily.com) to fetch content from webpages. You can get an API key from [by signing up](https://tavily.com/).\nIf you'd like to add or swap in different base retrievers (e.g. if you want to use your own data source), you can update the `get_retriever()` method in `main.py`.\n\n1. Install backend dependencies: `poetry install`.\n2. Make sure to set your environment variables to configure the application:\n```\nexport OPENAI_API_KEY=\nexport TAVILY_API_KEY=\n\n# for Anthropic\n# remove models from code if unused\nANTHROPIC_API_KEY=\n\n# if you'd like to use the You.com retriever\nexport YDC_API_KEY=\n\n# if you'd like to use the Google retriever\nexport GOOGLE_CSE_ID=\nexport GOOGLE_API_KEY=\n\n# if you'd like to use the Kay.ai retriever\nexport KAY_API_KEY=\n\n# for tracing\nexport LANGCHAIN_TRACING_V2=true\nexport LANGCHAIN_ENDPOINT=\"https://api.smith.langchain.com\"\nexport LANGCHAIN_API_KEY=\nexport LANGCHAIN_PROJECT=\n```\n\nYou will also need to put your Google Cloud credentials in a JSON file under `.google_vertex_ai_credentials.json` in the main directory if you would like to\nuse Google Vertex as an option. If you're not using Vertex, you'll need to remove `ChatVertexAI` from `main.py`.\n\n3. Start the Python backend with `poetry run make start`.\n4. Install frontend dependencies by running `cd nextjs`, then `yarn`.\n5. Run the frontend with `yarn dev` for frontend.\n6. Open [localhost:3000](http://localhost:3000) in your browser.\n\nUnder the hood, the chain is converted to a FastAPI server with various endpoints via [LangServe](https://github.com/langchain-ai/langserve).\nThis also includes a playground that you can use to interactively swap and configure various pieces of the chain.\nYou can find it running at `http://localhost:8080/chat/playground`.\n\n## ☕ Running locally (JS backend)\n\nNote that LangServe is not currently supported in JS, and customization of the retriever and model, as well as the playground, are unavailable.\n\n1. Install frontend dependencies by running `cd nextjs`, then `yarn`.\n2. Populate a `nextjs/.env.local` file with your own versions of keys from the `nextjs/.env.example` file, and set `NEXT_PUBLIC_API_BASE_URL` to `\"http://localhost:3000/api\"`.\n3. Run the app with `yarn dev`.\n4. Open [localhost:3000](http://localhost:3000) in your browser.\n\n## ⚙️ How it works\n\nThe general retrieval flow looks like this:\n\n1. Pull in raw content related to the user's initial query using a retriever that wraps Tavily's Search API.\n    - For subsequent conversation turns, we also rephrase the original query into a \"standalone query\" free of references to previous chat history.\n2. Because the size of the raw documents usually exceed the maximum context window size of the model, we perform additional [contextual compression steps](https://python.langchain.com/docs/modules/data_connection/retrievers/contextual_compression/) to filter what we pass to the model.\n    - First, we split retrieved documents using a [text splitter](https://python.langchain.com/docs/modules/data_connection/document_transformers/).\n    - Then we use an [embeddings filter](https://python.langchain.com/docs/modules/data_connection/retrievers/contextual_compression/#embeddingsfilter) to remove any chunks that do not meet a similarity threshold with the initial query.\n3. The retrieved context, the chat history, and the original question are passed to the LLM as context for the final generation.\n\nHere's a LangSmith trace illustrating the above:\n\nhttps://smith.langchain.com/public/f4493d9c-218b-404a-a890-31c15c56fff3/r\n\nIt's built using:\n\n- [Tavily](https://tavily.com) as a retriever\n- [LangChain](https://github.com/langchain-ai/langchain/) for orchestration\n- [LangServe](https://github.com/langchain-ai/langserve) to directly expose LangChain runnables as endpoints\n- [FastAPI](https://fastapi.tiangolo.com/)\n- [Next.js](https://nextjs.org) for the frontend\n\n## 🚀 Deployment\n\nThe live version is hosted on [Fly.dev](https://fly.dev) and [Vercel](https://vercel.com).\nThe backend Python logic is found in `main.py`, and the frontend Next.js app is under `nextjs/`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangchain-ai%2Fweblangchain","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flangchain-ai%2Fweblangchain","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangchain-ai%2Fweblangchain/lists"}