{"id":27370146,"url":"https://github.com/datastaxdevs/workshop-wikipedia-qa","last_synced_at":"2025-04-13T08:48:07.534Z","repository":{"id":204622051,"uuid":"712284750","full_name":"datastaxdevs/workshop-wikipedia-qa","owner":"datastaxdevs","description":"Real-time document Q\u0026A using Pulsar, Cassandra, LangChain, and open-source language models.","archived":false,"fork":false,"pushed_at":"2023-11-03T03:44:07.000Z","size":32,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2023-11-03T23:22:51.657Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datastaxdevs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-10-31T06:51:45.000Z","updated_at":"2023-11-03T23:22:51.658Z","dependencies_parsed_at":null,"dependency_job_id":"c85e758c-c2df-469c-a31e-3138563bd883","html_url":"https://github.com/datastaxdevs/workshop-wikipedia-qa","commit_stats":null,"previous_names":["datastaxdevs/workshop-wikipedia-qa"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastaxdevs%2Fworkshop-wikipedia-qa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastaxdevs%2Fworkshop-wikipedia-qa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastaxdevs%2Fworkshop-wikipedia-qa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastaxdevs%2Fworkshop-wikipedia-qa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datastaxdevs","download_url":"https://codeload.github.com/datastaxdevs/workshop-wikipedia-qa/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248688190,"owners_count":21145762,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-13T08:48:06.871Z","updated_at":"2025-04-13T08:48:07.526Z","avatar_url":"https://github.com/datastaxdevs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# wikipedia_demo\nReal-time document Q\u0026amp;A using Pulsar, Cassandra, LangChain, and open-source language models.\n\nDon't want to complete the exercises?  The complete working code is available on the `complete` branch.\n\n## Project overview\n\nThis workshop code runs a Retrieval Augmented Generation (RAG) application stack that takes data from Wikipedia, stores it in a vector database (Astra DB), and provides a chat interface for asking questions about the Wikipedia documents.\n\nThe project uses Astra Streaming (serverless Apache Pulsar) and Astra DB (serverless Apache Cassandra) and 4 microservices built using:\n\n- Python\n- LangChain for the LLM framework\n- Open source Instructor Embedding model\n- Open source Mistral 7B LLM\n- Gradio for a simple chat web UI\n- Fast API to provide the document embedding service\n  \n## Running the project\n\nThe project consists of 4 microservices\n\n- `docstream` Gets random Wikipedia articles in English and adds them to a Pulsar topic for processing\n- `embeddings` A RESTful API service that turns text into embeddings.\n- `procstream` Consumes articles from the Pulsar topic, scrapes the webpage to get the full text, generates embeddings, and stores in Astra DB\n- `chatbot` Provides both the UI for the chatbot and the agent code for running the chatbot\n\n### With docker\n\n`docker compose up --build`\n\nIndividual services can also be started directly.  Note that `procstream` and `chatbot` require that the `embeddings` microservice is running.  \n\n- `docker compose up --build docstream`\n- `docker compose up --build embeddings`\n- `docker compose up --build procstream`\n- `docker compose up --build chatbot`\n  \n### Without docker\n\nIf you do not wish to run with docker, you can run each of the 4 microservices separately. Use pip to install the requirements for each microservice and then run it directly with python.\n\n```\ncd docstream\npip install -r requirements.txt\npython app.py\n```\n\n```\ncd embeddings\npip install -r requirements.txt\ngunicorn --workers 1 -k uvicorn.workers.UvicornWorker app:app --bind 0.0.0.0:8000\n```\n\n```\ncd procstream\npip install -r requirements.txt\npython app.py\n```\n\n```\ncd chatbot\npip install -r requirements.txt\npython app.py\n```\n\n## Using the services\n\nYou can access the embeddings API in your Chrome browser at http://127.0.0.1:8000/docs.\n\nThe chatbot can be opened in your Chrome browser at http://127.0.0.1:7860. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastaxdevs%2Fworkshop-wikipedia-qa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatastaxdevs%2Fworkshop-wikipedia-qa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastaxdevs%2Fworkshop-wikipedia-qa/lists"}