{"id":19738109,"url":"https://github.com/linuxuser255/aiwebscraper","last_synced_at":"2026-02-07T18:31:00.974Z","repository":{"id":257704405,"uuid":"850184339","full_name":"LinuxUser255/AIWebScraper","owner":"LinuxUser255","description":"This webscraper uses the Ollama LLM to assist in web scraping. It prompts the user to supply a URL, and a description of they want scraped from the specified website. These instructions are then read by the LLM and parsed by the python code, and the results are returned to the user.","archived":false,"fork":false,"pushed_at":"2025-01-25T02:06:08.000Z","size":35,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-25T03:17:22.419Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LinuxUser255.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-31T04:50:18.000Z","updated_at":"2025-01-25T02:06:11.000Z","dependencies_parsed_at":"2025-01-16T23:16:15.955Z","dependency_job_id":null,"html_url":"https://github.com/LinuxUser255/AIWebScraper","commit_stats":null,"previous_names":["linuxuser255/aiwebscraper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LinuxUser255%2FAIWebScraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LinuxUser255%2FAIWebScraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LinuxUser255%2FAIWebScraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LinuxUser255%2FAIWebScraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LinuxUser255","download_url":"https://codeload.github.com/LinuxUser255/AIWebScraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241067688,"owners_count":19903864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T01:13:17.818Z","updated_at":"2026-02-07T18:31:00.924Z","avatar_url":"https://github.com/LinuxUser255.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI Assited Web Scraper\n\n### Built with [Streamlit](https://streamlit.io/)  \u0026 Powered by [Ollama](https://ollama.com/)\n\n\u003cimg width=\"452\" alt=\"Screenshot 2025-01-13 at 5 18 23 PM\" src=\"https://github.com/user-attachments/assets/9509b333-d71f-4e77-b222-c7b3297f5b55\" /\u003e\n\n\n![streamlit-mail-logo](https://github.com/user-attachments/assets/2bb87c8e-11e8-4236-bf85-162b099e7189)\n\n\n\n\n\nABOUT:\n\nThis webscraper uses the Ollama LLM to assist in web scraping.\nIt prompts the user to supply a URL, and a description of they want scraped from the specified website.\nThese instructions are then read by the LLM and parsed by the python code, and the results are returned to the user.\n\n## First:\n\n## You will need to download the Chrome Driver \n## and palce it in the same directory as the python files/scripts\n## --\u003e https://googlechromelabs.github.io/chrome-for-testing/#stable\n\n\u003cbr\u003e\n\n\n## Installation and use:\n**Clone the repository \u0026 Install the requirments**\n```shell\npip3 install -r requirements.txt\n```\n\n**Then, in the command line, in the `/AIWebScraper` directory run:**\n```shell\nstreamlit run main.py\n```\n\n\u003cbr\u003e\n\n# Quickstart\n\n### To run and chat with [Llama 3.1](https://ollama.com/library/llama3.1):\n\n## _! However, `deepseek-coder-v2` is very impressive_\n\n## And, You can select it by running the included install script\n\n## * See the Shell Script instructions below.\n\n\u003cbr\u003e\n\n**To pull a model do:**\n```shell\nollama pull llama3.1\n```\n\n\u003cbr\u003e\n\n\n## - Then run it interactively on the command line.\n```shell\nollama run llama3.1\n```\n\u003cbr\u003e\n\n## - _You can also use this shell script to automate selecting and pulling Ollama Models_\n```shell\nchmod +x install_ollama.sh\n./install_ollama.sh\n```\n\n\u003cbr\u003e\n\n### - _Before pulling a model number, keep in mind the memory constraints and limitations of your machine. Check the table below_\n\n\u003cbr\u003e\n\n### - _The following information is from [Ollama's GitHub](https://github.com/ollama/ollama) and is relevant to it's use in this web scraper_\n\n\u003cbr\u003e\n\n## - Model library\n\n**- Ollama supports a list of models available on [ollama.com/library](https://ollama.com/library 'ollama model library')**\n\n**- Here are some example models that can be downloaded:**\n\n| Model              | Parameters | Size  | Download                       |\n| ------------------ | ---------- | ----- | ------------------------------ |\n| Llama 3.1          | 8B         | 4.7GB | `ollama run llama3.1`          |\n| Llama 3.1          | 70B        | 40GB  | `ollama run llama3.1:70b`      |\n| Llama 3.1          | 405B       | 231GB | `ollama run llama3.1:405b`     |\n| Phi 3 Mini         | 3.8B       | 2.3GB | `ollama run phi3`              |\n| Phi 3 Medium       | 14B        | 7.9GB | `ollama run phi3:medium`       |\n| Gemma 2            | 2B         | 1.6GB | `ollama run gemma2:2b`         |\n| Gemma 2            | 9B         | 5.5GB | `ollama run gemma2`            |\n| Gemma 2            | 27B        | 16GB  | `ollama run gemma2:27b`        |\n| Mistral            | 7B         | 4.1GB | `ollama run mistral`           |\n| Moondream 2        | 1.4B       | 829MB | `ollama run moondream`         |\n| Neural Chat        | 7B         | 4.1GB | `ollama run neural-chat`       |\n| Starling           | 7B         | 4.1GB | `ollama run starling-lm`       |\n| Code Llama         | 7B         | 3.8GB | `ollama run codellama`         |\n| Llama 2 Uncensored | 7B         | 3.8GB | `ollama run llama2-uncensored` |\n| LLaVA              | 7B         | 4.5GB | `ollama run llava`             |\n| Solar              | 10.7B      | 6.1GB | `ollama run solar`             |\n\n\u003e [!NOTE]\n\u003e You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models.\n\n## Customize a model\n\n### Import from GGUF\n\nOllama supports importing GGUF models in the Modelfile:\n\n1. Create a file named `Modelfile`, with a `FROM` instruction with the local filepath to the model you want to import.\n\n   ```\n   FROM ./vicuna-33b.Q4_0.gguf\n   ```\n\n2. Create the model in Ollama\n\n   ```\n   ollama create example -f Modelfile\n   ```\n\n3. Run the model\n\n   ```\n   ollama run example\n   ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinuxuser255%2Faiwebscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinuxuser255%2Faiwebscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinuxuser255%2Faiwebscraper/lists"}