{"id":23353398,"url":"https://github.com/gamemann/web-extract-with-chatgpt","last_synced_at":"2025-10-28T22:30:49.086Z","repository":{"id":268650994,"uuid":"905052366","full_name":"gamemann/web-extract-with-chatgpt","owner":"gamemann","description":"A Python project that extracts data from websites with the option to process the data through @openai's ChatGPT API. The results are either printed to stdout or sent via a POST request.","archived":false,"fork":false,"pushed_at":"2025-01-20T12:55:52.000Z","size":2822,"stargazers_count":13,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-13T19:52:18.716Z","etag":null,"topics":["ai","api","beautifulsoup","beautifulsoup4","bs4","chatgpt","chatgpt-api","discourse","extract","gpt","gpt-3","gpt-4","openai","openai-api","post","python","selenium","web-extractor","web-scrape","web-scraper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gamemann.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-18T04:15:26.000Z","updated_at":"2025-02-06T23:04:28.000Z","dependencies_parsed_at":"2024-12-18T05:35:47.973Z","dependency_job_id":null,"html_url":"https://github.com/gamemann/web-extract-with-chatgpt","commit_stats":null,"previous_names":["gamemann/web-extract-with-chatgpt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamemann%2Fweb-extract-with-chatgpt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamemann%2Fweb-extract-with-chatgpt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamemann%2Fweb-extract-with-chatgpt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamemann%2Fweb-extract-with-chatgpt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gamemann","download_url":"https://codeload.github.com/gamemann/web-extract-with-chatgpt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238734087,"owners_count":19521557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","api","beautifulsoup","beautifulsoup4","bs4","chatgpt","chatgpt-api","discourse","extract","gpt","gpt-3","gpt-4","openai","openai-api","post","python","selenium","web-extractor","web-scrape","web-scraper"],"created_at":"2024-12-21T09:13:06.410Z","updated_at":"2025-10-28T22:30:48.638Z","avatar_url":"https://github.com/gamemann.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"A [Python](https://www.python.org/) project that extracts data from websites using [Selenium](https://www.selenium.dev/) and [BeautifulSoup4](https://pypi.org/project/beautifulsoup4/), then either prints the results to `stdout` or sends them via a POST request. Optionally, the extracted web data can be processed through [OpenAI](https://openai.com/)'s **[ChatGPT](https://openai.com/index/chatgpt/) API**, with the API response included in the final output!\n\nThis project was intended for Linux, but may work on Windows after adjustments.\n\n![Preview](./preview/preview.gif)\n\nBefore making this project, I created a private project for my modding [community](https://moddingcommunity.com) that utilized the above technologies which inspired me to make this open source project. I'm hoping the code in this repository helps other developers who want to integrate ChatGPT into their projects or wants to see how to parse and extract data from websites.\n\n## Extractors\nThe URL specified is parsed through *extractors* which are classes intended to extract specific data from the web page. As of right now, there are two extractors; `default` and `discourse_topic`.\n\nThe [`default`](./src/extract/extractors/default.py) extractor returns all *text* from within the `\u003cbody\u003e` tag when the web page is loaded. Keep in mind that this does not wait for data generated by JavaScript to load and only returns what the web server initially renders normally (I'll most likely implement better support for waiting for specific content to load later).\n\nThe last extractor is [`discourse_topic`](./src/extract/extractors/discourse_topic.py) which specifically extracts the *text* contents of a topic from a [Discourse](https://discourse.org) forum. I made this extractor because I used my modding [forum](https://forum.moddingcommunity.com) to test this project.\n\nAdditional extractors may be created for those interested! Feel free to take a look at the source code from [`src/extract`](./src/extract/). You may also be interested in [this](https://github.com/gamemann/How-To-Use-Selenium-And-BeautifulSoup) lab/guide I made on how to extract data from web pages using Selenium and BeautifulSoup4 which includes examples that utilizes JavaScript and waits for specific content to load.\n\n## Requirements\n* An OpenAI key which may be retrieved after setting up billing from [here](https://platform.openai.com/signup).\n* The URL to a web page that supports extracting data from one of the extractors [here](./src/extract/extractors).\n\n### Firefox Geckodriver\nIn order for Selenium to operate, you need the Firefox Geckodriver which may be found [here](https://github.com/mozilla/geckodriver/releases) along with the `firefox-esr` package.\n\n```bash\n# Download Geckodriver 0.35.0 driver.\nwget https://github.com/mozilla/geckodriver/releases/download/v0.35.0/geckodriver-v0.35.0-linux64.tar.gz\n\n# Extract Geckodriver binary.\ntar -xzvf geckodriver-v0.35.0-linux64.tar.gz\n\n# Move binary to /usr/bin.\nsudo mv geckodriver /usr/bin\n\n# Install the Firefox ESR package using apt (for Debian/Ubuntu-based systems).\nsudo apt install -y firefox-esr\n```\n\n### Python\nPython (version 3 or higher) is required to run this project. Additionally, the following Python packages are required.\n\n* Selenium (`4.27.1`)\n* Beautifulsoup4 (`4.12.3`)\n* Requests (`2.32.3`)\n* Jinja2 (`3.1.5`)\n* OpenAI (`1.57.4`)\n\nYou may install these packages using the following command after cloning this repository.\n\n```bash\npip3 install -r requirements.txt\n```\n\n#### Python Virtual Environment\nI recommend running this project within a virtual environment in Python. You can create a virtual environment using the following command assuming you have the required Python packages on your server.\n\n```bash\npython3 -m venv venv/\n```\n\nYou can then activate the virtual environment and install the required packages listed above inside of the environment.\n\n```bash\n# Activate virtual environment.\nsource venv/bin/activate\n\n# Download required packages in new environment.\npip3 install -r requirements.txt\n```\n\n## Command Line Usage \u0026 Running\nThe following command line arguments are supported when running this application.\n\n| Name | Default | Description |\n| ---- | ------- | ----------- |\n| -c, --cfg | `./conf.json` | Path to config file. |\n| -u, --url | `None` | The URL to parse. If not specified, you will need to input after starting the program. |\n| -e, --extractor | `None` | The extractor to use. If not specified, you will need to input after starting the program. |\n| -s, --silent | N/A | When set, verbose output will be disabled. |\n| -l, --list | N/A | Lists contents of the config settings and exits. |\n| -h, --help | N/A | Prints the help menu and exits. |\n\n\u003cdetails\u003e\n    \u003csummary\u003eExample(s)\u003c/summary\u003e\n\n### Run \u0026 Load Custom Config\n```bash\npython3 src/main.py -c /etc/mycustomconf.json\n```\n\n### Run \u0026 List Config Settings\n```bash\npython3 src/main.py -l\n```\n\n### Run \u0026 Print Help Menu\n```bash\npython3 src/main.py --help\n```\n\n### Run \u0026 Load Forum Topic From My Modding Forum\n```bash\npython3 src/main.py -u https://forum.moddingcommunity.com/t/discord-login-integration-added/170 -e discourse_topic\n```\n\u003c/details\u003e\n\n## Configuration\nThe default configuration file is located at `./conf.json`, but can be changed with the command line arguments mentioned above. I recommend copying the [`conf.ex.json`](./conf.ex.json) file to `conf.json` for new users.\n\nHere is a list of configuration settings.\n\n| Name | Type | Default | Description |\n| ---- | ---- | ------- | ----------- |\n| save_to_fs | bool | `false` | Saves config to file system after parsing. This is good for automatic formatting and saving all available settings to config file. |\n| templates_dir | string | `templates` | The path to the [`templates/`](./templates/) directory without the last forward slash. |\n| extract | Extract Object | `{}` | The extract object (read below). |\n| chatgpt | ChatGPT Object | `{}` | The ChatGPT object (read below). |\n| output | Output Object | `{}` | The output object (read below). |\n\n\u003cdetails\u003e\n    \u003csummary\u003eExample(s)\u003c/summary\u003e\n\n### Save To Filesystem\n```json\n{\n    \"save_to_fs\": true,\n    \"extract\": {},\n    \"chatgpt\": {},\n    \"output\": {}\n}\n```\n\u003c/details\u003e\n\n### Extract Object\nThe extract object contains settings related to extracting the web data.\n\n| Name | Type | Default | Description |\n| ---- | ---- | ------- | ----------- |\n| drv_path | string | `/usr/bin/geckodriver` | The path to the Geckodriver binary file. |\n| agents | string array | `[]` | A list of user agents to randomly select from when sending web request. |\n\n\u003cdetails\u003e\n    \u003csummary\u003eExample(s)\u003c/summary\u003e\n\n#### Use Custom User Agents\n```json\n{\n    \"agents\":\n    [\n        \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0\"\n    ]\n}\n```\n\u003c/details\u003e\n\n### ChatGPT Object\nThe ChatGPT object contains settings related to OpenAI and ChatGPT.\n\n| Name | Type | Default | Description |\n| ---- | ---- | ------- | ----------- |\n| enabled | bool | `true` | When enabled, send extracted web data through the ChatGPT API and includes API response in final output. |\n| key | string | `null` | The OpenAI key (required). |\n| model | string | `gpt-3.5-turbo` | The ChatGPT model to use. |\n| max_tokens | int | `500` | The maximum tokens to use with the request. |\n| temperature | float | `0.7` | The temperature to use with the ChatGPT request. Read more about this [here](https://aimresearch.co/leaders-opinion/leaders-opinion-how-temperature-affects-chatgpt-with-rachael-chudoba). |\n| max_input | int | `500` | The maximum characters to send to the ChatGPT API (input). |\n| role_template | string | `chatgpt_role` | The template file name to use with the system's role in ChatGPT from the [`templates/`](./templates/) directory without the file extension (`.tpl`). |\n| prompt_template | string | `chatgpt_prompt` | The template file name to use with the user's prompt in ChatGPT from the [`templates/`](./templates/) directory without the file extension (`.tpl`). |\n\nHere is a list of model code names you can use with the `model` setting.\n\n* *gpt-3.5-turbo*\n* *gpt-3.5-turbo-16k*\n* *gpt-4*\n* *gpt-4-32k*\n* *gpt-4o*\n* *gpt-4o-mini*\n* *o1-mini*\n* *o1*\n* *o1-pro*\n\n**NOTE** - There are multiple models you may use including GPT-4o and GPT-4o-mini. However, I've found the newer models are **very** expensive with the API compared to GPT-3. Therefore, I recommend using the GPT-3 model when you can. For a list of models, check [here](https://platform.openai.com/docs/models/gp)! For information on pricing, check [here](https://openai.com/api/pricing/)!\n\n\u003cdetails\u003e\n    \u003csummary\u003eExample(s)\u003c/summary\u003e\n\n#### Use GPT-4o (EXPENSIVE!)\n```json\n{\n    \"key\": \"CHANGEME\",\n    \"model\": \"gpt-4o\",\n    \"max_tokens\": 500,\n    \"temperature\": 0.5,\n    \"max_input\": 1000\n}\n```\n\u003c/details\u003e\n\n## Output Object\nThe output object is used for sending the ChatGPT response somewhere.\n\n| Name | Type | Default | Description |\n| ---- | ---- | ------- | ----------- |\n| type | string | `stdout` | The type of output to use (`stdout` or `post` are supported right now). |\n| stdout | Stdout Object | `{}` | Settings for the Stdout type. |\n| post | POST Object | `{}` | Settings for the POST type. |\n\nWhen a POST request is sent, the content type is `application/json` and an example of the request body may be found below.\n\n```json\n{\n    \"url\": \"\u003cURL Parsed\u003e\",\n    \"extractor\": \"\u003cExtractor Used\u003e\",\n    \"web_data\": \"\u003cWeb Data Extracted\u003e\",\n    \"chatgpt_res\": \"\u003cChatGPT Response\u003e\"\n}\n```\n\n### Stdout Object\nThe Stdout object contains settings when using the `stdout` type to output the results to the `stdout` pipe.\n\n| Name | Type | Default | Description |\n| ---- | ---- | ------- | ----------- |\n| use_json | bool | `false` | Outputs results in JSON format. |\n| file_path | string | `null` | If set, will write the results to a file at this path. |\n| file_append | bool | `false` | If true, will append the results to the file. |\n\n\u003cdetails\u003e\n    \u003csummary\u003eExample(s)\u003c/summary\u003e\n\n#### Output With JSON \u0026 Append Results To File\n```json\n{\n    \"use_json\": true,\n    \"file_path\": \"./ewc.json\",\n    \"file_append\": true\n}\n```\n\u003c/details\u003e\n\n### POST Object\nThe POST object contains settings when using the `post` type to send results to a web endpoint via a POST request.\n\n| Name | Type | Default | Description |\n| ---- | ---- | ------- | ----------- |\n| url | string | `http://localhost` | The URL to send the POST request to. |\n| headers | Object \u003cstring, string\u003e | `{}` | Any headers to send with the request. |\n\n\u003cdetails\u003e\n    \u003csummary\u003eExample(s)\u003c/summary\u003e\n\n#### Send POST Request With Auth Token\n```json\n{\n    \"type\": \"post\",\n    \"post\":\n    {\n        \"url\": \"https://api.mydomain.com/web-extract\",\n        \"headers\":\n        {\n            \"Authorization\": \"Bearer \u003cMY TOKEN\u003e\"\n        }\n    }\n}\n```\n\u003c/details\u003e\n\n## Notes\n### Template System For ChatGPT Role \u0026 Prompt\nA basic template system is used when formatting what role and prompt to send ChatGPT. Templates may be found in the [`templates/`](./templates/) directory.\n\nThe variables `url` and `content` are passed to the templates, so feel free to use them (e.g. `{{ content }}`).\n\n## Credits\n* [Christian Deacon](https://github.com/gamemann)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgamemann%2Fweb-extract-with-chatgpt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgamemann%2Fweb-extract-with-chatgpt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgamemann%2Fweb-extract-with-chatgpt/lists"}