Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/craigomatic/webscraper-aiplugin
AI Plugin that can be used to scrape useful information from a given URL
https://github.com/craigomatic/webscraper-aiplugin
Last synced: about 1 month ago
JSON representation
AI Plugin that can be used to scrape useful information from a given URL
- Host: GitHub
- URL: https://github.com/craigomatic/webscraper-aiplugin
- Owner: craigomatic
- License: mit
- Created: 2023-07-26T20:17:29.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-09-02T00:54:34.000Z (over 1 year ago)
- Last Synced: 2024-08-01T13:18:04.360Z (4 months ago)
- Language: C#
- Size: 46.9 KB
- Stars: 10
- Watchers: 4
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-semantickernel - Webscraper AIplugin
README
# Webscraper AIplugin
AI Plugin that can be used to scrape useful information from a given URL.
## Configuration
### Local dev
Open ```local.settings.json``` and add values for each of the following fields:
```CompletionConfig:AIService```
One of AzureOpenAI or OpenAI
```CompletionConfig:Endpoint```
Your Azure endpoint if using Azure OpenAI, or empty if using OpenAI
```CompletionConfig:DeploymentOrModelId```
Your Azure deployment name if using Azure OpenAI, or model name if using OpenAI
```CompletionConfig:Key```
Your API key
### Deploy to Azure
You'll need to have all the same values listed above entered into Configuration -> Application Settings
## Usage
Run it locally with this command:
```func start```
You can test functionality using the swagger UI, which will default to: http://localhost:7071/api/swagger/ui
Note that the first time the function runs, it will attempt to install the Playwright dependencies (chromium, etc) by running a powershell script and that requests to the endpoint will result in 503s while this is happening.
Once you've verified it locally using swagger, the next step is to try running it in the context of _other_ AI.
Some options I like for this are:
1. [Chat Copilot](https://github.com/microsoft/chat-copilot)
2. [sk-researcher](https://github.com/craigomatic/sk-researcher)