{"id":18602102,"url":"https://github.com/build-on-aws/bedrock-agents-webscraper","last_synced_at":"2025-04-10T19:31:00.502Z","repository":{"id":221624821,"uuid":"753728381","full_name":"build-on-aws/bedrock-agents-webscraper","owner":"build-on-aws","description":"This repo provides guidance on setting up a bedrock agent to webscrape and internet search via action groups","archived":false,"fork":false,"pushed_at":"2025-01-16T00:05:26.000Z","size":39591,"stargazers_count":45,"open_issues_count":2,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-25T03:02:19.665Z","etag":null,"topics":["ai-agent","amazon-bedrock","bedrock-agent","claude","internet-search","webscraper"],"latest_commit_sha":null,"homepage":"https://www.youtube.com/watch?v=kzzlchi0DzU","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit-0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/build-on-aws.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-06T17:24:12.000Z","updated_at":"2025-03-24T08:37:32.000Z","dependencies_parsed_at":"2024-06-25T17:13:36.251Z","dependency_job_id":"2d774341-8feb-4479-8bf1-ca9f9c19e941","html_url":"https://github.com/build-on-aws/bedrock-agents-webscraper","commit_stats":null,"previous_names":["build-on-aws/bedrock-agents-webscraper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/build-on-aws%2Fbedrock-agents-webscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/build-on-aws%2Fbedrock-agents-webscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/build-on-aws%2Fbedrock-agents-webscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/build-on-aws%2Fbedrock-agents-webscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/build-on-aws","download_url":"https://codeload.github.com/build-on-aws/bedrock-agents-webscraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248281395,"owners_count":21077423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agent","amazon-bedrock","bedrock-agent","claude","internet-search","webscraper"],"created_at":"2024-11-07T02:10:19.370Z","updated_at":"2025-04-10T19:30:55.469Z","avatar_url":"https://github.com/build-on-aws.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Setup Amazon Bedrock Agent to Webscrape \u0026 Internet Search Using Natural Language\n\n## Table of Contents\n1. [Introduction](#introduction)\n2. [Prerequisites](#prerequisites)\n3. [Library dependencies](#library-dependencies)\n4. [Diagram](#diagram)\n5. [Grant Model Access](#grant-model-access)\n6. [Deploy Resources via AWS CloudFormation for the Amazon Bedrock Agent](#deploy-resources-via-aws-cloudformation-for-the-amazon-bedrock-agent)\n7. [Step-by-step Setup for the Amazon Bedrock Agent](#step-by-step-setup-for-the-amazon-bedrock-agent)\n   - [Step 1: AWS Lambda Function Configuration](#step-1-aws-lambda-function-configuration)\n   - [Step 2: Create \u0026 Attach an AWS Lambda Layer](#step-2-create--attach-an-aws-lambda-layer)\n   - [Step 3: Setup Bedrock Agent and Action Group](#step-3-setup-bedrock-agent-and-action-group)\n   - [Step 4: Create an Alias](#step-4-create-an-alias)\n8. [Step 5: Testing the Setup](#step-5-testing-the-setup)\n   - [Testing the Bedrock Agent](#testing-the-bedrock-agent)\n9. [Step 6: Setup and Run Streamlit App on EC2 (Optional)](#step-6-setup-and-run-streamlit-app-on-ec2-optional)\n10. [Cleanup](#cleanup)\n11. [Security](#security)\n12. [License](#license)\n\n\n## Introduction\nIn this project, we will set up an Amazon Bedrock agent with two action groups. The first action group will enable the agent to web scrape a specific URL provided by the user, while the second action group allows the agent to perform an internet search based on a user query without requiring a URL.\n\nFor those who prefer an Infrastructure-as-Code (IaC) solution, we provide an AWS CloudFormation template to deploy all the necessary resources, including the Bedrock agent, action groups, and Lambda functions. If you prefer to deploy using AWS CloudFormation, please refer to the guide in the section below.\n\nAlternatively, this README will guide you through the step-by-step process of setting up the Amazon Bedrock agent manually using the AWS Console.\n\n## Prerequisites\n- An active AWS Account.\n- Familiarity with AWS services like Amazon Bedrock, and AWS Lambda.\n- Access will need to be granted to the **Anthropic Claude 3.5 Sonnet** model from the Amazon Bedrock console.\n  \n## Library dependencies\n- [googlesearch-python](https://pypi.org/project/googlesearch-python/)\n- [beautifulsoup4](https://pypi.org/project/beautifulsoup4/)\n\n\n## Diagram\n\n![Diagram](images/bedrock-agent-webscrape-diagram.jpg)\n\n\n### Grant Model Access\n\n- We will need to grant access to the models that will be needed for the Amazon Bedrock agent. Navigate to the Amazon Bedrock console, then on the left of the screen, scroll down and select **Model access**. On the right, select the orange **Enable specific models** button.\n\n![Model access](images/model_access.png)\n\n- To have access to the required models, scroll down and select the checkbox for the **Anthropic: Claude 3.5 Sonnet** model. Then in the bottom right, select **Next**, then **Submit**.\n\n\n- After, verify that the Access status of the Model is green with **Access granted**.\n\n![Access granted](images/access_granted.png)\n\n\n## Deploy resources via AWS Cloudformation for the Amazon Bedrock Agent:\n*Here are the instructions to deploy the resources within your environment:*\n\n***Step 1***\n\nDownload the Cloudformation templates from below, then deploy in order:\n\nClick here to download template 1 🚀 - [1 - Agent-Lambda Stack](https://github.com/build-on-aws/bedrock-agents-webscraper/blob/main/cfn/1-bedrock-agent-lambda-template.yaml) \n- This next template will create an Amazon bedrock agent, action group, with an associated Lambda function.\n\nClick here to download template 2 🚀 - [2 - EC2 UI Stack](https://github.com/build-on-aws/bedrock-agents-webscraper/blob/main/cfn/2-ec2-streamlit-template.yaml)\n- This template will be used to deploy an EC2 instance that will run the code for the Streamlit UI.\n\n***Step 2***\n\n   - In your mangement console, search, then go to the CloudFormation service.\n   - Create a stack with new resources (standard)\n\n   ![Create stack](images/create_stack.png)\n\n   - Prepare template: ***Choose existing template*** -\u003e Specify template: ***Upload a template file*** -\u003e upload the template downloaded from the previous step. \n\n  ![Create stack config](images/create_stack_webscrape.png)\n\n   - Next, Provide a stack name like ***webscrape-agent***, then an alias. After, select Next.\n\n   ![Stack details](images/stack_details.png)\n\n   - On the ***Configure stack options*** screen, leave every setting as default, then go to Next. \n\n   - Scroll down to the capabilities section, and acknowledge the warning message before submitting. \n\n   - Once the stack is complete, follow the same steps to deploy the second template.\n\n![Stack complete](images/stack_complete.png)\n\n\n***Step 3***\n\n### Testing the Bedrock Agent\n\n- Navigate to the Bedrock console. Go to the toggle on the left, and under **Builder tools** select ***Agents***, then the `WebscrapeAgent-{Alias}` that was created.\n\n![navigate to agent](images/navigate_to_agent.png)\n\n\n- In the management console on the right, you have a test user interface. Enter prompts in the user interface to test your Bedrock agent.\n\n![Agent test](images/agent_test.png)\n\n\n- Example prompts for Action Groups:\n\n    1.  Webscrape this url and tell me the main features of pikachu \"https://www.pokemon.com/us/pokedex/pikachu\".\n\n    2. Webscrape this url and tell me the main villians that Goku had to fight on planet earth \"https://en.wikipedia.org/wiki/Goku\".\n\n    3. Do an internet search and tell me the top 3 best traits about lebron james\n\n    4. Do an internet search and tell me how do I know what foods are healthy for me\n\n\n- If you would like to launch the Streamlit app user interface, refer to **Step 6** below to configure the EC2 instance.\n\n\n\n## Step-by-step Setup for the Amazon Bedrock Agent\n\n### Step 1: AWS Lambda Function Configuration\n\n- Navigate to the AWS Lambda management console, and create a Lambda function (Python 3.12) for the Bedrock agent's action group. We will call this Lambda function `bedrock-agent-webscrape`. \n\n![Create Function](images/create_function.png)\n\n![Create Function2](images/create_function_2.png)\n\n \n- Copy the provided code from [here](https://github.com/build-on-aws/bedrock-agents-webscraper/blob/main/function/lambda_webscrape.py), or from below into the Lambda function.\n\n\n```python\nimport urllib.request\nimport os\nimport shutil\nimport json\nimport gzip\nimport io\nfrom bs4 import BeautifulSoup\nimport sys  # Import sys to get the size of the response\n\nMAX_RESPONSE_SIZE = 22000  # 22KB limit\n\ndef get_page_content(url):\n    try:\n        req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})\n        with urllib.request.urlopen(req) as response:\n            # Check if the content is compressed with GZIP\n            if response.info().get('Content-Encoding') == 'gzip':\n                print(f\"Content from {url} is GZIP encoded, decompressing...\")\n                buf = io.BytesIO(response.read())\n                with gzip.GzipFile(fileobj=buf) as f:\n                    content = f.read().decode('utf-8')\n            else:\n                content = response.read().decode('utf-8')\n            \n            if response.geturl() != url:  # Check if there were any redirects\n                print(f\"Redirect detected for {url}\")\n                return None\n\n            return content\n    except Exception as e:\n        print(f\"Error while fetching content from {url}: {e}\")\n        return None\n\ndef empty_tmp_folder():\n    try:\n        for filename in os.listdir('/tmp'):\n            file_path = os.path.join('/tmp', filename)\n            if os.path.isfile(file_path) or os.path.islink(file_path):\n                os.unlink(file_path)\n            elif os.path.isdir(file_path):\n                shutil.rmtree(file_path)\n        print(\"Temporary folder emptied.\")\n        return \"Temporary folder emptied.\"\n    except Exception as e:\n        print(f\"Error while emptying /tmp folder: {e}\")\n        return None\n\ndef save_to_tmp(filename, content):\n    try:\n        if content is not None:\n            print(content)\n            with open(f'/tmp/{filename}', 'w') as file:\n                file.write(content)\n            print(f\"Saved {filename} to /tmp\")\n            return f\"Saved {filename} to /tmp\"\n        else:\n            raise Exception(\"No content to save.\")\n    except Exception as e:\n        print(f\"Error while saving {filename} to /tmp: {e}\")\n        return None\n\ndef check_tmp_for_data(query):\n    try:\n        data = []\n        for filename in os.listdir('/tmp'):\n            if query in filename:\n                with open(f'/tmp/{filename}', 'r') as file:\n                    data.append(file.read())\n        print(f\"Found {len(data)} file(s) in /tmp for query {query}\")\n        return data if data else None\n    except Exception as e:\n        print(f\"Error while checking /tmp for query {query}: {e}\")\n        return None\n\ndef handle_search(event):\n    # Extract inputURL from the requestBody content\n    request_body = event.get('requestBody', {})\n    input_url = ''\n    \n    # Check if the inputURL exists within the properties\n    if 'content' in request_body:\n        properties = request_body['content'].get('application/json', {}).get('properties', [])\n        input_url = next((prop['value'] for prop in properties if prop['name'] == 'inputURL'), '')\n\n    # Handle missing URL\n    if not input_url:\n        return {\"error\": \"No URL provided\"}\n\n    # Ensure URL starts with http or https\n    if not input_url.startswith(('http://', 'https://')):\n        input_url = 'http://' + input_url\n\n    # Check for existing data in /tmp\n    tmp_data = check_tmp_for_data(input_url)\n    if tmp_data:\n        return {\"results\": tmp_data}\n\n    # Clear /tmp folder\n    empty_tmp_result = empty_tmp_folder()\n    if empty_tmp_result is None:\n        return {\"error\": \"Failed to empty /tmp folder\"}\n\n    # Get the page content\n    content = get_page_content(input_url)\n    if content is None:\n        return {\"error\": \"Failed to retrieve content\"}\n\n    # Parse and clean the HTML content\n    cleaned_content = parse_html_content(content)\n\n    # Save the content to /tmp\n    filename = input_url.split('//')[-1].replace('/', '_') + '.txt'\n    save_result = save_to_tmp(filename, cleaned_content)\n\n    if save_result is None:\n        return {\"error\": \"Failed to save to /tmp\"}\n\n    # Check the size of the response and truncate if necessary\n    response_data = {'url': input_url, 'content': cleaned_content}\n    response_size = sys.getsizeof(json.dumps(response_data))\n\n    if response_size \u003e MAX_RESPONSE_SIZE:\n        print(f\"Response size {response_size} exceeds limit. Truncating content...\")\n        truncated_content = cleaned_content[:(MAX_RESPONSE_SIZE - response_size)]\n        response_data['content'] = truncated_content\n\n    return {\"results\": response_data}\n\ndef parse_html_content(html_content):\n    soup = BeautifulSoup(html_content, 'html.parser')\n    for script_or_style in soup([\"script\", \"style\"]):\n        script_or_style.decompose()\n    text = soup.get_text()\n    lines = (line.strip() for line in text.splitlines())\n    chunks = (phrase.strip() for line in lines for phrase in line.split(\"  \"))\n    cleaned_text = '\\n'.join(chunk for chunk in chunks if chunk)\n\n    max_size = 25000\n    if len(cleaned_text) \u003e max_size:\n        cleaned_text = cleaned_text[:max_size]\n\n    return cleaned_text\n\ndef lambda_handler(event, context):\n    response_code = 200\n    action_group = event['actionGroup']\n    api_path = event['apiPath']\n\n    print(\"THE EVENT: \", event)\n\n    if api_path == '/search':\n        result = handle_search(event)\n    else:\n        response_code = 404\n        result = f\"Unrecognized api path: {action_group}::{api_path}\"\n\n    response_body = {\n        'application/json': {\n            'body': result\n        }\n    }\n\n    action_response = {\n        'actionGroup': event['actionGroup'],\n        'apiPath': event['apiPath'],\n        'httpMethod': event['httpMethod'],\n        'httpStatusCode': response_code,\n        'responseBody': response_body\n    }\n\n    api_response = {'messageVersion': '1.0', 'response': action_response}\n    print(\"action_response: \", action_response)\n    print(\"response_body: \", response_body)\n    return api_response\n```\n\n\n- This above code takes the url from the event passed in from the bedrock agent, then uses the **urllib.request** library to call, then scrape the webpage. The **beatifulsoup** library is used to clean up the scraped data. The scraped data is saved to the `/tmp` directory of the Lambda function, then passed into the response back to the agent. Review the code, then **Deploy** the Lambda before moving to the next step.\n\n\n![Lambda deploy](images/lambda_deploy.png)\n\n- Next, apply a resource policy to the Lambda to grant Bedrock agent access. To do this, we will switch the top tab from **code** to **configuration** and the side tab to **Permissions**. Then, scroll to the **Resource-based policy statements** section and click the **Add permissions** button.\n\n![Permissions config](images/permissions_config.png)\n\n![Lambda resource policy create](images/lambda_resource_policy_create.png)\n\n- Here is an example of the resource policy. (At this part of the setup, we will allow any Bedrock agent to access our Lambda, however, as best practice limit access to a specific Bedrock agent Source ARN. So, enter in `arn:aws:bedrock:us-west-2:{YOUR_ACCOUNT_ID}:agent/*`. You can include the ARN once it’s generated in step 4 after creating the Bedrock agent)\n\n\n![Lambda resource policy](images/lambda_resource_policy.png)\n\n\n- Next, we will adjust the configuration on the Lambda so that it has enough time, and CPU to handle the request. Navigate back to the Lambda function screen, go to the Configurations tab, then General configuration and select Edit.\n\n![Lambda config 1](images/lambda_config_1.png)\n\n\n- Update Memory to **4048MB**, Ephemeral storage to **1024MB**, and Timeout to **1 minute**. Leave the other settings as default, then select Save.\n\n![Lambda config 2](images/lambda_config_2.png)\n\n\n- You are now done setting up the webscrape Lambda function. Now, you will need to create another Lambda function following the exact same process for the **internet-search**. Name this Lambda function **bedrock-agent-internet-search**. Copy and paste the python code below into your Lambda function, then select the **Deploy** button:\n\n```python\nimport json\nimport urllib.request\nfrom googlesearch import search\nfrom bs4 import BeautifulSoup\nimport sys  # Import sys to check the size of the response\n\nMAX_RESPONSE_SIZE = 22000  # 22KB limit\n\ndef get_page_content(url):\n    try:\n        req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})\n        with urllib.request.urlopen(req) as response:\n            if response:\n                soup = BeautifulSoup(response.read().decode('utf-8'), 'html.parser')\n                for script_or_style in soup([\"script\", \"style\"]):\n                    script_or_style.decompose()\n                text = soup.get_text()\n                lines = (line.strip() for line in text.splitlines())\n                chunks = (phrase.strip() for line in lines for phrase in line.split(\"  \"))\n                cleaned_text = '\\n'.join(chunk for chunk in chunks if chunk)\n                return cleaned_text\n            else:\n                raise Exception(\"No response from the server.\")\n    except Exception as e:\n        print(f\"Error while fetching and cleaning content from {url}: {e}\")\n        return None\n\ndef search_google(query):\n    try:\n        search_results = []\n        for j in search(query, sleep_interval=5, num_results=10):\n            search_results.append(j)\n        return search_results\n    except Exception as e:\n        print(f\"Error during Google search: {e}\")\n        return []\n\ndef handle_search(event):\n    # Extract the query from the requestBody\n    request_body = event.get('requestBody', {})\n    query = \"\"\n\n    # Check if the query exists within the requestBody\n    if 'content' in request_body:\n        properties = request_body['content'].get('application/json', {}).get('properties', [])\n        query = next((prop['value'] for prop in properties if prop['name'] == 'query'), '')\n\n    # Fallback to 'inputText' if 'query' is not provided\n    if not query:\n        query = event.get('inputText', '')\n\n    print(f\"Performing Google search for query: {query}\")\n    urls_to_scrape = search_google(query)\n\n    aggregated_content = \"\"\n    total_size = 0  # Track the total size of the response\n    truncated = False  # Flag to indicate if the content is truncated\n    search_results = []  # To store the actual content results\n\n    for url in urls_to_scrape:\n        print(\"URLs Used: \", url)\n        content = get_page_content(url)\n        if content:\n            print(\"CONTENT: \", content)\n            content_to_add = f\"URL: {url}\\n\\n{content}\\n\\n{'='*100}\\n\\n\"\n            \n            # Check size before adding more content\n            if total_size + sys.getsizeof(content_to_add) \u003e MAX_RESPONSE_SIZE:\n                print(f\"Response exceeds size limit. Truncating content...\")\n                # Add as much content as possible\n                remaining_size = MAX_RESPONSE_SIZE - total_size\n                truncated_content = content_to_add[:remaining_size]\n                aggregated_content += truncated_content\n                search_results.append({\"content\": truncated_content, \"warning\": \"Content truncated due to size limits\"})\n                truncated = True  # Set the flag to indicate truncation\n                break  # Stop adding content\n\n            aggregated_content += content_to_add\n            total_size = sys.getsizeof(aggregated_content)  # Update the size tracker\n            search_results.append({\"content\": content})\n        else:\n            search_results.append({'url': url, 'error': 'Failed to fetch content'})\n\n    return {\"results\": search_results}\n\ndef lambda_handler(event, context):\n    print(\"THE EVENT: \", event)\n\n    response_code = 200\n    if event.get('apiPath') == '/search':\n        result = handle_search(event)\n    else:\n        response_code = 404\n        result = {\"error\": \"Unrecognized api path\"}\n\n    response_body = {\n        'application/json': {\n            'body': json.dumps(result)\n        }\n    }\n\n    action_response = {\n        'actionGroup': event['actionGroup'],\n        'apiPath': event['apiPath'],\n        'httpMethod': event['httpMethod'],\n        'httpStatusCode': response_code,\n        'responseBody': response_body\n    }\n\n    api_response = {'messageVersion': '1.0', 'response': action_response}\n    print(\"RESPONSE: \", action_response)\n\n    return api_response\n```\n\n\n### Step 2: Create \u0026 attach an AWS Lambda layer\n\n- In order to create this Lambda layer, you will need a .zip file of dependencies for the Lambda function that are not natively provided. We are using the **urllib.request** and **googlesearch(not native)** libraries for internet searching and web scraping. The dependencies are already packaged, and can be download from [here](https://github.com/build-on-aws/bedrock-agents-webscraper/raw/main/lambda-layer/layer-python-requests-googlesearch-beatifulsoup.zip).  \n\n- After, navigate to the AWS Lambda console, then select **layers** from the left-side panel, then create layer.\n  ![lambda layer 1](images/lambda_layer_1.png)\n\n- Name your lambda layer `googlesearch_requests_layer`. Select **Upload a .zip file** and choose the .zip file of dependencies. Choose **x86_64** for your Compatible architectures, and Python 3.12 for your runtime (3.11 version is optional). Your choices should look similar to the example below.\n  \n![lambda layer 2](images/lambda_layer_2.png)\n\n- Navigate back to Lambda function `bedrock-agent-webscrape`, with **Code** tab selected. Scroll to the Layers section and select **Add a Layer**\n\n- Choose the **Custom layers** option from the radio buttons, select the layer you created **googlesearch_requests_layer**, and version 1, then **Add**. Navigate back to your Lambda function, and verify that the layer has been added.\n\n\n![lambda layer 3](images/lambda_layer_3.gif)\n\n\n- You are now done creating and adding the dependencies needed via Lambda layer for your webscrape function. Now, add this same layer to the Lambda function `bedrock-agent-internet-search`, and verify that it has been added successfully.\n\n\n### Step 3: Setup Bedrock Agent and Action Group \n- Navigate to the Bedrock console. Go to the toggle on the left, and under **Builder tools** select ***Agents***, then ***Create Agent***. Provide an agent name, like `athena-agent` then ***Create***.\n\n\n![agent_create](images/agent_create.png)\n\n- For this next screen, agent description is optional. Use the default new service role. For the model, select **Anthropic Claude 3.5 Sonnet**. Next, provide the following instruction for the agent:\n\n\n```instruction\nYou are a research analyst that webscrapes the internet when provided a {question}. You use web searches to find relevant websites and information, or a webscraper to retrieve the content of individual webpages for review. Do not use both options unless explicitly told to do so.You should prefer information from reliable sources, such as the official website, crunchbase, or news organizations. Some companies are startups and may be hard to find, so you should try multiple searches. Some websites will block the webscraper, you should try alternative sources. If you can't determine a relatable response based on the request provided, answer false. Your output should be a JSON document that includes the company name, a yes/no answer, and a summary of your explanation. If your output is an error, you should also respond with a JSON document that includes the error.\n```\n\nIt should look similar to the following: \n\n![agent instruction](images/agent_instruction.png)\n\n\n- Scroll to the top, then select ***Save***.\n\n- Keep in mind that these instructions guide the generative AI application in its role as a research agent that uses specific urls to webscrape the internet. Alternatively, the user has an option to not specify a url, and do a general internet search based on request.\n\n\n- Next, we will add an action group. Scroll down to `Action groups` then select ***Add***.\n\n- Call the action group `webscrape`. In the `Action group type` section, select ***Define with API schemas***. For `Action group invocations`, set to ***Select an existing Lambda function***. For the Lambda function, select `bedrock-agent-webscrape`.\n\n- For the `Action group Schema`, we will choose ***Define with in-line OpenAPI schema editor***. Replace the default schema in the **In-line OpenAPI schema** editor with the schema provided below. You can also retrieve the schema from the repo [here](https://github.com/build-on-aws/bedrock-agents-webscraper/blob/main/schema/webscrape-schema.json). After, select ***Add***.\n`(This API schema is needed so that the bedrock agent knows the format structure and parameters needed for the action group to interact with the Lambda function.)`\n\n```schema\nopenapi: 3.0.0\ninfo:\n  title: Webscrape API\n  version: 1.0.0\n  description: An API that will take in a URL, then scrape the internet to return the results.\npaths:\n  /search:\n    post:\n      summary: Scrape content from the provided URL\n      description: Takes in a URL and scrapes content from it.\n      operationId: scrapeContent\n      requestBody:\n        required: true\n        content:\n          application/json:\n            schema:\n              type: object\n              properties:\n                inputURL:\n                  type: string\n                  description: The URL from which to scrape content\n              required:\n                - inputURL\n      responses:\n        \"200\":\n          description: Successfully scraped content from the URL\n          content:\n            application/json:\n              schema:\n                type: object\n                properties:\n                  scraped_content:\n                    type: string\n                    description: The content scraped from the URL.\n        \"400\":\n          description: Bad request. The input URL is missing or invalid.\n```\n\nYour configuration should look like the following:\n\n\n![ag create gif](images/action_group_creation.gif)\n\n\n- After, hit **Create** and **Save and exit**.\n\n- You are now done setting up the webscrape action group. You will need to create another action group following the exact same process for the ***internet-search***, using the schema below:\n\n```schema\nopenapi: 3.0.0\ninfo:\n  title: Internet Search API\n  version: 1.0.0\n  description: An API that will take in user input, then conduct an internet search that matches the inquiry as close as possible.\npaths:\n  /search:\n    post:\n      summary: Conduct an internet search based on user input\n      description: Takes a user input query, conducts an internet search, and returns the search results.\n      operationId: conductInternetSearch\n      requestBody:\n        description: The search query and additional internet search parameters.\n        required: true\n        content:\n          application/json:\n            schema:\n              type: object\n              properties:\n                query:\n                  type: string\n                  description: The search query text provided by the user.\n                depth:\n                  type: integer\n                  description: The maximum search depth to limit the results.\n              required:\n                - query\n      responses:\n        \"200\":\n          description: Successfully conducted the search and returned results.\n          content:\n            application/json:\n              schema:\n                type: array\n                items:\n                  type: string\n                  description: URLs of search results.\n        \"400\":\n          description: Bad request. The search query is missing or invalid.\n```\n\n\n### Step 4: Create an alias\n- At the top, select **Save**, then **Prepare**. After, select **Save and exit**. Then, scroll down to the **Alias** section and select ***Create***. Choose a name of your liking, then create the alias. Make sure to copy and save your **AliasID**. Also, scroll to the top and save the **Agent ID** located in the **Agent overview** section. You will need this in step 7. Refer to the screenshots below.\n \n ***Alias Agent ID***\n\n![Create alias](images/create_alias.png)\n\n ***Agent ID***\n \n![Agent ARN2](images/agent_arn2.png)\n\n\n\n## Step 5: Testing the Setup\n\n### Testing the Bedrock Agent\n- In the test UI on the right, select **Prepare**. Then, enter prompts in the user interface to test your Bedrock agent.\n\n![Agent test](images/agent_test.png)\n\n\n- Example prompts for **webscrape** action group:\n  ```\n   Webscrape this url and tell me the main features of pikachu \"https://www.pokemon.com/us/pokedex/pikachu\" \n  ```\n  ```\n  Webscrape this url and tell me the main villians that Goku had to fight on planet earth \"https://en.wikipedia.org/wiki/Goku\"\n  ```\n  ```\n  Webscrape this url and tell me about data modeling: https://schema.org/docs/datamodel.html\n  ```\n  ```\n  What is the exchange rate between US Dollars and MXN based on this website? \"https://www.xoom.com/mexico/send-money\"\n  ```\n\n![Agent test 2](images/agent_test_2.png)\n\n\n- Example prompts for **internet search** action group:\n ```\n   Do an internet search and tell me the top 3 best traits about lebron james\n ```\n ```   \n   Do an internet search and tell me how do I know what foods are healthy for me\n ```\n ```\n   Do an internet search and tell me the top 3 strongest features of charizard from pokemon\n ```   \n\n**(If you receive an error saying that the request rate is too high, switch to the Anthropic Claude 3 Haiku model, save \u0026 exit, then prepare before testing again. Also, make sure that you enable access to the Haiku model.)**\n\n![Agent test 3](images/agent_test_3.png)\n\n   (After executing the internet-search function, you can navigate to the CloudWatch logs for this Lambda function thats connected to the action group, and observe the URLs that the data was scraped from with details. You will notice that all URLs will not allow scraping, so the code is designed to error those attempts, and continue with the operation.)\n\n![Lambda logs](images/lambda_logs.png)\n\n\n- **PLEASE NOTE:** when using the **webscraper** and **internet-search** functionality, you could experience some level of hallucincation, inaccuracies, or error if you attempt to ask about information that is very recent, if the prompt is too vague, or if the endpoint cannot be accessed or has a redirect. \n\n   There is also minimal control over which urls are selected during the internet search, except for the # of urls selected from within the google search function parameters. In order to help control this behavior, more engineering will need to be involved. \n\n\n## Step 6: Setup and Run Streamlit App on EC2 (Optional)\n1. **Obtain CF template to launch the streamlit app**: Download the Cloudformation template from [here](https://github.com/build-on-aws/bedrock-agents-streamlit/blob/main/ec2-streamlit-template.yaml). This template will be used to deploy an EC2 instance that has the Streamlit code to run the UI.\n\n\n2. **Deploy template via Cloudformation**:\n   - In your mangement console, search, then go to the AWS CloudFormation service.\n   - Create a stack with new resources (standard).\n\n   ![Create stack](images/create_stack.png)\n\n   - Prepare template: Choose existing template -\u003e Specify template: Upload a template file -\u003e upload the template downloaded from the previous step. \n\n  ![Create stack config](images/create_stack_config.png)\n\n   - Next, Provide a stack name like ***ec2-streamlit***. Keep the instance type on the default of t3.small, then go to Next.\n\n   ![Stack details](images/stack_details.png)\n\n   - On the ***Configure stack options*** screen, leave every setting as default, then go to Next. \n\n   - Scroll down to the capabilities section, and acknowledge the warning message before submitting. \n\n   - Once the stack is complete, go to the next step.\n\n![Stack complete](images/stack_complete.png)\n\n\n3. **Edit the app to update agent IDs**:\n   - Navigate to the EC2 instance management console. Under instances, you should see `EC2-Streamlit-App`. Select the checkbox next to it, then connect to it via `EC2 Instance Connect`.\n\n   ![ec2 connect clip](images/ec2_connect.gif)\n\n   - Next, use the following command  to edit the invoke_agent.py file:\n     ```bash\n     sudo vi app/streamlit_app/invoke_agent.py\n     ```\n\n   - Press ***i*** to go into edit mode. Then, update the ***AGENT ID*** and ***Agent ALIAS ID*** values. \n   \n   ![file_edit](images/file_edit.png)\n   \n   - After, hit `Esc`, then save the file changes with the following command:\n     ```bash\n     :wq!\n     ```   \n\n   - Now, start the streamlit app by running the following command:\n     ```bash\n     streamlit run app/streamlit_app/app.py\n     ```\n  \n   - You should see an external URL. Copy \u0026 paste the URL into a web browser to start the streamlit application.\n\n![External IP](images/external_ip.png)\n\n\n   - Once the app is running, please test some of the sample prompts provided. (On 1st try, if you receive an error, try again.)\n\n![Running App ](images/running_app.png)\n\n\n   - Optionally, you can review the [trace events](https://docs.aws.amazon.com/bedrock/latest/userguide/trace-events.html) in the left toggle of the screen. This data will include the **Preprocessing, Orchestration**, and **PostProcessing** traces.\n\n![Trace events ](images/trace_events.png)\n\n\n\n## Cleanup\nAfter completing the setup and testing of the Bedrock agent, follow these steps to clean up your AWS environment and avoid unnecessary charges:\n\n1. Delete S3 Buckets:\n- Navigate to the S3 console.\n- Select the buckets \"artifacts-bedrock-agent-webscrape-alias\". Make sure that this bucket is empty by deleting the files. \n- Choose 'Delete' and confirm by entering the bucket name.\n\n2.\tRemove the Lambda Functions and Layers:\n- Go to the Lambda console.\n- Select the \"bedrock-agent-internet-search\" function.\n- Click 'Delete' and confirm the action. Do the same for the webscraper function\n- Be sure to navigate to the layers tab in the Lambda console, and delete \"googlesearch_requests_layer\"\n\n3.\tDelete Bedrock Agent:\n- In the Bedrock console, navigate to 'Agents'.\n- Select the created agent, then choose 'Delete'.\n\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis library is licensed under the MIT-0 License. See the LICENSE file.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuild-on-aws%2Fbedrock-agents-webscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbuild-on-aws%2Fbedrock-agents-webscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuild-on-aws%2Fbedrock-agents-webscraper/lists"}