https://github.com/unstructured-io/aws-blog-post-example
Script to accompany the AWS blog post on unstructured data ETL with Unstructured Ingest library
https://github.com/unstructured-io/aws-blog-post-example
Last synced: about 1 year ago
JSON representation
Script to accompany the AWS blog post on unstructured data ETL with Unstructured Ingest library
- Host: GitHub
- URL: https://github.com/unstructured-io/aws-blog-post-example
- Owner: Unstructured-IO
- License: apache-2.0
- Created: 2024-10-16T12:58:12.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-10-16T16:57:32.000Z (over 1 year ago)
- Last Synced: 2025-02-15T20:56:31.097Z (over 1 year ago)
- Language: Python
- Size: 8.79 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# aws-blog-post-example
This repository contains a script to accompany the Unstructured.io blog post in collaboration with AWS.
_Link to the blog post is coming soon._
The blog post illustrates how Unstructured.io's Serverless API can transform unstructured data into a
structured JSON format that can be used by RAG systems on AWS. It provides a step-by-step guide on how
to use the Unstructured API, detailing each stage of the data transformation process including ingestion,
partitioning, extraction, chunking, embedding with Bedrock, and syncing with OpenSearch.
To use this example:
1) Download and install Python version 3.9.0 or later.
2) Clone the repo, and create a virtual environment.
3) In the new virtual environment install the required dependencies:
* Open your terminal in the root directory of the cloned repo.
* Run either `pip install "unstructured-ingest[s3, opensearch, pdf, bedrock]"` to install the latest library versions, or `pip install -r requirements.txt` to use specific versions as defined in the `requirements.txt` file.
4) Open the `run_pipeline.py`, and add your values for the environment variables required to authenticate you with Unstructured Serverless API, Amazon OpenSearch, S3, and Bedrock.
You can now run the script from your terminal by executing:
```bash
python run_pipeline.py
```