{"id":27933448,"url":"https://github.com/timescale/unstructured-pgai-example","last_synced_at":"2025-05-07T04:58:47.898Z","repository":{"id":254023775,"uuid":"845255324","full_name":"timescale/unstructured-pgai-example","owner":"timescale","description":"Example showing unstructured.io + timescaledb + PGAI","archived":false,"fork":false,"pushed_at":"2024-11-15T05:04:38.000Z","size":64,"stargazers_count":14,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-07T04:58:43.098Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timescale.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-20T22:32:26.000Z","updated_at":"2025-04-09T10:44:22.000Z","dependencies_parsed_at":"2024-11-13T19:35:45.946Z","dependency_job_id":null,"html_url":"https://github.com/timescale/unstructured-pgai-example","commit_stats":null,"previous_names":["jonatas/unstructured-timescaledb","timescale/unstructured-pgai-example"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timescale%2Funstructured-pgai-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timescale%2Funstructured-pgai-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timescale%2Funstructured-pgai-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timescale%2Funstructured-pgai-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timescale","download_url":"https://codeload.github.com/timescale/unstructured-pgai-example/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252816948,"owners_count":21808704,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T04:58:47.474Z","updated_at":"2025-05-07T04:58:47.886Z","avatar_url":"https://github.com/timescale.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Unstructured + pgai\n\nThis is a quick start to use Unstructured to ingest any document into PostgreSQL and run semantic search on it.\n\nWith this setup, the document on your postgresql database will be vectorized and you can run semantic search on it.\n\nThe [import.sh](import.sh) script imports the document into PostgreSQL and vectorizes the content of the document.\n\n```mermaid\nsequenceDiagram\n    participant U as User\n    participant S as Unstructured\n    participant T as PostgreSQL\n    participant P as pgai\n    participant O as pgai Vectorizer\n\n    U-\u003e\u003eS: Input raw documents\n    S-\u003e\u003eS: Extract and structure data\n    S-\u003e\u003eT: Store structured data\n    P-\u003e\u003eO: Configure pgai Vectorizer\n    T-\u003e\u003eO: Process text data\n    O-\u003e\u003eO: Generate vector embeddings\n    O-\u003e\u003eT: Store vector embeddings\n    U-\u003e\u003eT: Query data\n    T-\u003e\u003eP: Generate embedding for query data\n    T-\u003e\u003eT: Perform vector similarity search\n    T-\u003e\u003eU: Return relevant results\n    U-\u003e\u003eU: Analyze insights\n```\n\nThe best part of unstructured is that it can ingest a lot of different file types and structure the content of the file in a way that is easy to query.\n\nIt supports csv, html, pdf, word, excel, image, markdown, and more.\n\nThis example covers a generic ingest of any data file and also a python script to ingest a web page.\n\n# Setup\n\nCopy the `.envrc.example` file to `.envrc` and edit it to set the correct environment variables.\n\n```bash\ncp .envrc.example .envrc\n```\n\nOpen `.envrc` and set the correct environment variables. The file includes settings for:\n\n- PostgreSQL connection details (for local setup or Timescale Cloud)\n- OpenAI API configuration\n- Unstructured API settings\n- Other project-specific settings\n\nFor local setup, you can use a pre-built Docker container:\nhttps://github.com/timescale/pgai?tab=readme-ov-file#use-a-pre-built-docker-container\n\nFor production use, we recommend using Timescale Cloud:\nhttps://github.com/timescale/pgai?tab=readme-ov-file#use-a-timescale-cloud-service\nSign up for a free trial and enable the pgai extension for your service.\n\nMake sure to set the following key variables:\n\n- `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASSWORD`: TimescaleDB connection details\n- `OPENAI_API_KEY`: Your OpenAI API key for text processing tasks\n- `UNSTRUCTURED_API_KEY`: API key for the Unstructured service (if using the hosted version)\n\nYou can adapt the openai model or the embedding model to your needs directly in the schema.sql file.\n\nAdjust other settings as needed for your specific use case.\n\nAfter setting up the environment variables, run `install.sh` to install the necessary dependencies.\n\nRun `import.sh` to import the data into TimescaleDB.\n\nExample:\n\n```bash\n./install.sh\n./import.sh import/my/file.pdf\n```\n\n# Database schema\n\nBy default, the database schema is created using the `schema.sql` file.\n\nIt creates a table `elements` that includes all the fields that Unstructured can extract from a document.\n\n\n# Importing data from HTML\n\nYou can use the import.sh script to import data from HTML files too.\n\n```bash\n./import.sh import/my/file.html\n```\n\nIn this case, if you don't have data files to play, I recommend you to use the unstructured project that contains a lot of data files to play with.\n\n```\ngit clone https://github.com/Unstructured-IO/unstructured\n```\n\nThen, you can run the following command to import the data from the HTML files:\n\n```bash\n./import.sh ../unstructured/example-docs/example-10k.html\n```\n\nNow, with a 10k file, it will have an output like this:\n\n```\n2024-08-21 15:05:34,322 MainProcess INFO     Created index with configs: {\"input_path\": \"../../unstructured/example-docs/example-10k.html\", \"recursive\": false}, connection configs: {\"access_config\": \"**********\"}\n2024-08-21 15:05:34,322 MainProcess INFO     Created download with configs: {\"download_dir\": null}, connection configs: {\"access_config\": \"**********\"}\n2024-08-21 15:05:34,322 MainProcess INFO     Created filter with configs: {\"file_glob\": null, \"max_file_size\": null}\n2024-08-21 15:05:34,323 MainProcess INFO     Created partition with configs: {\"strategy\": \"auto\", \"ocr_languages\": null, \"encoding\": null, \"additional_partition_args\": {\"split_pdf_page\": \"true\", \"split_pdf_allow_failed\": \"true\", \"split_pdf_concurrency_level\": 15}, \"skip_infer_table_types\": null, \"fields_include\": [\"element_id\", \"text\", \"type\", \"metadata\", \"embeddings\"], \"flatten_metadata\": false, \"metadata_exclude\": [], \"metadata_include\": [\"id\", \"element_id\", \"text\", \"embeddings\", \"type\", \"system\", \"layout_width\", \"layout_height\", \"points\", \"url\", \"version\", \"date_created\", \"date_modified\", \"date_processed\", \"permissions_data\", \"record_locator\", \"category_depth\", \"parent_id\", \"attached_filename\", \"filetype\", \"last_modified\", \"file_directory\", \"filename\", \"languages\", \"page_number\", \"links\", \"page_name\", \"link_urls\", \"link_texts\", \"sent_from\", \"sent_to\", \"subject\", \"section\", \"header_footer_type\", \"emphasized_text_contents\", \"emphasized_text_tags\", \"text_as_html\", \"regex_metadata\", \"detection_class_prob\"], \"partition_endpoint\": \"https://api.unstructured.io/general/v0/general\", \"partition_by_api\": false, \"api_key\": null, \"hi_res_model_name\": null}\n2024-08-21 15:05:34,323 MainProcess INFO     Created upload_stage with configs: {}\n2024-08-21 15:05:34,323 MainProcess INFO     Created upload with configs: {\"batch_size\": 50}, connection configs: {\"access_config\": \"**********\", \"db_type\": \"postgresql\", \"database\": \"vector_playground\", \"host\": \"localhost\", \"port\": 5432, \"connector_type\": \"sql\"}\n2024-08-21 15:05:34,391 MainProcess INFO     Running local pipline: index (LocalIndexer) -\u003e filter -\u003e download (LocalDownloader) -\u003e filter -\u003e partition (auto) -\u003e upload_stage (SQLUploadStager) -\u003e upload (SQLUploader) with configs: {\"reprocess\": false, \"verbose\": true, \"tqdm\": false, \"work_dir\": \"data\", \"num_processes\": 2, \"max_connections\": null, \"raise_on_error\": false, \"disable_parallelism\": false, \"preserve_downloads\": false, \"download_only\": false, \"max_docs\": null, \"re_download\": false, \"uncompress\": false, \"status\": {}, \"semaphore\": null}\n2024-08-21 15:05:35,819 MainProcess DEBUG    Generated file data: {\"identifier\": \"/Users/jonatasdp/code/unstructured/example-docs/example-10k.html\", \"connector_type\": \"local\", \"source_identifiers\": {\"filename\": \"example-10k.html\", \"fullpath\": \"/Users/jonatasdp/code/unstructured/example-docs/example-10k.html\", \"rel_path\": \"example-10k.html\"}, \"doc_type\": \"file\", \"metadata\": {\"url\": null, \"version\": null, \"record_locator\": {\"path\": \"/Users/jonatasdp/code/unstructured/example-docs/example-10k.html\"}, \"date_created\": \"1724179566.867809\", \"date_modified\": \"1724179566.8738236\", \"date_processed\": \"1724263535.8178291\", \"permissions_data\": [{\"mode\": 33188}], \"filesize_bytes\": 2456707}, \"additional_metadata\": {}, \"reprocess\": false}\n2024-08-21 15:05:35,822 MainProcess INFO     Calling FilterStep with 1 docs\n2024-08-21 15:05:35,822 MainProcess INFO     processing content across processes\n2024-08-21 15:05:35,822 MainProcess INFO     processing content serially\n2024-08-21 15:05:35,824 MainProcess INFO     FilterStep [cls] took 0.002730131149291992 seconds\n2024-08-21 15:05:35,824 MainProcess INFO     Calling DownloadStep with 1 docs\n2024-08-21 15:05:35,825 MainProcess INFO     processing content async\n2024-08-21 15:05:35,826 MainProcess DEBUG    Skipping download, file already exists locally: /Users/jonatasdp/code/unstructured/example-docs/example-10k.html\n2024-08-21 15:05:35,826 MainProcess INFO     DownloadStep [cls] took 0.001313924789428711 seconds\n2024-08-21 15:05:35,826 MainProcess INFO     Calling FilterStep with 1 docs\n2024-08-21 15:05:35,826 MainProcess INFO     processing content across processes\n2024-08-21 15:05:35,826 MainProcess INFO     processing content serially\n2024-08-21 15:05:35,827 MainProcess INFO     FilterStep [cls] took 0.0011758804321289062 seconds\n2024-08-21 15:05:35,827 MainProcess INFO     Calling PartitionStep with 1 docs\n2024-08-21 15:05:35,827 MainProcess INFO     processing content across processes\n2024-08-21 15:05:35,827 MainProcess INFO     processing content serially\n2024-08-21 15:05:35,828 MainProcess DEBUG    Skipping partitioning, output already exists: /Users/jonatasdp/code/timescale/unstructured-timescaledb/data/partition/8323bca93f7c.json\n2024-08-21 15:05:35,829 MainProcess INFO     PartitionStep [cls] took 0.001569986343383789 seconds\n2024-08-21 15:05:35,829 MainProcess INFO     Calling UploadStageStep with 1 docs\n2024-08-21 15:05:35,829 MainProcess INFO     processing content across processes\n2024-08-21 15:05:35,829 MainProcess INFO     processing content serially\n...\n2024-08-21 15:05:35,874 MainProcess INFO     UploadStageStep [cls] took 0.04475522041320801 seconds\n2024-08-21 15:05:35,874 MainProcess INFO     Calling UploadStep with 1 docs\n2024-08-21 15:05:35,882 MainProcess DEBUG    uploading 313 entries to vector_playground\n2024-08-21 15:08:54,376 MainProcess INFO     UploadStep [cls] took 198.50195574760437 seconds\n2024-08-21 15:08:54,377 MainProcess INFO     Finished ingest process in 200.0532009601593s\n```\n\n:warning: The above output is truncated. Also, note that the skip_download and skip_partition options are set to true, so the partition and download steps are skipped as they're already done.\n\nNow, let's take a look at the data filtering by the `type` field:\n\n```sql\nselect languages, type, text from elements where type = 'Title' ;\n languages | type  |                                        text\n-----------+-------+------------------------------------------------------------------------------------\n {eng}     | Title | UNITED STATES\n {eng}     | Title | SECURITIES AND EXCHANGE COMMISSION\n {eng}     | Title | Washington, D.C. 20549\n {eng}     | Title | FORM\n {eng}     | Title | For the transition period from to\n {eng}     | Title | Commission file number:\n {eng}     | Title | ITEM 1. BUSINESS\n {eng}     | Title | ANNUAL REPORT ON FORM 10-K FOR THE YEAR ENDED DECEMBER 31, 2021\n {eng}     | Title | TABLE OF CONTENTS\n {eng}     | Title | SPECIAL NOTE REGARDING FORWARD-LOOKING STATEMENTS\n```\n\n\nYou can also find nearest elements by the embeddings using the `\u003c=\u003e` operator:\n\n```sql\nvector_playground=# select type, text from elements order by embeddings \u003c=\u003e embedding('Tax')  limit 10;\n       type        |   text\n-------------------+--------------------------------------------------------------------------------------------------------\n Title             | FORM\n Title             | BUSINESS\n NarrativeText     | The income tax expense was $48,637 in 2021 based on an effective rate of 2.25 percent compared to the benefit of ($605,936) in 2020 based on an effective rate of 17.42 percent. The 2.25 percent effective tax rate for 2021 differed from the statutory federal income tax rate of 21.0 percent and was primarily attributable to (i) increased tax benefit from the exercise of stock options; (ii) the increased foreign rate differential and (iii) the Company maintaining a valuation allowance against its deferred tax assets.\n UncategorizedText | 11\n Title             | GOVERNMENT REGULATION\n UncategorizedText | 35\n UncategorizedText | 41\n Title             | UNITED STATES\n UncategorizedText | 45\n```\n\nThe `embeddings` field is the embedding of the text column of the element. And, you can use a semantic search to find the nearest elements to a given embedding.\n\n```sql\nvector_playground=# select text from elements where type = 'Title' and embeddings \u003c=\u003e embedding('first day of the year') \u003c 0.2;\n                  text\n----------------------------------------\n YEARS ENDED December 31, 2021 AND 2020\n YEARS ENDED DECEMBER 31, 2021 AND 2020\n YEARS ENDED DECEMBER 31, 2021 AND 2020\n YEARS ENDED DECEMBER 31, 2021 AND 2020\n(4 rows)\n```\n\n# Importing content from web pages\n\nThe other example fetches the TimescaleDB documentation and inserts into the database using the `fetch_timescale_pages.py` script.\n\n```bash\npython fetch_timescale_pages.py\n```\nIt will fetch the documentation from the TimescaleDB website and insert it into the database.\n\nThen, you can run a semantic search using the following SQL:\n\n```sql\nSELECT url, text FROM elements\nWHERE url ~ 'docs.timescale.com'\nORDER BY embeddings \u003c=\u003e embeddings('install timescaledb-ha via docker') LIMIT 1;\n```\n\n# Importing content from PDF files\n\nUnstructured has a lot of data files to play with. If you have a PDF file, you can import it into the database using the following command:\n\n```\n./import.sh ../unstructured/example-docs/embedded-images.pdf\n```\n\nAnd, you can run a semantic search using the following SQL:\n\n```sql\nSELECT text\nFROM elements\nWHERE filename ~ 'embedded-images.pdf'\nORDER BY embeddings \u003c=\u003e embedding('Hands Free') LIMIT 2 ;\n                    text\n---------------------------------------------------------------------------------------------------------\n \"HANDS-FREE\" CARD: use (2/4)\n Hands-free unlocking, when approaching the vehicle; With the card in access zone 3, the ve- hicle will unlock. Unlocking is indicated by one flash of the hazard warning lights and the indicator lights.\n(2 rows)\n```\n\n# Extra resources\n\n- [Unstructured](https://unstructured.io/) is the company behind the Unstructured library.\n- [PGVector](https://github.com/pgvector/pgvector) is the vector extension for postgresql.\n- [pgai](https://github.com/timescale/pgai) is the ai extension for postgresql that can be used to run queries and create the vector embeddings.\n\n# Contributing\n\nThis project is a simple example of how to use Unstructured with PGAI.\n\nWe welcome contributions to improve this project! Here are some ways you can contribute:\n\n1. Report bugs or suggest features by opening an issue.\n2. Submit pull requests to fix bugs or add new features.\n3. Improve documentation or add more examples.\n4. Share your experience using the project with others.\n\nBefore contributing, please read our contributing guidelines [CONTRIBUTING.md](CONTRIBUTING.md) for more information on our development process, coding standards, and how to submit pull requests.\n\nThank you for helping make this project better!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimescale%2Funstructured-pgai-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimescale%2Funstructured-pgai-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimescale%2Funstructured-pgai-example/lists"}