{"id":13624734,"url":"https://github.com/langchain-ai/langchain-extract","last_synced_at":"2025-05-16T06:06:09.704Z","repository":{"id":225217195,"uuid":"765364980","full_name":"langchain-ai/langchain-extract","owner":"langchain-ai","description":"🦜⛏️ Did you say you like data?","archived":false,"fork":false,"pushed_at":"2024-06-27T13:56:45.000Z","size":1060,"stargazers_count":1123,"open_issues_count":25,"forks_count":120,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-08T15:09:32.436Z","etag":null,"topics":["extraction","extraction-data","fastapi","langchain","langchain-python","llm","llms"],"latest_commit_sha":null,"homepage":"https://extract.langchain.com/","language":"Rich Text Format","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/langchain-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-29T19:25:18.000Z","updated_at":"2025-04-04T21:58:25.000Z","dependencies_parsed_at":"2024-04-26T23:26:31.397Z","dependency_job_id":"1995e3e9-4d63-49f4-8d12-86d37bbb7ebd","html_url":"https://github.com/langchain-ai/langchain-extract","commit_stats":null,"previous_names":["langchain-ai/langchain-extract"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Flangchain-extract","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Flangchain-extract/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Flangchain-extract/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Flangchain-extract/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/langchain-ai","download_url":"https://codeload.github.com/langchain-ai/langchain-extract/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478189,"owners_count":22077676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["extraction","extraction-data","fastapi","langchain","langchain-python","llm","llms"],"created_at":"2024-08-01T21:01:45.717Z","updated_at":"2025-05-16T06:06:04.694Z","avatar_url":"https://github.com/langchain-ai.png","language":"Rich Text Format","funding_links":[],"categories":["Rich Text Format","Repos"],"sub_categories":[],"readme":"🚧 Under Active Development 🚧\n\nThis repo is under active developments. Do not use code from `main`. Instead please checkout code from [releases](https://github.com/langchain-ai/langchain-extract/releases)\n\nThis repository is not a library, but a jumping point for your own application -- so do not be surprised to find breaking changes between releases!\n\nCheckout the demo service deployed at [extract.langchain.com/](https://extract.langchain.com/).\n\n# 🦜⛏️ LangChain Extract\n\nhttps://github.com/langchain-ai/langchain-extract/assets/26529506/6657280e-d05f-4c0f-9c47-07a0ef7c559d\n\n[![CI](https://github.com/langchain-ai/langchain-extract/actions/workflows/ci.yml/badge.svg)](https://github.com/langchain-ai/langchain-extract/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Twitter](https://img.shields.io/twitter/url/https/twitter.com/langchainai.svg?style=social\u0026label=Follow%20%40LangChainAI)](https://twitter.com/langchainai)\n[![](https://dcbadge.vercel.app/api/server/6adMQxSpJS?compact=true\u0026style=flat)](https://discord.gg/6adMQxSpJS)\n[![Open Issues](https://img.shields.io/github/issues-raw/langchain-ai/langchain-extract)](https://github.com/langchain-ai/langchain-extract/issues)\n\n`langchain-extract` is a simple web server that allows you to extract information from text and files using LLMs. It is build using [FastAPI](https://fastapi.tiangolo.com/), [LangChain](https://python.langchain.com/) and [Postgresql](https://www.postgresql.org/).\n\nThe backend closely follows the [extraction use-case documentation](https://python.langchain.com/docs/use_cases/extraction) and provides\na reference implementation of an app that helps to do extraction over data using LLMs.\n\nThis repository is meant to be a starting point for building your own extraction application which\nmay have slightly different requirements or use cases.\n\n## Functionality\n\n- 🚀 FastAPI webserver with a REST API\n- 📚 OpenAPI Documentation\n- 📝 Use [JSON Schema](https://json-schema.org/) to define what to extract\n- 📊 Use examples to improve the quality of extracted results\n- 📦 Create and save extractors and examples in a database\n- 📂 Extract information from text and/or binary files\n- 🦜️🏓 [LangServe](https://github.com/langchain-ai/langserve) endpoint to integrate with LangChain `RemoteRunnnable`\n\n## Releases:\n\n0.0.1: https://github.com/langchain-ai/langchain-extract/releases/tag/0.0.1\n0.0.2: https://github.com/langchain-ai/langchain-extract/releases/tag/0.0.2\n\n## 📚 Documentation\n\nSee the example notebooks in the [documentation](https://github.com/langchain-ai/langchain-extract/tree/main/docs/source/notebooks)\nto see how to create examples to improve extraction results, upload files (e.g., HTML, PDF) and more.\n\nDocumentation and server code are both under development!\n\n## 🍯 Example API\n\nBelow are two sample `curl` requests to demonstrate how to use the API.\n\nThese only provide minimal examples of how to use the API,\nsee the [documentation](https://github.com/langchain-ai/langchain-extract/tree/main/docs/source/notebooks) for more information\nabout the API and the [extraction use-case documentation](https://python.langchain.com/docs/use_cases/extraction) for more information about how to extract\ninformation using LangChain.\n\nFirst we generate a user ID for ourselves. **The application does not properly manage users or include legitimate authentication**. Access to extractors, few-shot examples, and other artifacts is controlled via this ID. Consider it secret.\n\n```sh\nUSER_ID=$(uuidgen)\nexport USER_ID\n```\n\n### Create an extractor\n\n```sh\ncurl -X 'POST' \\\n  'http://localhost:8000/extractors' \\\n  -H 'accept: application/json' \\\n  -H 'Content-Type: application/json' \\\n  -H \"x-key: ${USER_ID}\" \\\n  -d '{\n  \"name\": \"Personal Information\",\n  \"description\": \"Use to extract personal information\",\n  \"schema\": {\n      \"type\": \"object\",\n      \"title\": \"Person\",\n      \"required\": [\n        \"name\",\n        \"age\"\n      ],\n      \"properties\": {\n        \"age\": {\n          \"type\": \"integer\",\n          \"title\": \"Age\"\n        },\n        \"name\": {\n          \"type\": \"string\",\n          \"title\": \"Name\"\n        }\n      }\n    },\n  \"instruction\": \"Use information about the person from the given user input.\"\n}'\n```\n\nResponse:\n\n```json\n{\n  \"uuid\": \"e07f389f-3577-4e94-bd88-6b201d1b10b9\"\n}\n```\n\nUse the extract endpoint to extract information from the text (or a file)\nusing an existing pre-defined extractor.\n\n```sh\ncurl -s -X 'POST' \\\n'http://localhost:8000/extract' \\\n-H 'accept: application/json' \\\n-H 'Content-Type: multipart/form-data' \\\n-H \"x-key: ${USER_ID}\" \\\n-F 'extractor_id=e07f389f-3577-4e94-bd88-6b201d1b10b9' \\\n-F 'text=my name is chester and i am 20 years old. My name is eugene and I am 1 year older than chester.' \\\n-F 'mode=entire_document' \\\n-F 'file=' | jq .\n```\n\nResponse:\n\n```json\n{\n  \"data\": [\n    {\n      \"name\": \"chester\",\n      \"age\": 20\n    },\n    {\n      \"name\": \"eugene\",\n      \"age\": 21\n    }\n  ]\n}\n```\n\nAdd a few shot example:\n\n```sh\ncurl -X POST \"http://localhost:8000/examples\" \\\n    -H \"Content-Type: application/json\" \\\n    -H \"x-key: ${USER_ID}\" \\\n    -d '{\n          \"extractor_id\": \"e07f389f-3577-4e94-bd88-6b201d1b10b9\",\n          \"content\": \"marcos is 10.\",\n          \"output\": [\n            {\n              \"name\": \"MARCOS\",\n              \"age\": 10\n            }\n          ]\n        }' | jq .\n```\n\nThe response will contain a UUID for the example. Examples can be deleted with a DELETE request. This example is now persisted and associated with our extractor, and subsequent extraction runs will incorporate it.\n\n## ✅ Running locally\n\nThe easiest way to get started is to use `docker-compose` to run the server.\n\n**Configure the environment**\n\nAdd `.local.env` file to the root directory with the following content:\n\n```sh\nOPENAI_API_KEY=... # Your OpenAI API key\n```\n\nAdding `FIREWORKS_API_KEY` or `TOGETHER_API_KEY` to this file would enable additional models. You can access available models for the server and other information via a `GET` request to the `configuration` endpoint.\n\nBuild the images:\n\n```sh\ndocker compose build\n```\n\nRun the services:\n\n```sh\ndocker compose up\n```\n\nThis will launch both the extraction server and the postgres instance.\n\nVerify that the server is running:\n\n```sh\ncurl -X 'GET' 'http://localhost:8000/ready'\n```\n\nThis should return `ok`.\n\nThe UI will be available at [http://localhost:3000](http://localhost:3000).\n\n## Contributions\n\nFeel free to develop in this project for your own needs!\nFor now, we are not accepting pull requests, but would love to hear [questions, ideas or issues](https://github.com/langchain-ai/langchain-extract/discussions).\n\n## Development\n\nTo set up for development, you will need to install [Poetry](https://python-poetry.org/).\n\nThe backend code is located in the `backend` directory.\n\n```sh\ncd backend\n```\n\nSet up the environment using poetry:\n\n```sh\npoetry install --with lint,dev,test\n```\n\nRun the following script to create a database and schema:\n\n```sh\npython -m scripts.run_migrations create\n```\n\nFrom `/backend`:\n\n```sh\nOPENAI_API_KEY=[YOUR API KEY] python -m server.main\n```\n\n### Testing\n\nCreate a test database. The test database is used for running tests and is\nseparate from the main database. It will have the same schema as the main\ndatabase.\n\n```sh\npython -m scripts.run_migrations create-test-db\n```\n\nRun the tests\n\n```sh\nmake test\n```\n\n### Linting and format\n\nTesting and formatting is done using a Makefile inside `[root]/backend`\n\n```sh\nmake format\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangchain-ai%2Flangchain-extract","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flangchain-ai%2Flangchain-extract","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangchain-ai%2Flangchain-extract/lists"}