{"id":18231473,"url":"https://github.com/velocitybolt/open-extract","last_synced_at":"2025-04-03T16:30:36.576Z","repository":{"id":247998243,"uuid":"827465609","full_name":"velocitybolt/open-extract","owner":"velocitybolt","description":"Structured Data Extractor for AI Agents. Search your documents or the web for specific data and get it back in JSON or Markdown in a single tool call.","archived":false,"fork":false,"pushed_at":"2025-03-29T13:40:07.000Z","size":9347,"stargazers_count":162,"open_issues_count":0,"forks_count":13,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-29T14:30:49.242Z","etag":null,"topics":["agent-tools","ai","autogen","context-aware","context-aware-structured-outputs","crewai","etl","etl-automation","etl-framework","langchain","langgraph","llm","openai","python","rag","structured-outputs","unstructured-data"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/velocitybolt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-11T17:41:20.000Z","updated_at":"2025-03-29T13:40:11.000Z","dependencies_parsed_at":"2024-10-31T01:23:02.011Z","dependency_job_id":"93036c6c-6785-406f-b588-ba28b170af8e","html_url":"https://github.com/velocitybolt/open-extract","commit_stats":null,"previous_names":["velocitybolt/marly","marly-ai/marly","velocitybolt/open-extract"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/velocitybolt%2Fopen-extract","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/velocitybolt%2Fopen-extract/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/velocitybolt%2Fopen-extract/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/velocitybolt%2Fopen-extract/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/velocitybolt","download_url":"https://codeload.github.com/velocitybolt/open-extract/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247036807,"owners_count":20873018,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-tools","ai","autogen","context-aware","context-aware-structured-outputs","crewai","etl","etl-automation","etl-framework","langchain","langgraph","llm","openai","python","rag","structured-outputs","unstructured-data"],"created_at":"2024-11-04T12:02:24.569Z","updated_at":"2025-04-03T16:30:36.561Z","avatar_url":"https://github.com/velocitybolt.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# open-extract\n\n[Features](#-features) • [What is a Schema?](#-what-is-a-schema) • [Use Cases](#-use-cases) • [Getting Started](#-getting-started) • [Documentation](#-documentation)\n\n\u003c/div\u003e\n\n---\n\nopen-extract simplifies the ingestion and processing of unstructured data for those building AI Agents/Agentic Workflows using frameworks such as LangGraph, AG2, and CrewAI.\n\n---\n\n## 🚀 Features\n\n📄 Extract Relevant Information Seamlessly: Give your applications the ability to identify and extract relevant data from one or many large documents and websites with just a single API call. Get the content back in JSON or Markdown formats, making it easy to integrate into your workflows.\n\n🔍 Multi-Schema/Multi-Document Support: Extract data based one or many predefined schemas from a variety of document types, without needing a vector database or specifying page numbers.\n\n🔄 Built-in Caching: With built-in caching, previously extracted schemas can be instantly retrieved, enabling rapid repeat extractions without having to reprocess the original documents.\n\n🚫 No Vendor Lock-In: Enjoy complete flexibility with your choice of model provider. Whether using open-source or closed-source models, you're never tied to a specific vendor, ensuring full control.\n\n---\n\n## 🧰 What is a Schema?\n\nA schema is a set of key-value pairs describing what needs to be extracted from a particular document.\n\n\u003cdetails\u003e\n\u003csummary\u003e📋 Example Schema\u003c/summary\u003e\n\n```\n{\n    \"Firm\": \"The name of the firm\",\n    \"Number of Funds\": \"The number of funds managed by the firm\",\n    \"Commitment\": \"The commitment amount in millions of dollars\",\n    \"% of Total Comm\": \"The percentage of total commitment\",\n    \"Exposure (FMV + Unfunded)\": \"The exposure including fair market value and unfunded commitments in millions of dollars\",\n    \"% of Total Exposure\": \"The percentage of total exposure\",\n    \"TVPI\": \"Total Value to Paid-In multiple\",\n    \"Net IRR\": \"Net Internal Rate of Return as a percentage\"\n}\n```\n\n\u003c/details\u003e\n\n\u003c/details\u003e\n\n---\n\n## 🎯 Use Cases\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003e💼 Financial Report Analysis\u003c/b\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003e📊 Customer Feedback Processing\u003c/b\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003e🔬 Research Assistant\u003c/b\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003e🧠 Legal Contract Parsing\u003c/b\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eExtract key financial metrics from quarterly PDF reports\u003c/td\u003e\n    \u003ctd\u003eCategorize feedback from various document types\u003c/td\u003e\n    \u003ctd\u003eProcess research papers, extracting methodologies and findings\u003c/td\u003e\n    \u003ctd\u003eExtract key legal terms and conditions from contracts\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n---\n\n## 🛠️ Getting Started\n\n### Build the Platform\n\n---\n\nTo build the platform from source, run the following command:\n\n```bash\n./start-oe.sh\n```\n\n---\n\n### Run an example script or notebook\n\nOnce the platform is running you can test it out by trying one of our examples\n\n1. Navigate to the examples folder:\n\n   ```bash\n   cd examples\n   ```\n2. Navigate to the scripts or notebooks folder:\n\n   ```bash\n   cd scripts\n   ```\n   or\n   ```bash\n   cd notebooks/autogen_example\n   ```\n3. Run one of our example scripts:\n   ```bash\n   python azure_example.py\n   ```\n\n---\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvelocitybolt%2Fopen-extract","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvelocitybolt%2Fopen-extract","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvelocitybolt%2Fopen-extract/lists"}