{"id":24499923,"url":"https://github.com/luillyfe/data-pipelines","last_synced_at":"2025-09-02T17:38:39.764Z","repository":{"id":247719531,"uuid":"820369105","full_name":"luillyfe/data-pipelines","owner":"luillyfe","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-20T02:46:37.000Z","size":54,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-21T22:16:19.375Z","etag":null,"topics":["apache-beam","educational-ai","llm"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luillyfe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-26T10:31:20.000Z","updated_at":"2024-07-15T20:40:20.000Z","dependencies_parsed_at":"2024-07-15T01:31:29.954Z","dependency_job_id":null,"html_url":"https://github.com/luillyfe/data-pipelines","commit_stats":null,"previous_names":["luillyfe/data-pipelines"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luillyfe%2Fdata-pipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luillyfe%2Fdata-pipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luillyfe%2Fdata-pipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luillyfe%2Fdata-pipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luillyfe","download_url":"https://codeload.github.com/luillyfe/data-pipelines/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243696706,"owners_count":20332823,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-beam","educational-ai","llm"],"created_at":"2025-01-21T22:16:23.866Z","updated_at":"2025-03-15T07:20:59.396Z","avatar_url":"https://github.com/luillyfe.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM-Powered Contextual Data Augmentation Pipeline\n\nThis project implements a pipeline for contextual data augmentation using Large Language Models (LLMs). It focuses on enhancing free-response questions by converting them into multiple-choice format, leveraging the power of LLMs for intelligent and context-aware transformations. The pipeline utilizes Apache Beam for scalable and parallel data processing.\n\n## Key Features\n\n- LLM-Powered Augmentation: Uses Anthropic's Claude API to intelligently generate multiple-choice options for questions\n- Contextual Understanding: Leverages LLM's ability to understand question context, including exam sections and labels\n- Scalable Processing: Utilizes Apache Beam for efficient, parallel processing of large question sets\n- Data Validation: Ensures input questions meet required schema before processing\n- Firestore Integration: Stores augmented questions for easy retrieval and further use\n\n## How It Works\n\n1. Input Processing: Reads free-response questions from a JSON input file\n2. Data Validation: Validates the structure and content of each question\n3. LLM Augmentation: Sends each question to the Claude API, which generates:\n   - Four contextually relevant multiple-choice options\n   - The correct answer\n   - An explanation (if requested)\n4. Data Transformation: Converts the free-response questions into multiple-choice format\n5. Data Storage: Writes the augmented questions to Firestore for persistence\n\n## Prerequisites\n\n- Go 1.15 or higher\n- Google Cloud Project with Firestore enabled\n- Anthropic API key for access to Claude\n- Apache Beam SDK for Go\n\n## Environment Setup\n\nSet the following environment variables:\n\n- `PROJECT_ID`: Your Google Cloud Project ID\n- `COLLECTION`: Firestore collection name for storing augmented questions\n- `CREDENTIALS_PATH`: Path to your Google Cloud credentials file\n- `CLAUDE_API_KEY`: Your Anthropic API key\n\n## Installation\n\n1. Clone the repository:\n   ```\n   git clone https://github.com/yourusername/llm-data-augmentation-pipeline.git\n   cd llm-data-augmentation-pipeline\n   ```\n\n2. Install dependencies:\n   ```\n   go mod tidy\n   ```\n\n## Usage\n\nRun the augmentation pipeline with:\n\n```\ngo run . --input=path/to/your/questions.json\n```\n\n## Project Structure\n\n- `main.go`: Entry point, sets up and runs the augmentation pipeline\n- `llm_client.go`: Implements the LLM client for question augmentation\n- `firestore_client.go`: Handles writing augmented questions to Firestore\n- `question_parser.go`: Defines data structures and parsing logic for questions\n- `file_utils.go`: Utility functions for file operations\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License:\n\n```\nMIT License\n\nCopyright (c) [2024] [Fermin Blanco]\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluillyfe%2Fdata-pipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluillyfe%2Fdata-pipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluillyfe%2Fdata-pipelines/lists"}