{"id":21834719,"url":"https://github.com/srjchsv/etl-finetuning","last_synced_at":"2026-05-19T00:01:28.782Z","repository":{"id":205634082,"uuid":"714404140","full_name":"srjchsv/etl-finetuning","owner":"srjchsv","description":"Collecting training data for fine tuning a model and building pipeline on the cloud to train the model.","archived":false,"fork":false,"pushed_at":"2023-11-05T21:05:43.000Z","size":51,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-26T09:43:18.425Z","etag":null,"topics":["appengine","cloud","cloudfunctions","dataengineering","etl","firestore","flask","gcp","lucidchart","pipeline","python","trainingdata"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/srjchsv.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-11-04T19:58:32.000Z","updated_at":"2023-11-05T18:20:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"defc1f3a-7613-4d9e-98ad-94df0eb4a9f2","html_url":"https://github.com/srjchsv/etl-finetuning","commit_stats":{"total_commits":2,"total_committers":1,"mean_commits":2.0,"dds":0.0,"last_synced_commit":"34b1d6082e9fdafb9f6812267838042560bf41d2"},"previous_names":["srjchsv/etl-finetuning"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srjchsv%2Fetl-finetuning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srjchsv%2Fetl-finetuning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srjchsv%2Fetl-finetuning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srjchsv%2Fetl-finetuning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/srjchsv","download_url":"https://codeload.github.com/srjchsv/etl-finetuning/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244811374,"owners_count":20514276,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["appengine","cloud","cloudfunctions","dataengineering","etl","firestore","flask","gcp","lucidchart","pipeline","python","trainingdata"],"created_at":"2024-11-27T20:14:36.706Z","updated_at":"2026-05-19T00:01:23.758Z","avatar_url":"https://github.com/srjchsv.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Fine-tuning LLM model\n\n![alt text](ETL-openai.jpeg \"Title\")\n\n- Create a project or select existing\n- Enable App Engine and Firestore API's\n- Create Firestore database\n- Create bucket for the training data:\n\n`gsutil mb -l us-central1 gs://fine-tuning-training-data`\n\n- Run `pip install -r requirements. txt`\n- Run `gcloud app create`\n- Run `gcloud app deploy`\n- To create a cloud function for extracting the training data and transforming in to .jsonl file, inside the /cloudfunction dir run:\n\n`gcloud functions deploy export_to_storage --runtime python39 --trigger-http --region us-central1 --entry-point export_to_storage --allow-unauthenticated --timeout 540s --memory 128MB`\n\n- Test cloud function with:\n\n`curl -m 550 -X POST https://us-central1-YOUR_PROJECT_ID_HERE.cloudfunctions.net/export_to_storage \\\n-H \"Authorization: bearer $(gcloud auth print-identity-token)\" \\\n-H \"Content-Type: application/json\" \\\n-d '{}'` dont forget to set your project id./\n\n## TODO\n\n- Train the model at OpenAI API with custom data\n- Build the full pipeline on the cloud\n\n\\*maybe some instructions are missing\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrjchsv%2Fetl-finetuning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsrjchsv%2Fetl-finetuning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrjchsv%2Fetl-finetuning/lists"}