{"id":20196018,"url":"https://github.com/francois-lenne/biomethane","last_synced_at":"2026-05-07T23:41:11.446Z","repository":{"id":232802874,"uuid":"785202990","full_name":"Francois-lenne/biomethane","owner":"Francois-lenne","description":"Data pipeline in order to retrieve the csv with the french production of biomethane upload every day","archived":false,"fork":false,"pushed_at":"2024-04-17T09:29:43.000Z","size":13,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-10T08:56:30.983Z","etag":null,"topics":["cicd","cloudfunction","google-cloud","pubsub","python","snowflake","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Francois-lenne.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-11T12:10:32.000Z","updated_at":"2024-04-18T07:06:09.000Z","dependencies_parsed_at":"2025-01-13T19:35:24.307Z","dependency_job_id":"c3450d21-9b59-48c9-9dea-124b1c85d0ec","html_url":"https://github.com/Francois-lenne/biomethane","commit_stats":null,"previous_names":["francois-lenne/biomethane"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Francois-lenne/biomethane","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Francois-lenne%2Fbiomethane","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Francois-lenne%2Fbiomethane/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Francois-lenne%2Fbiomethane/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Francois-lenne%2Fbiomethane/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Francois-lenne","download_url":"https://codeload.github.com/Francois-lenne/biomethane/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Francois-lenne%2Fbiomethane/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32760953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-07T02:14:30.463Z","status":"ssl_error","status_checked_at":"2026-05-07T02:14:29.405Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cicd","cloudfunction","google-cloud","pubsub","python","snowflake","sql"],"created_at":"2024-11-14T04:21:16.316Z","updated_at":"2026-05-07T23:41:11.419Z","avatar_url":"https://github.com/Francois-lenne.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Purpuse of this project\r\n\r\n\r\nThe purpuse of this project is to apply the knowledge of the snowflake data engineering badge. For this project i will creata a data pipeline in order to retrieve the production of biomethane in france. Indeed everyday, in data.gouv the french governement publish \r\na csv files with the production of biomethane in france by date, productor, are (region). The csv publish everyday contains all the historic + the production of the precedent day. \r\n\r\n\r\n\r\n\r\n# Schema \r\n\r\n\r\n\r\n![schéma biomethane](https://github.com/Francois-lenne/biomethane/assets/114836746/e55e013a-81bd-4cb9-8a93-2f83a76ab776)\r\n\r\n\r\n\r\n## Explanation of the schema \r\n\r\n\r\n\r\nLike we say in the purpuse first we retrieve with the link provide by data.gouv the latest csv files. We use a google cloud function that run a python code. This python code load the csv in google cloud storage. \r\nThis google cloud function is trigger every night at 23h00 by a google cloud scheduler and i also implement a CI/CD between this github repo and the python file. By using Google cloud deploy. Every time a file is load in the bucket a message is send to a topic. Snowflake who is a subscriber so every time a message is send in the topic a pipe is trigger in order to retrieve to integrate the last CSV into the stage table. After that every twelve hours (we can't trigger a snowpipe with an event in april 2024). We merge into the prod table the latest raw. And then we compute the metrics.\r\n\r\n\r\n\r\n# How to implement the project\r\n\r\n\r\nFisrt in order to implement this project you need to have :\r\n\r\n* Snowflake account\r\n* GCP account with a facturation\r\n* Activate the API of pub/sub,GCS, Cloud function, cloud schedueler\r\n* Link you github account with google cloud build\r\n* Having the google cloud SDK install in your local machine or VM\r\n\r\n\r\n\r\n\r\n\r\n## 1) create the bucket \r\n\r\nTwo way to create a bucke : \r\n\r\n* Using CLI\r\n* Using the console\r\n\r\n\r\nImportant you need to create a folder inside you bucket ! \r\n\r\n\r\n\r\n\r\n\r\n## 2) develop the google cloud fucntions \r\n\r\nDevelop the python program with a main function with one paramater (requets) the python program need to be name main.py (you can name the python program whatever you want but you need to modify the gcloud code).\r\n\r\nthen deploy it using this commandands in the local repository\r\n\r\n``` Powershell\r\ngcloud functions deploy \"name_function\" --runtime python39 --trigger-http --allow-unauthenticated --set-env-vars GOOGLE_CLOUD_PROJECT= [name_secret] # we set up a environnement secret for this project       \r\n```\r\n\r\n\r\n\r\n``` Powershell\r\n\r\ngcloud scheduler jobs create http \"name_scheduler\" \\\r\n\u003e\u003e --schedule \"0 23 * * *\" \\                                                                                                                     \r\n\u003e\u003e --http-method GET \\\r\n\u003e\u003e --uri url_function \\\r\n\u003e\u003e --oidc-service-account-email \"service-account-email\" \\\r\n```\r\n\r\n\r\n\r\n\r\n## 4) create the integration of the GCS bucket in stage\r\n\r\n\r\nYou need to follow the documentation in this link : \r\n\r\nhttps://docs.snowflake.com/en/user-guide/data-load-gcs-config \r\n\r\n\r\n\r\n\r\n## 5) create the pub/sub link between GCS and snowflake \r\n\r\n\r\nfirst you need to follow the documentation here :\r\n\r\nhttps://docs.snowflake.com/en/user-guide/data-load-snowpipe-auto-gcs\r\n\r\nbut the documentation have two problems first you need to gave to the iam account of snowflake the monitor view right and the seconds is they use a quote between the name but that triger an error as you can see in my stack overflow blog :\r\n\r\nhttps://stackoverflow.com/questions/78327105/troubleshoot-to-create-a-notification-integrations-between-gcs-and-snowflake?noredirect=1#comment138092331_78327105\r\n\r\n\r\n\r\n\r\n## 6) create CI/CD for the python code\r\n\r\nThis video explain well how to do that :\r\n\r\nhttps://www.youtube.com/watch?v=TeguiTFprFo\u0026pp=ygUhQ0kvQ0QgZ2l0aHViIGNsb3VkIGZuY3Rpb24gcHl0aG9u\r\n\r\ndon't forget to add a .cloudingore for ignoring all the file that isn't useful (copy my file) \r\n\r\n\r\n\r\n\r\n## 7) Create CI/CD between snowflake and the github repo \r\n\r\nAgain follow the new snowflake documentation :\r\n\r\nhttps://docs.snowflake.com/en/developer-guide/git/git-overview\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrancois-lenne%2Fbiomethane","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrancois-lenne%2Fbiomethane","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrancois-lenne%2Fbiomethane/lists"}