{"id":41879478,"url":"https://github.com/leadtechie/bambooconnect","last_synced_at":"2026-01-25T13:00:42.200Z","repository":{"id":47404860,"uuid":"505582032","full_name":"LeadTechie/BambooConnect","owner":"LeadTechie","description":"Bamboo Connect is a lightweight ETL (Extract, Transform, Load) library with examples and templates. It enables developers to quickly extract, transform, reconcile and then load resulting data securely. This avoids time consuming manual error prone tasks.","archived":false,"fork":false,"pushed_at":"2023-12-22T11:45:02.000Z","size":9742,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-01-02T21:57:38.699Z","etag":null,"topics":["etl","etl-framework","etl-pipeline","pandas","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LeadTechie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"support/__init__.py","governance":null,"roadmap":null,"authors":null}},"created_at":"2022-06-20T20:03:55.000Z","updated_at":"2023-02-21T06:41:41.000Z","dependencies_parsed_at":"2023-12-22T12:40:36.943Z","dependency_job_id":"dbd646cf-1f6c-4c3d-b6a4-d5eef3d349a1","html_url":"https://github.com/LeadTechie/BambooConnect","commit_stats":{"total_commits":125,"total_committers":2,"mean_commits":62.5,"dds":0.06399999999999995,"last_synced_commit":"cc07014da83428b4744ac68c81e9c76f6c529532"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/LeadTechie/BambooConnect","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeadTechie%2FBambooConnect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeadTechie%2FBambooConnect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeadTechie%2FBambooConnect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeadTechie%2FBambooConnect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LeadTechie","download_url":"https://codeload.github.com/LeadTechie/BambooConnect/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeadTechie%2FBambooConnect/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28753411,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-25T10:25:12.305Z","status":"ssl_error","status_checked_at":"2026-01-25T10:25:11.933Z","response_time":113,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","etl-framework","etl-pipeline","pandas","python"],"created_at":"2026-01-25T13:00:32.916Z","updated_at":"2026-01-25T13:00:42.192Z","avatar_url":"https://github.com/LeadTechie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Note: This prooject is no longer supported\n\nTo quote Michael Nygard from Craft Conference, 2023 talk: https://youtu.be/55qSt0PsU_0?si=VF5jJc3ycESM7xDA\n```\nBuilding an abstraction layer is like buying an option:\n* Those options are not often exercised\n* It’s likely that whatever your abstraction layer is it will be a poor representation of what’s underneath it and it won’t be general enough to support the next new thing you want to swap in.\n```\n-\u003e Therefore think very carefully about building one at start\n\nShould have watched this first, but it was fun building it out as PoC!\n\n\n\n# Bamboo Connect Quick Start\n\n```\nconda create -n python3-9-12 python=3.9.12 anaconda  \nconda activate python3-9-12  \n```\n\n## Setup Environment Variables\n\ndev environment:\n\n```\nexport JIRA_TOKEN=\u003ctoken\u003e\nexport JIRA_EMAIL=\u003cemail\u003e\nexport CREDENTIALS_JSON=\u003cemail\u003e\nexport BAMBOO_CACHE_PATH=\u003cemail\u003e\n\necho \"$JIRA_TOKEN\"\necho \"$JIRA_EMAIL\"\necho \"$CREDENTIALS_JSON\"\necho \"$BAMBOO_CACHE_PATH\"\n```\n\nGenerating Tokens:\n- JIRA_TOKEN https://id.atlassian.com/manage-profile/security/api-tokens.\n- Google CREDENTIALS_JSON: https://github.com/LeadTechie/BambooConnect/blob/v3/readme/README.md\n\n```\npip install notebook\npip install jupytext\n```\n\n## Run Tests\n```\npython -m unittest discover -s test/end2end/ -p 'test_e2e*.py'\npython -m unittest discover -s test/integration/ -p 'test_integration*.py'\npython -m unittest discover -s test/system/ -p 'test_system*.py'\npython -m unittest discover -s test/unit/ -p 'test_unit*.py'\npython -m unittest discover -p 'test_*.py' -t .\n\n```\n\n## Run Against Local BambooConnect Code\n```\npip install -e ./BambooConnect\n```\n## Example Interactive Editing with Jupyter Notebooks\n```\n\npip install jupytext\npip install jupyter\n\njupytext --to notebook extractors/google_drive_file_extractor.py\njupytext --set-formats ipynb,py extractors/google_drive_file_extractor.ipynb\njupytext --sync extractors/google_drive_file_extractor.ipynb\n\njupyter notebook\n```\n\n## Example Interactive Editing with Jupyter Notebooks\n```\npython jupyter_start.py\n```\nSelect . for current subdirectory\nThen enter eg e2e to see just files with e2e\njupyter notebook will then be started\n\n# Bamboo Connect\n\nHow much time do you waste manually keeping track of data from multiple systems. Different systems, different formats. Is your list up to date? How to match users or ids across multiple systems?\n\n**Bamboo Connect is a lightweight ETL (Extract, Transform, Load) library with examples and templates. It enables developers to quickly extract, transform, reconcile and then load resulting data securely. This avoids time consuming manual error prone tasks.**\n\nIf you’re low volume (\u003c10k records), low frequency (max hourly), already have GitHub and Google Sheets available to you and have development skills then Bamboo Connect is for you.\n\nExample use cases:\n- How to link component documentation from the JIRA with an extended Google sheets list\n- How can you tell who has a JIRA account but it not yet in GitHub\n- How to keep multiple team lists with so many changes and new starters across different systems\n\n\n### Overview\n![BambooConnect-Intro-Architecture-Overview.jpg](readme/BambooConnect-Intro-Architecture-Overview.jpg?raw=true)\n\n![BambooConnect-Intro-Architecture-Overview1.jpg](readme/BambooConnect-Intro-Architecture-Overview1.jpg?raw=true)\n\n### Quick View\n\n![BambooConnect-Intro-Architecture-Overview2.jpg](readme/BambooConnect-Intro-Architecture-Overview2.jpg?raw=true)\n\n\n### Recon Tools High Level Architecture And Flow\n![ReconToolsArchitectureDiagramHighLevelArchitectureAndFlow.jpg](readme/ReconToolsArchitectureDiagramHighLevelArchitectureAndFlow.jpg?raw=true)\n\n### Recon Tools Hosting / Production Setup\n![BambooConnect-Intro-Architecture-Overview3.jpg](readme/BambooConnect-Intro-Architecture-Overview3.jpg?raw=true)\n\n\n### Recon Tools Test Approach\n![ReconToolsArchitectureDiagramTestApproach.jpg](readme/ReconToolsArchitectureDiagramTestApproach.jpg?raw=true)\n\n### Further Details\n- See the sub pages for [details on setting up the credentials and access tokens for Google and JIRA](readme/README.md)\n\n### Python Environment Manager - Install Conda:  \n- Install - [https://docs.conda.io/en/latest/miniconda.html](https://docs.conda.io/en/latest/miniconda.html)  \n[https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html)  \n- Update [https://www.geeksforgeeks.org/set-up-virtual-environment-for-python-using-anaconda/](https://www.geeksforgeeks.org/set-up-virtual-environment-for-python-using-anaconda/)  \nSet version 3.9.12  \n\n```\nconda create -n python3-9-12 python=3.9.12 anaconda  \nconda activate python3-9-12  \n```\n\n### Install Packages\n```\npip install -r ./requirements.txt  \n```\n\n### Set Environment Variables for login\nPlace your Google credentials.json file in the directory below the project directory then run\nSee here for [how to create your credentials.json file](readme/credentials/README.md)\n```\npython authentication_support.py\n```\nThis will create the base64 encoded string you need for the CREDENTIALS_JSON\n\nGenerate your JIRA Token https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/\n\n```\nexport RECON_TOOLS_JIRA_EMAIL=\"\u003cYour value here\u003e\"\nexport RECON_TOOLS_JIRA_TOKEN=\"\u003cYour value here\u003e\"\"\nexport CREDENTIALS_JSON=\"\u003cYour value here base64 encoded json file\u003e\"  \n```\n\n### Check Environment Variables for login\n```\necho \"$RECON_TOOLS_JIRA_TOKEN\"  \necho \"$RECON_TOOLS_JIRA_EMAIL\"  \necho \"$CREDENTIALS_JSON\"\n```\n\n### Test Setup\n\nTesting is setup at 4 levels:\n1. Unit Tests: All tests and test data is in test files\n2. System Tests: Uses test files in subdirectory /test_data/\n3. Integration Tests: Links to 3rd party systems but relies on minimumd data in these systems so you can run these tests\n4. end2end Tests: Integration tests relying on specfic sestup or external systems (JIRA \u0026 Sheets) so won't work for you unless you get access to my projects or use test files to recreate base data\n\n### Run local tests to check working\n```\npython -m unittest discover -s test/unit -p 'test_*.py'\npython -m unittest discover -s test/system -p 'test_*.py'\n```\n\n### Run an example of a component reconciliation using this script\n\nTODO: Currently this requires access to a test JIRA account\n```\npython poc_test.py\n```\n\n### Py Recon Tools Docs\n\nSee this [Google Presentations](https://docs.google.com/presentation/d/1nKeGEwgP3xvYbnmz0WEcTWl8kNGfS48Pi-6drKdufVo/edit#slide=id.gf47d2de6cc_0_43) for latest version of these docs\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleadtechie%2Fbambooconnect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleadtechie%2Fbambooconnect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleadtechie%2Fbambooconnect/lists"}