{"id":20544411,"url":"https://github.com/toddbirchard/jira-database-etl","last_synced_at":"2025-04-14T09:52:34.597Z","repository":{"id":54582902,"uuid":"175013883","full_name":"toddbirchard/jira-database-etl","owner":"toddbirchard","description":":mens: 💾 Script to import issues from a JIRA instance into a database.","archived":false,"fork":false,"pushed_at":"2022-12-08T10:53:54.000Z","size":1031,"stargazers_count":56,"open_issues_count":20,"forks_count":20,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-27T23:11:20.731Z","etag":null,"topics":["etl","etl-pipeline","flask","flask-sqlalchemy","jira-rest-api","pandas","python3"],"latest_commit_sha":null,"homepage":"https://hackersandslackers.com/jira-to-sql-etl/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/toddbirchard.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-11T14:09:26.000Z","updated_at":"2024-12-18T16:42:55.000Z","dependencies_parsed_at":"2022-08-13T20:30:56.765Z","dependency_job_id":null,"html_url":"https://github.com/toddbirchard/jira-database-etl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toddbirchard%2Fjira-database-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toddbirchard%2Fjira-database-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toddbirchard%2Fjira-database-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toddbirchard%2Fjira-database-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/toddbirchard","download_url":"https://codeload.github.com/toddbirchard/jira-database-etl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248860016,"owners_count":21173339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","etl-pipeline","flask","flask-sqlalchemy","jira-rest-api","pandas","python3"],"created_at":"2024-11-16T01:44:18.530Z","updated_at":"2025-04-14T09:52:34.570Z","avatar_url":"https://github.com/toddbirchard.png","language":"Python","funding_links":["https://www.buymeacoffee.com/hackersslackers"],"categories":[],"sub_categories":[],"readme":"# JIRA-to-Database Import\n\n![Python](https://img.shields.io/badge/Python-v3.8-blue.svg?logo=python\u0026longCache=true\u0026logoColor=white\u0026colorB=5e81ac\u0026style=flat-square\u0026colorA=4c566a)\n![Pandas](https://img.shields.io/badge/Pandas-v^1.0.0-blue.svg?logo=python\u0026longCache=true\u0026logoColor=white\u0026colorB=5e81ac\u0026style=flat-square\u0026colorA=4c566a)\n![Requests](https://img.shields.io/badge/Requests-v2.22.0-red.svg?longCache=true\u0026logo=python\u0026longCache=true\u0026style=flat-square\u0026logoColor=white\u0026colorA=4c566a\u0026colorB=5e81ac)\n![SQLAlchemy](https://img.shields.io/badge/SQLAlchemy-v1.3.1-red.svg?longCache=true\u0026style=flat-square\u0026logo=scala\u0026logoColor=white\u0026colorA=4c566a\u0026colorB=bf616a)\n![PyMySQL](https://img.shields.io/badge/PyMySQL-v0.9.3-red.svg?longCache=true\u0026logo=mysql\u0026longCache=true\u0026style=flat-square\u0026logoColor=white\u0026colorA=4c566a\u0026colorB=bf616a)\n![GitHub Last Commit](https://img.shields.io/github/last-commit/google/skia.svg?style=flat-square\u0026colorA=4c566a\u0026colorB=a3be8c\u0026logo=GitHub)\n\n![JIRA SQL ETL](https://storage.googleapis.com/hackersandslackers-cdn/2019/03/jira-etl-3-3@2x.jpg)\n\nExtracts issues from a JIRA instance via the JIRA REST API, transforms the data, and loads data to a database.\n\nTo derive epic-based information from tickets, the script creates an `JiraEpic` table as well as a `JiraIssue` table. The `JiraIssue` table is joined with the former table to easily perform analysis on aggregated epic data.\n\nAccompanying tutorial can be found here: https://hackersandslackers.com/jira-to-sql-etl/\n\n## Installation\n\n**Installation via `requirements.txt`**:\n\n```shell\n$ git clone https://github.com/toddbirchard/jira-database-etl.git\n$ cd jira-database-etl\n$ python3 -m venv myenv\n$ source myenv/bin/activate\n$ pip3 install -r requirements.txt\n$ flask run\n```\n\n**Installation via [Pipenv](https://pipenv-fork.readthedocs.io/en/latest/)**:\n\n```shell\n$ git clone https://github.com/toddbirchard/jira-database-etl.git\n$ cd jira-database-etl\n$ pipenv shell\n$ pipenv update\n$ flask run\n```\n\n## Configuration\n\nThe following environment variables are needed to run this script:\n\n\n* `SQLALCHEMY_DATABASE_URI`: A URI for the database intended to store these tables (ie: _mysql+pymysql://[USER]:[PASSWORD]@d[DATABASE_HOST]:[PORT]/[DATABASE_NAME]_)\n* `SQLALCHEMY_EPIC_TABLE`: Name of database table to store epics.\n* `SQLALCHEMY_JIRA_TABLE`: Name of database table to store JIRA issues.\n* `JIRA_ENDPOINT`: Your JIRA Cloud API endpoint for JQL searching (such as *https://mydomain.atlassian.net/rest/api/3/search*)\n* `JIRA_USERNAME`: Your JIRA username.\n* `JIRA_API_KEY`: An API key associated with the JIRA user.\n* `JIRA_ISSUES_JQL`: JQL to get JIRA issues.\n* `JIRA_ISSUES_FIELDS`: Specific fields to retrieve from the JIRA query.\n* `JIRA_EPICS_JQL`: JQL to get JIRA epics.\n* `JIRA_EPICS_FIELDS`: Specific fields to retrieve from the epics query.\n\n\n### Troubleshooting\n\nMake sure your database contains tables named `JiraEpic` and `JiraIssue` prior to running this script (columns/schema don't matter, these will be overridden).\n\n-----\n\nThis project and all publically-visible repositories are free of charge. If you've found this project to be helpful, a [small donation](https://www.buymeacoffee.com/hackersslackers) would be greatly appreciated to keep us in business. All proceeds go towards coffee, and all coffee goes towards improving these projects.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoddbirchard%2Fjira-database-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftoddbirchard%2Fjira-database-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoddbirchard%2Fjira-database-etl/lists"}