{"id":24565811,"url":"https://github.com/porkcharsui/jira_dumper","last_synced_at":"2026-05-19T03:04:41.021Z","repository":{"id":273780274,"uuid":"920851322","full_name":"porkcharsui/jira_dumper","owner":"porkcharsui","description":"A Playwright-based scraping tool for exporting all Jira issues and attachments.","archived":false,"fork":false,"pushed_at":"2025-01-22T22:21:02.000Z","size":580,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-22T23:23:48.255Z","etag":null,"topics":["export-csvfile","jira","playwright","playwright-python","scraper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/porkcharsui.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-22T22:00:57.000Z","updated_at":"2025-01-22T22:26:52.000Z","dependencies_parsed_at":"2025-01-22T23:23:57.533Z","dependency_job_id":"90da707b-f546-428b-b80a-3e31a9fd1d76","html_url":"https://github.com/porkcharsui/jira_dumper","commit_stats":null,"previous_names":["porkcharsui/jira_dumper"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/porkcharsui%2Fjira_dumper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/porkcharsui%2Fjira_dumper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/porkcharsui%2Fjira_dumper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/porkcharsui%2Fjira_dumper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/porkcharsui","download_url":"https://codeload.github.com/porkcharsui/jira_dumper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243960634,"owners_count":20375108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["export-csvfile","jira","playwright","playwright-python","scraper"],"created_at":"2025-01-23T12:16:27.369Z","updated_at":"2026-05-19T03:04:40.969Z","avatar_url":"https://github.com/porkcharsui.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Jira Dumper\n\n![image](logo.webp)\n\n\"Listen up, fool! The Jira Dumper's here to scrape issues, slice 'em, dice 'em, and pack 'em into XML for ultimate archiving power. Don't mess with your data—preserve it like a champ!\"\n\n## Install / Requirements\n\n### Prerequisites\n\n- Python 3.8+\n- [Poetry](https://python-poetry.org/) installed globally for dependency management\n\n### Installation Steps\n\n1. Clone this repository:\n\n   ```shell\n   git clone git@github.com:porkcharsui/jira_dumper.git\n   cd jira_dumper\n   ```\n\n2. Create a `.env` file and set the required environment variables:\n\n   ```shell\n   # Example of a .env file\n   JIRA_BASE_URL=https://your.jira.instance\n   JIRA_DUMPER_OUTPUT_DIR=output\n   ```\n\n3. Install dependencies using Poetry:\n\n   ```shell\n   poetry install\n   ```\n\n4. Use Poetry to run the tools within this project:\n\n   ```shell\n   poetry run jira_fetch_issues\n   poetry run jira_export_xml\n   poetry run jira_export_xml_attachments\n   ```\n\n## Features\n\n### Authentication\n\nJira Dumper uses [Playwright](https://github.com/microsoft/playwright) with the real Chrome browser to handle authentication. If the session is not authenticated, the tools will pause and allow the user to manually log in through the browser. Once the login is complete, the user must press the ENTER key in the terminal to continue.\n\nThis process captures the authentication state in a file (`auth_state.json`), which is shared across all Jira Dumper tools. Simply delete the file to force a re-authentication on the next run, ensuring a fresh session when needed.\n\n### Data Handling\n\nJira Dumper is designed to preserve all issue data as structured XML data / attachments for long-term archival and further analysis. Unlike PDF exports, which flatten information into non-structured data, XML exports ensures data remains machine-readable and accessible for integration into other workflows or tools.\n\nAttachments are saved with this pattern:\n\n```\n{JIRA_DUMPER_OUTPUT_DIR}/PROJECT_KEY/attachments/ISSUE_KEY/ID-{attachment_id}__{filename}\n```\n\n### Explanation of Variables\n\n- `{JIRA_DUMPER_OUTPUT_DIR}`: The base directory where all output files are stored, as specified in the `.env` file.\n- `{PROJECT_KEY}`: The key of the JIRA project associated with the issues (e.g., `INFRA`, `HR`).\n- `{ISSUE_KEY}`: The unique key for the specific JIRA issue (e.g., `INFRA-123`).\n- `{attachment_id}`: The unique ID of the attachment in JIRA, used to ensure filenames are unique.\n- `{filename}`: The original filename of the attachment. This guarantees uniqueness, prevents overwrites, and maintains clear associations with the original data.\n\n### Efficiency\n\nThe tool skips already downloaded attachments based on timestamps, reducing redundant downloads and saving time. It also dynamically adjusts query ranges to efficiently handle large datasets.\n\n## Steps\n\n### Step 1: Gather Issues\n\nThis script collects Jira issues for one or more projects within a specified date range and saves the results as CSV files.\n\n#### Usage\n\n```shell\n$ poetry run jira_fetch_issues --help\nUsage: jira_fetch_issues [OPTIONS]\n\n  CLI to scrape JIRA issues for specific projects and date ranges.\n\nOptions:\n  --project TEXT     Project key to scrape (e.g., INFRA). Use multiple\n                     `--project` flags for multiple projects.  [required]\n  --start-date TEXT  Start date for the range in YYYY-MM-DD format (default: 2\n                     years ago).\n  --end-date TEXT    End date for the range in YYYY-MM-DD format (default:\n                     today).\n  --base-url TEXT    Base URL of the JIRA instance.\n  --help             Show this message and exit.```\n```\n\n#### Final Command For Multiple Projects\n\n```shell\npoetry run jira_fetch_issues \\\n    --start-date 1984-01-01 \\\n    --project AP --project INFRA --project IT --project HR --project USA\n```\n\n## Step 2: Export Issues\n\nThis script takes one or more CSV files generated in Step 1, loads each issue key, and exports the issue's XML content.\n\n### Usage\n\n```shell\n$ poetry run jira_export_xml --help \nUsage: jira_export_xml [OPTIONS] [FILES]...\n\n  Download issues from CSV files and save their XML exports.\n\n  FILES are one or more paths to CSV files. Globs can be expanded by the shell\n  (e.g., output/*.csv).\n\nOptions:\n  --base-url TEXT  The base URL of the Jira instance (set via JIRA_BASE_URL in\n                   .env file).\n  --help           Show this message and exit.\n```\n\n#### Export a Single CSV\n\n```shell\npoetry run jira_export_xml output/INFRA.csv\n```\n\n#### Export Using Shell Glob for All CSVs\n\n```shell\npoetry run jira_export_xml output/*.csv\n```\n\n## Step 3: Export Attachments\n\n### Usage\n\n```shell\n$ poetry run jira_export_xml_attachments --help \nUsage: jira_export_xml_attachments [OPTIONS] [FILE_PATTERNS]...\n\n  Extract attachments from one or more Jira XML export files and optionally\n  download them. For each file, a new directory named after the issue key will\n  be created at the same path level as the input FILE.\n\n  FILE_PATTERNS are the glob patterns for the XML files.\n\nOptions:\n  --dry-run  Simulate the download and display the attachment details.\n  --force    Force download of all files, regardless of existing timestamps.\n  --help     Show this message and exit.\n```\n\nThis script processes Jira XML files to extract attachments and organize them for archival or backup purposes.\n\nEach attachment is:\n\n1. Stored in a directory named after its issue key.\n2. Located relative to the XML file being processed.\n3. Preserves timestamp metadata to ensure archival accuracy and speed up downloads for existing files, reducing redundant downloads.\n4. Saved with a filename that is always unique, using the format `ID-{attachment_id}__{filename}`.\n\n### Example XML Issue Input and File Output Structure\n\nHere is an example of a `output/USA/USA-1984.xml` file that serves as input:\n\n```xml\n\u003cissue\u003e\n  \u003ckey\u003eUSA-1984\u003c/key\u003e\n  \u003csummary\u003eExample Issue Summary\u003c/summary\u003e\n  \u003cattachments\u003e\n    \u003cattachment id=\"18113\" name=\"the-a-team.png\" size=\"12995\" author=\"Mr. T\" created=\"Wed, 31 Oct 1984 14:38:48 -0700\"/\u003e\n  \u003c/attachments\u003e\n\u003c/issue\u003e\n```\n\nAttachments are downloaded into directories named after their issue keys, relative to the XML file's location (`output/USA/USA-1984.xml`). For example:\n\n```\noutput/USA/attachments/USA-1984/ID-18113__the-a-team.png\n```\n\n### Download Attachments Examples\n\n#### Single Project Glob\n\n```shell\npoetry run jira_export_xml_attachments \"output/INFRA/*.xml\"\n```\n\n#### All Projects Deep Glob\n\n```shell\npoetry run jira_export_xml_attachments \"output/**/*.xml\"\n```\n\nNOTE: Use the `--force` flag to re-download all files regardless of existing timestamps.\n\n\n## Compatibility Note\n\nThis tool has been only been thoroughly tested with Jira OnPrem version v8.3.4. If you experience any issues on different versions of Jira, please report them as an issue.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fporkcharsui%2Fjira_dumper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fporkcharsui%2Fjira_dumper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fporkcharsui%2Fjira_dumper/lists"}