{"id":17669304,"url":"https://github.com/snimmagadda1/stack-exchange-dump-to-mysql","last_synced_at":"2025-03-30T13:44:08.721Z","repository":{"id":102028435,"uuid":"322180635","full_name":"snimmagadda1/stack-exchange-dump-to-mysql","owner":"snimmagadda1","description":"Batch pipeline to import Stack Exchange XML data dumps to relational DB","archived":false,"fork":false,"pushed_at":"2022-02-08T18:45:25.000Z","size":53261,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-05T15:42:47.938Z","etag":null,"topics":["batch","data","mysql","spring-batch","stackoverflow"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snimmagadda1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-17T04:30:39.000Z","updated_at":"2022-01-30T22:07:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"8611c69b-8c00-43b9-9bc4-76f77e7256a4","html_url":"https://github.com/snimmagadda1/stack-exchange-dump-to-mysql","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snimmagadda1%2Fstack-exchange-dump-to-mysql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snimmagadda1%2Fstack-exchange-dump-to-mysql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snimmagadda1%2Fstack-exchange-dump-to-mysql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snimmagadda1%2Fstack-exchange-dump-to-mysql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snimmagadda1","download_url":"https://codeload.github.com/snimmagadda1/stack-exchange-dump-to-mysql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246326600,"owners_count":20759436,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","data","mysql","spring-batch","stackoverflow"],"created_at":"2024-10-24T00:08:27.906Z","updated_at":"2025-03-30T13:44:08.696Z","avatar_url":"https://github.com/snimmagadda1.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003estack-exchange-dump-to-mysql 👋\u003c/h1\u003e\n\u003cp\u003e\n  \u003cimg alt=\"Version\" src=\"https://img.shields.io/badge/version-0.0.1-blue.svg?cacheSeconds=2592000\" /\u003e\n  \u003ca href=\"LICENSE.md\" target=\"_blank\"\u003e\n    \u003cimg alt=\"License: MIT\" src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003e A quick pipeline to import [Stack Exchange XML dump](https://archive.org/details/stackexchange) data to a relational db\n\n### 🏠 [TODO](https://s11a.com)\n\n## Install\n\n```sh\nmvn clean package\n```\n\n## Usage\n\nBefore the pipeline is run, the `schema-base.sql` must be executed on the desired output schema. This will initialize\nthe tables and create necessary indices for the data dump.\n\nRun with docker (taking care to pass the required app.datasource.xxx and spring.datasource.xxx properties as env vars):\n\n```sh\ndocker run -e APP_DATASOURCE_URL=XXXXX -e ... snimmagadda/stacke-batch-mysql:latest \n```\n\nTo run from source, `app.datasource.xxx` properties should be updated accordingly. Metrics job/task metadata by default\nare output to an in-memory HSQL DB which can be overridden with the `spring.datasource.xxx` properties. Example yaml:\n\n```\napp:\n  datasource:\n    dialect: org.hibernate.dialect.MySQLDialect\n    driver-class-name: org.mariadb.jdbc.Driver\n    url: \"jdbc:mysql://localhost:3306/stacke\"\n    username: \"root\"\n    password: \"password\"\n```\n\nStreamlined ways to import are a W.I.P. For now, manual configuration of application.yaml is required, and running from\nsource is the simplest way to pass in custom datafiles. Once properties are configured, you can run locally with the\nfollowing:\n\n```sh\nmvn spring-boot:run\n```\n\n## Run tests\n\n```sh\nmvn test\n```\n\n## Author\n\n👤 **Sai Nimmagadda**\n\n* Website: https://s11a.com\n* Github: [@snimmagadda1](https://github.com/snimmagadda1)\n\n## 🤝 Contributing\n\nContributions, issues and feature requests are welcome!\u003cbr /\u003eFeel free to\ncheck [issues page](https://github.com/snimmagadda1/stackexchange-dump-to-mysql/issues).\n\n## 📝 License\n\nCopyright © 2020 [Sai Nimmagadda](https://github.com/snimmagadda1).\u003cbr /\u003e\nThis project is [MIT](LICENSE.md) licensed.\n\n***\n_This README was generated with ❤️ by [readme-md-generator](https://github.com/kefranabg/readme-md-generator)_\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnimmagadda1%2Fstack-exchange-dump-to-mysql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnimmagadda1%2Fstack-exchange-dump-to-mysql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnimmagadda1%2Fstack-exchange-dump-to-mysql/lists"}