{"id":34751329,"url":"https://github.com/qrun-io/qbit-sftp-data-integration","last_synced_at":"2026-05-21T09:02:52.338Z","repository":{"id":279700083,"uuid":"939613649","full_name":"QRun-IO/qbit-sftp-data-integration","owner":"QRun-IO","description":"QBit for SFTP file transfers. User-defined imports and exports via SFTP.","archived":false,"fork":false,"pushed_at":"2025-12-28T22:44:04.000Z","size":110,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-12-31T17:45:13.395Z","etag":null,"topics":["etl","file-transfer","java","qbit","qqq","sftp"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/QRun-IO.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-02-26T20:28:58.000Z","updated_at":"2025-12-26T01:32:48.000Z","dependencies_parsed_at":"2025-02-26T23:33:27.929Z","dependency_job_id":"f114a3e5-a8e2-48f1-8548-825668589d42","html_url":"https://github.com/QRun-IO/qbit-sftp-data-integration","commit_stats":null,"previous_names":["kingsrook/qbit-sftp-data-integration","qrun-io/qbit-sftp-data-integration"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/QRun-IO/qbit-sftp-data-integration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QRun-IO%2Fqbit-sftp-data-integration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QRun-IO%2Fqbit-sftp-data-integration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QRun-IO%2Fqbit-sftp-data-integration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QRun-IO%2Fqbit-sftp-data-integration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/QRun-IO","download_url":"https://codeload.github.com/QRun-IO/qbit-sftp-data-integration/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QRun-IO%2Fqbit-sftp-data-integration/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33295246,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-21T02:57:32.698Z","status":"ssl_error","status_checked_at":"2026-05-21T02:57:31.990Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","file-transfer","java","qbit","qqq","sftp"],"created_at":"2025-12-25T05:19:11.928Z","updated_at":"2026-05-21T09:02:52.332Z","avatar_url":"https://github.com/QRun-IO.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# QBit: SFTP Data Integration\n\n[![Version](https://img.shields.io/badge/version-0.1.0-blue.svg)](https://github.com/Kingsrook/qbit-sftp-data-integration)\n[![License](https://img.shields.io/badge/license-GNU%20Affero%20GPL%20v3-green.svg)](https://www.gnu.org/licenses/agpl-3.0.en.html)\n[![Java](https://img.shields.io/badge/java-17+-blue.svg)](https://adoptium.net/)\n\n## Overview\n*Note:  This is one of the original QBit implementations - so, some of the mechanics of how\nit is loaded and used by an application are not exactly fully defined at the time of its\ncreation... Please excuse any dust or not-quite-round wheels you find here!*\n\nThis QBit provides tables and processes to implement an SFTP-based data integration, using\nQQQ's Bulk Load mechanism to allow users to define file import mappings, and user-defined\nReports to define file exports.\n\nThe architecture of such an integration looks like:\n* Shared by both imports and exports:\n  * User-defined records in an `SFTPConnection` table which define connection credentials\n* For imports:\n    * User-defined records in an `SFTPImportConfig` table, which reference an `SFTPConnection` as the source of files to import\n    and a `BulkLoadProfile` as the instructions on how and where to load file contents.\n    * A table built on the QQQ SFTP-backend module, using the `SFTPImportConfig` table as its variant-options table, to serve\n    as the source for files to be imported.\n    * A process that syncs files from the file-source table to a file staging table, along with rows in the `ImportFile` table\n    to track their status.\n    * A process that executes the bulk load process, using the selected bulk load profile, against records in the `ImportFile`\n    table, accessing file contents from the staging file-table. \n* For exports:\n    * User-defined records in an `SFTPExportConfig` table, which reference an `SFTPConnection` as the destination to write files\n      to, whose contents are generated by rendering a `SavedReport` (referencing data from any table in the application)\n    * A table built on the QQQ SFTP-backend module, using the `SFTPExportConfig` table as its variant-options table, to serve\n      as the destination for files to be imported.\n    * A process that renders saved reports, writing their contents to the destination SFTP backend.\n\n## Prerequisites\n\n- **Java 17+** (required for QQQ features)\n- **Maven 3.8+** (for build system)\n- **QQQ Framework** (qqq-backend-module-filesystem \u003e= 0.24.0 for SFTP support)\n- **Active Quartz Scheduler** in your QQQ instance for running scheduled cron jobs\n- **Scheduled Jobs table** in your QQQ instance\n\n## Usage\n\n### Pom dependency\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.kingsrook.qbits\u003c/groupId\u003e\n    \u003cartifactId\u003eqbit-sftp-data-integration\u003c/artifactId\u003e\n    \u003cversion\u003e${TODO}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n### Setup\nUsing this QBit follows the (evolving, but as of this time, standard) QBit Config + Produce pattern.\n\nRequired configs here are:\n* For your SFTP source file table:\n  * Whether you want this QBit to define this table and its SFTP Backend, in which case, you would call this method on the Config object:\n    * `.withSourceFileTableConfig(ProvidedOrSuppliedTableConfig.provideTableUsingBackendNamed(SFTPImportSourceFileBackendMetaDataProducer.NAME))`\n  * Or if you want to define that table yourself and supply it to this QBit:\n      * `.withSourceFileTableConfig(ProvidedOrSuppliedTableConfig.useSuppliedTaleNamed(\"mySftpSourceFileTableName\"))`\n* Similarly, for Staging file table:\n    * If you want this QBit to define this table - note that you will need to always supply your own backend name:\n        * `.withStagingFileTableConfig(ProvidedOrSuppliedTableConfig.provideTableUsingBackendNamed(\"myStagingFileBackendName\"))`\n    * Or if you want to define that table yourself and supply it to this QBit:\n        * `.withStagingFileTableConfig(ProvidedOrSuppliedTableConfig.useSuppliedTaleNamed(\"myStagingFileTableName\"))`\n* For your SFTP destination file table:\n  * Whether you want this QBit to define this table and its SFTP Backend, in which case, you would call this method on the Config object:\n      * `.withDestinationFileTableConfig(ProvidedOrSuppliedTableConfig.provideTableUsingBackendNamed(SFTPExportDestinationFileBackendMetaDataProducer.NAME))`\n  * Or if you want to define that table yourself and supply it to this QBit:\n      * `.withDestinationFileTableConfig(ProvidedOrSuppliedTableConfig.useSuppliedTaleNamed(\"mySftpDestinationFileTableName\"))`\n* The name of your instance's scheduler:\n  * `.withSchedulerName(\"myScheduler\")`\n\nOptional configs include:\n* To enable public-key authentication to SFTP servers, you can load a private key into your application\nserver's sftp-backend, loading it through an environment variable, whose name you can set in the config\nfield: `sftpPrivateKeyEnvVarName`.\n    * By convention, you may wish to set this config value to: `SFTP_PRIVATE_KEY_PEM`.\n    * If this field is not set, then all SFTP connections will require password authentication.\n* A `MetaDataCustomizerInterface\u003cQTableMetaData\u003e` can be run against all tables produced by this QBit:\n    `.withTableMetaDataCustomizer(tableMetaDataCustomizer)`\n* A ProcessTracerCodeReference, which will be used on scheduled jobs ran by this QBit, as in:\n  * `.withProcessTracerCodeReference(new QCodeReference(StandardProcessTracer.class))`\n* If you add additional fields to the `SFTPImportConfig` and `ImportFile` tables (e.g., for record security locks), you need to\ntell the QBit to copy values from `SFTPImportConfig` records `ImportFile` records as they are built:\n  * `.withAdditionalFieldsToCopyFromSftpImportConfigToImportFile(Map.of(\"clientId\", \"clientId\"))`\n\nA full Config \u0026 Produce flow then, in a `MetaDataProducer\u003cMetaDataProducerMultiOutput\u003e`, may look like:\n\n```java\npublic class ImportFileBulkLoadQBitMetaDataLoader extends MetaDataProducer\u003cMetaDataProducerMultiOutput\u003e\n{\n   @Override\n   public MetaDataProducerMultiOutput produce(QInstance qInstance) throws QException\n   {\n      MetaDataProducerMultiOutput rs = new MetaDataProducerMultiOutput();\n\n      MetaDataCustomizerInterface\u003cQTableMetaData\u003e tableMetaDataCustomizer = (instance, table) -\u003e\n      {\n         // any customizations you may want on all tables\n         return (table);\n      };\n\n      ImportFileBulkLoadQBitConfig SFTPDataIntegrationQBitConfig = new ImportFileBulkLoadQBitConfig()\n         .withSourceFileTableConfig(ProvidedOrSuppliedTableConfig.provideTableUsingBackendNamed(SFTPImportSourceFileBackendMetaDataProducer.NAME))\n         .withDestinationFileTableConfig(ProvidedOrSuppliedTableConfig.provideTableUsingBackendNamed(SFTPExportDestinationFileBackendMetaDataProducer.NAME))\n         .withStagingFileTableConfig(ProvidedOrSuppliedTableConfig.provideTableUsingBackendNamed(SFTPImportStagingFileBackendMetaDataProducer.NAME))\n         .withSchedulerName(QuartzSchedulerMetaDataProducer.NAME)\n         .withTableMetaDataCustomizer(tableMetaDataCustomizer)\n         .withProcessTracerCodeReference(new QCodeReference(StandardProcessTracer.class))\n         .withAdditionalFieldsToCopyFromSftpImportConfigToImportFile(Map.of(\"clientId\", \"clientId\"));\n\n      new ImportFileBulkLoadQBitProducer()\n         .withImportFileBulkLoadQBitConfig(SFTPDataIntegrationQBitConfig)\n         .produce(qInstance);\n\n      // any additional customizations you may want\n\n      return (rs);\n   }\n}\n```\n\n## Provides\n### Tables\n* `sftpConnection` - Defines a connection to an SFTP server.\n* `sftpImportConfig` - Defines the usage of an SFTP Server and a Bulk Load Config for loading files into records.\n* `sftpExportConfig` - Defines the usage of an SFTP Server and a saved Report for exporting data.\n* `importFile` - Records which track files imported from an SFTP server, as they are bulk loaded.\n* `SFTPImportSourceFileTable` - _If so configured_ - SFTP-backed file-table, where files to be sync'ed and bulk-loaded come from.\n* `SFTPExportDestinationFileTable` - _If so configured_ - SFTP-backed file-table, where exported reports are written to.\n* `SFTPImportStagingFileTable` - _If so configured_ - Filesystem table, where files imported from an SFTP Import source are stored.\n\n### Processes\n* `SFTPConnectionTester` - Used to validate the setup of SFTP Connection records.\n* `syncSFTPImportConfigScheduledJob` - Table Sync process that manages Scheduled Job records for SFTPImportConfig records with a cron schedule.\n* `SFTPImportFileSyncProcess` - Sync files from an SFTP server source file table (as defined in an SFTP Import Config) to the staging table.\n* `ImportFileBulkLoadProcess` - Run the Bulk Load Process against ImportFile records and their corresponding file data.\n* `RenderReportForSFTPExportProcess` - Render a report, writing it to an SFTP server as specified in an SFTPExportConfig, on a cron schedule.\n\n## Dependencies\n* `qqq-backend-module-filesystem` \u003e= 0.24.0 (for SFTP support)\n* An active Quartz scheduler in your QQQ instance, for running scheduled cron jobs, along with Scheduled Jobs table.\n* Meta-data objects produced by `SavedBulkLoadProfileMetaDataProvider` from qqq-backend-core.\n* Meta-data objects produced by `SavedReportsMetaDataProvider` from qqq-backend-core.\n\n## 🤝 Contributing\n\n**Important**: This repository is a component of the QQQ framework. All contributions, issues, and discussions should go through the main QQQ repository.\n\n### Development Workflow\n\n1. **Fork the main QQQ repository**: https://github.com/Kingsrook/qqq\n2. **Create a feature branch**: `git checkout -b feature/amazing-feature`\n3. **Make your changes** (including QBit changes if applicable)\n4. **Run tests**: `mvn test`\n5. **Commit your changes**: `git commit -m 'Add amazing feature'`\n6. **Push to the branch**: `git push origin feature/amazing-feature`\n7. **Open a Pull Request** to the main QQQ repository\n\n### Code Standards\n\n- **Maven**: Follow Maven best practices\n- **Documentation**: Update relevant documentation\n- **Versioning**: Follow semantic versioning\n\n## 📄 License\n\nThis project is licensed under the GNU Affero General Public License v3.0 - see the [LICENSE](LICENSE) file for details.\n\n```\nQBit: SFTP Data Integration\nCopyright (C) 2021-2024 Kingsrook, LLC\n651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States\ncontact@kingsrook.com | https://github.com/Kingsrook/\n```\n\n**Note**: This is a component of the QQQ framework. For the complete license and more information, see the main QQQ repository: https://github.com/Kingsrook/qqq\n\n## 🏢 About Kingsrook\n\nQBit: SFTP Data Integration is built by **[Kingsrook](https://qrun.io)** - making engineers more productive through intelligent automation and developer tools.\n\n- **Website**: [https://qrun.io](https://qrun.io)\n- **Contact**: [contact@kingsrook.com](mailto:contact@kingsrook.com)\n- **GitHub**: [https://github.com/Kingsrook](https://github.com/Kingsrook)\n\n## 🆘 Support \u0026 Community\n\n### ⚠️ Important: Use Main QQQ Repository\n\n**All support, issues, discussions, and community interactions should go through the main QQQ repository:**\n\n- **Main Repository**: https://github.com/Kingsrook/qqq\n- **Issues**: https://github.com/Kingsrook/qqq/issues\n- **Discussions**: https://github.com/Kingsrook/qqq/discussions\n- **Wiki**: https://github.com/Kingsrook/qqq.wiki\n\n### Why This Repository Exists\n\nThis repository is maintained separately from the main QQQ repository to:\n- **Enable independent QBit development** and versioning\n- **Allow QBit-specific CI/CD** and deployment pipelines\n- **Provide clear separation** between QBit components and core framework concerns\n- **Support different release cycles** for QBits vs. core framework\n\n### Getting Help\n\n- **Documentation**: Check the [QQQ Wiki](https://github.com/Kingsrook/qqq.wiki)\n- **Issues**: Report bugs and feature requests on [Main QQQ Issues](https://github.com/Kingsrook/qqq/issues)\n- **Discussions**: Join community discussions on [Main QQQ Discussions](https://github.com/Kingsrook/qqq/discussions)\n- **Questions**: Ask questions in the main QQQ repository\n\n### Contact Information\n\n- **Company**: Kingsrook, LLC\n- **Email**: contact@kingsrook.com\n- **Website**: https://kingsrook.com\n- **Main GitHub**: https://github.com/Kingsrook/qqq\n\n## 🙏 Acknowledgments\n\n- **QQQ Framework Team**: For the underlying low-code platform\n- **Maven Community**: For the robust build system\n- **Open Source Community**: For the tools and libraries that make this possible\n\n---\n\n**Built with ❤️ by the Kingsrook Team**\n\n**This is a QBit component of the QQQ framework. For complete information, support, and community, visit: https://github.com/Kingsrook/qqq**\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqrun-io%2Fqbit-sftp-data-integration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqrun-io%2Fqbit-sftp-data-integration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqrun-io%2Fqbit-sftp-data-integration/lists"}