{"id":23094866,"url":"https://github.com/ibm/diem","last_synced_at":"2025-08-16T10:33:09.724Z","repository":{"id":37799073,"uuid":"341052442","full_name":"IBM/diem","owner":"IBM","description":"DIEM Data Integration Engine Multipurpose","archived":false,"fork":false,"pushed_at":"2024-09-16T11:06:23.000Z","size":62682,"stargazers_count":10,"open_issues_count":6,"forks_count":5,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-12-15T13:40:44.986Z","etag":null,"topics":["cloud","cron","data-transfers","diem","diem-tribe","docker","etl","execute-sql-statements","pipelines","python","scheduling","slack","spark","sql-statements","typescript"],"latest_commit_sha":null,"homepage":"https://diem.mybluemix.net/diem-help/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IBM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null}},"created_at":"2021-02-22T02:03:50.000Z","updated_at":"2023-06-01T07:44:33.000Z","dependencies_parsed_at":"2023-12-10T19:34:09.912Z","dependency_job_id":null,"html_url":"https://github.com/IBM/diem","commit_stats":{"total_commits":643,"total_committers":6,"mean_commits":"107.16666666666667","dds":0.08864696734059097,"last_synced_commit":"1db15b2a6d77278c26f0b138366d43a3d4adb6a5"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fdiem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fdiem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fdiem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fdiem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IBM","download_url":"https://codeload.github.com/IBM/diem/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230032692,"owners_count":18162491,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud","cron","data-transfers","diem","diem-tribe","docker","etl","execute-sql-statements","pipelines","python","scheduling","slack","spark","sql-statements","typescript"],"created_at":"2024-12-16T22:18:28.822Z","updated_at":"2024-12-16T22:18:32.942Z","avatar_url":"https://github.com/IBM.png","language":"TypeScript","readme":"\u003c!-- markdownlint-disable MD033 --\u003e\n\n# DIEM\n\n[![GitHub issues](https://img.shields.io/github/issues/IBM/diem)](https://github.com/IBM/diem/issues)\n[![GitHub forks](https://img.shields.io/github/forks/IBM/diem)](https://github.com/IBM/diem/network)\n[![GitHub stars](https://img.shields.io/github/stars/IBM/diem)](https://github.com/IBM/diem/stargazers)\n[![GitHub license](https://img.shields.io/github/license/IBM/diem)](https://github.com/IBM/diem/blob/main/LICENSE)\n[![Build Status](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2FIBM%2Fdiem%2Fbadge%3Fref%3Dmain\u0026style=flat)](https://actions-badge.atrox.dev/IBM/diem/goto?ref=main)\n\u003cimg alt=\"Helm\" src=\"https://img.shields.io/badge/dynamic/yaml?label=helm\u0026query=version\u0026prefix=v\u0026url=https://raw.githubusercontent.com/IBM/diem/main/charts/diem/Chart.yaml\"/\u003e\n\u003ca href=\"https://bestpractices.coreinfrastructure.org/projects/5900\"\u003e\u003cimg src=\"https://bestpractices.coreinfrastructure.org/projects/5900/badge\"\u003e\u003c/a\u003e\n\u003cbr/\u003e\u003cbr/\u003e\n\u003cimg alt=\"NodeJS\" src=\"https://img.shields.io/badge/node.js%20-%2343853D.svg?\u0026style=for-the-badge\u0026logo=node.js\u0026logoColor=white\"/\u003e\n\u003cimg alt=\"Angular\" src=\"https://img.shields.io/badge/angular%20-%23DD0031.svg?\u0026style=for-the-badge\u0026logo=angular\u0026logoColor=white\"/\u003e\n\u003cimg alt=\"Python\" src=\"https://img.shields.io/badge/python%20-%2314354C.svg?\u0026style=for-the-badge\u0026logo=python\u0026logoColor=white\"/\u003e\n\u003cimg alt=\"TypeScript\" src=\"https://img.shields.io/badge/typescript%20-%23007ACC.svg?\u0026style=for-the-badge\u0026logo=typescript\u0026logoColor=white\"/\u003e\n\u003cimg alt=\"Webpack\" src=\"https://img.shields.io/badge/webpack%20-%238DD6F9.svg?\u0026style=for-the-badge\u0026logo=webpack\u0026logoColor=black\" /\u003e\n\u003cimg alt=\"Docker\" src=\"https://img.shields.io/badge/docker%20-%230db7ed.svg?\u0026style=for-the-badge\u0026logo=docker\u0026logoColor=white\"/\u003e\n\u003cimg alt=\"Kubernetes\" src=\"https://img.shields.io/badge/kubernetes%20-%23326ce5.svg?\u0026style=for-the-badge\u0026logo=kubernetes\u0026logoColor=white\"/\u003e\n\u003cimg alt=\"GitHub Actions\" src=\"https://img.shields.io/badge/github%20actions%20-%232671E5.svg?\u0026style=for-the-badge\u0026logo=github%20actions\u0026logoColor=white\"/\u003e\n\n\u003e Python, Spark, REST, Scala, Pipelines, Scheduling, API, Custom Jobs, SQL Statements, Openshift, Cloud Native, Machine Learning, Sendgrid, Kubernetes, Slack, Cloud Object Storage, JDBC, Box\n\nDiem can be used to create, display, execute and maintain data transfers between hardware and database platforms. It will cover how to create and manage transfers and assign them to a schedule to execute regularly without human intervention.\n\nDiem provides a front end for SPARK ETL (Extract, Transform, Load) – an SQL data pipeline that can be used to synchronize data between RDMS platforms. Composed of individual transfer operations called jobs, the tool will execute SQL statements to select data from a source system and insert or replicate the data on a target system.\n\nDiem allows the user to create scripts using the interpreted programming language Python, and to create sophisticated schedules using Cron (a work scheduler for Unix systems.) The combination of Python and Cron, along with the intrinsic ability to define and execute custom SQL statements, allows a range of activities from simple data transfers to more sophisticated job streams.\n\nDiem also allows quick and easy definition of connections, as well as a scheduler and log display. An interface to Slack can be used to send the results of jobs to a specified Slack channel.\n\n# Application Features\n\n| Feature | Feature Summary | Benefits |\n| -------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| **Spaces** | Support for Multiple Organisations | Multiple Organisations can make use of DIEM, each org can have it's own space. You can even have multiple spaces per Org and use it for test, pre-prod or production |\n| **Data Transfer** | NodyPy | Fast transfer of small data sets \u003c100 k using pandas jdbc sqlalchemy |\n| **Data Transfer** | Spark | Bulk Transfer of big data using spark, both pyspark an scala.\u003cbr/\u003e Partition your data for paralel inserts.\u003cbr/\u003eWrite you sql online and easy manage your job.\u003cbr/\u003eInclude it in a pipeline.\u003cbr/\u003eGet notified via slack or mail |\n| **Custom Code** | Write your own python code | Write your own python code using pyspark or python. Integrate your favorite library, use your jdbc connection, integrate your config maps, code snippets, webhooks all in one pl;ace, creating a unique experience |\n| **API Services** | Rest services for external use | Create jobs that can provide REST Services. Connect external applications to your code and provide rest services for them |\n| **Machine Learning** | Embed Machine Learning in your code | Make use of the latest ML Libraries like SciPy, matplotlib, seaborn, pandas etc.. to create machine learning models that can be used in your code |\n| **Connections** | DB2\u003cbr/\u003eNetezza\u003cbr/\u003eProgreSQL\u003cbr/\u003eMany more | JDBC connectins into various sources, easy to add and manage.\u003cbr/\u003eSecrets kept secure if personal |\n| **Webhooks** | Bring in your own webhook | Webooks can be to integrate into your applications. You can bring in your git or slack webhook and use it n your applications |\n| **Slack** | Slack Integration | Either you use the default slack channels or bring in your own slack api key. All job progress are logged to your slack channels. You can even integrate them in your custom jobs. Provide custom content and subject messages |\n| **Pipelines** | Pipelins of Jobs | Group your jobs together and form a pipleline. Start each job at the same time or in order. Manage dependences and organize them in steps |\n| **Scheduling** | Cron Schedule | Schedule to run jor jobs using an advance Cron schedule that can handle any type of timeframe and schedule |\n| **Mail** | Mail Functionality | Send mail on Completion or Failure of jour job to your audiance |\n| **Mail Integration** | Mail Functionality for your code | Integrate mail functionality in your code, send data reports as html, csv , xls to your audience based on your query. Customize headers and body content. |\n| **Files** | Upload, Download or integrate files | Each space is connected to it's own Cloud Object Storage Buckewt and can be integrated in your code. You can also specify any other COS instance |\n| **Box** | Upload, Download from BOX | You can now directly download and upload files from Box |\n| **Config Maps** | Manage parameters and config values | Config maps are vary usefull as you can spererate your code from it's values. They can be kept private and secure so you can use them for storing your own tokens. |\n| **Tags** | Define your own tags | You can set up your own tags for easy job search, classification and job management |\n| **Templates** | Reusable or shared Templates | Your code could be based of a template, that you can clone from , you can lso have shared code which is the same amongst your jobs but only different in configuration |\n| **Code Snipptes** | Reusabel adn sharable code | Create reusable code, share use it in your jobs.\u003cbr/\u003eThis allows you to reuse your code in multiple jobs, maintaining key code centrally |\n| **Job Log** | Audit trails of completed job | Each started job will have it's own audit trail, so you can go back to view errors and integrate it in your reporting for performance review |\n| **Organization** | Organization Profile | View your Profile and your access rights organisation |\n| **Organizations** | Organizations you belong to | See all organisations you belong to |\n| **Space Selector** | Easy move between spaces | You can at any time easily swtich between organisations your belong to |\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibm%2Fdiem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fibm%2Fdiem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibm%2Fdiem/lists"}