{"id":21592520,"url":"https://github.com/ylem-co/ylem","last_synced_at":"2026-01-12T14:54:01.596Z","repository":{"id":258969468,"uuid":"851204715","full_name":"ylem-co/ylem","owner":"ylem-co","description":"Ylem is an open-source platform for real-time data streaming orchestration","archived":false,"fork":false,"pushed_at":"2025-05-07T18:37:16.000Z","size":6150,"stargazers_count":71,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T19:41:55.876Z","etag":null,"topics":["data","data-visualization","dataorchestration","etl","etl-framework","etl-pipeline","ide","ingestion","orchestration","pipelines","processing","real-time","reverse-etl","scheduler","streaming","streaming-data","transformation","workflows"],"latest_commit_sha":null,"homepage":"https://ylem.co/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ylem-co.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-02T16:13:54.000Z","updated_at":"2025-05-07T18:37:18.000Z","dependencies_parsed_at":"2024-11-06T11:41:59.881Z","dependency_job_id":null,"html_url":"https://github.com/ylem-co/ylem","commit_stats":null,"previous_names":["ylem-co/ylem"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ylem-co/ylem","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ylem-co%2Fylem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ylem-co%2Fylem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ylem-co%2Fylem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ylem-co%2Fylem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ylem-co","download_url":"https://codeload.github.com/ylem-co/ylem/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ylem-co%2Fylem/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28340411,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T12:22:26.515Z","status":"ssl_error","status_checked_at":"2026-01-12T12:22:10.856Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","data-visualization","dataorchestration","etl","etl-framework","etl-pipeline","ide","ingestion","orchestration","pipelines","processing","real-time","reverse-etl","scheduler","streaming","streaming-data","transformation","workflows"],"created_at":"2024-11-24T17:01:19.857Z","updated_at":"2026-01-12T14:54:01.575Z","avatar_url":"https://github.com/ylem-co.png","language":"JavaScript","readme":"\u003cp align=\"center\"\u003e\n  \u003cimg width=\"748\" title=\"Ylem. The open-source data streaming platform\" alt=\"Ylem. The open-source data streaming platform\" src=\"https://github.com/user-attachments/assets/7753c3a6-ef48-451b-bca2-722f1dd0d8c7\" /\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n  ![GitHub branch check runs](https://img.shields.io/github/check-runs/ylem-co/ylem/main?color=green)\n  ![Static Badge](https://img.shields.io/badge/Go-1.23-black)\n  ![Static Badge](https://img.shields.io/badge/React-18.3.1-black)\n  \u003ca href=\"https://github.com/ylem-co/ylem?tab=Apache-2.0-1-ov-file\"\u003e![Static Badge](https://img.shields.io/badge/license-Apache%202.0-black)\u003c/a\u003e\n  \u003ca href=\"https://github.com/ylem-co/ylem/tags\"\u003e![Static Badge](https://img.shields.io/badge/tag-v0.0.1_pre_release-black)\u003c/a\u003e\n  \u003ca href=\"https://ylem.co\" target=\"_blank\"\u003e![Static Badge](https://img.shields.io/badge/website-ylem.co-black)\u003c/a\u003e\n  \u003ca href=\"https://docs.ylem.co\" target=\"_blank\"\u003e![Static Badge](https://img.shields.io/badge/documentation-docs.ylem.co-black)\u003c/a\u003e\n  \u003ca href=\"https://join.slack.com/t/ylem-co/shared_invite/zt-2nawzl6h0-qqJ0j7Vx_AEHfnB45xJg2Q\" target=\"_blank\"\u003e![Static Badge](https://img.shields.io/badge/community-join%20Slack-black)\u003c/a\u003e\n\u003c/div\u003e\n\n# Ylem\nThe open-source data streaming platform is a one-stop-shop solution for orchestrating data streams on top of Apache Kafka, Amazon SQS, Google Pub/Sub, RabbitMQ, various APIs, and data storages.\n\n\u003cimg width=\"1158\" alt=\"Screenshot 2024-10-18 at 13 20 37\" src=\"https://github.com/user-attachments/assets/fee384d3-bc10-4681-a3d7-a8a9f5dc8983\"\u003e\n\n|    |  |  |\n| ------------- | ---------- | ----------- |\n| \u003cimg  alt=\"User dashboard\" src=\"https://github.com/user-attachments/assets/3740b8bd-f127-4ccb-8496-3b68888c5846\"\u003e | \u003cimg alt=\"Pipeline running\" src=\"https://github.com/user-attachments/assets/00520fa1-3912-4c9a-850e-546fdd541a06\"\u003e | \u003cimg alt=\"Pipeline log\" src=\"https://github.com/user-attachments/assets/628e1be5-5a45-4935-a9fa-6769edada491\"\u003e  |\n\n# Installation\n\n## Install Docker 4\n\nIf you don't yet have Docker 4 installed, [install](https://www.docker.com/products/docker-desktop/) it from their official website for your OS.\n\n## Install Ylem\n\n### Option 1. Install from pre-build containers\n\nThe best way to install Ylem is to clone the repository https://github.com/ylem-co/ylem-installer and follow the installation instructions from it. It will install Ylem from the latest version of pre-build containers stored on Docker Hub.\n\nYlem will be available at http://localhost:7331/\n\n### Option 2. Build and install from the source\n\nIf you want to compile Ylem from the source, run `docker compose up` or `docker compose up -d` from this repository. It will compile the code and run all the necessary containers.\n\nYlem is available at http://127.0.0.1:7330/\n\n:warning: Please pay attention. Compiling from the source might take some time and will keep the resources of your machine busy.\n\n#### To rebuild a particular container\n\nIf you want to rebuild a particular container from a source locally, run the following:\n\n``` bash\ndocker compose build --no-cache %%CONTAINER_NAME%%\n```\n\nE.g.\n\n``` bash\ndocker compose build --no-cache ylem_users\n```\n\n# Using your own Apache Kafka cluster\n\nYlem uses Apache Kafka to exchange messages for processing pipelines and tasks. By default Ylem already comes with the pre-configured Apache Kafka container, however, you might already have an Apache Kafka cluster in your infrastructure and might want to reuse it.\n\nIn this case, you need to take the steps described [here in our documentation](https://docs.ylem.co/open-source-edition/usage-of-apache-kafka).\n\n# Configuring environment variables in .env files\n\nSome particular integrations might require extra steps and using `.env` files. Configure them if you need to.\n\nThe list of such integrations and more information about them is in [our documentation](https://docs.ylem.co/open-source-edition/configuring-integrations-with-.env-variables).\n\n# Folder structure in this repository\n\nYlem is a set of microservices. Each microservice is represented by one or more containers in the same network and communicates with each other via the API.\n\n``` bash\n|-- api                  # api microservice\n|-- backend\n|--|-- integrations      # integrations with external APIs, databases and other software\n|--|-- pipelines         # pipelines, tasks, connectors\n|--|-- statistics        # statistics of pipeline and task runs\n|--|-- users             # users and organizations\n|-- database             # a container for storing databases for all the microservices\n|-- processor\n|--|-- python_processor  # processor of the Python code written in pipelines \n|--|-- taskrunner        # task runner and load balancer\n|-- server               # Nginx container in front of all the microservice APIs allowing to avoid CORS issues on the UI side\n|-- ui                   # user interface\n```\n\nEach microservice has its own README file containing more information about its usage and functionality.\n\n# Documentation\n\nThe user and developer documentation of Ylem is available at https://docs.ylem.co/.\n\nThe [open-source section](https://docs.ylem.co/open-source-edition) contains information about the [task-processing architecture](https://docs.ylem.co/open-source-edition/task-processing-architecture) and [configuration of integrations](https://docs.ylem.co/open-source-edition/configuring-integrations-with-.env-variables) using .env files and parameters.\n\n# Explore our additional integration packages\n\n| Integration   | Repository | Description |\n| ------------- | ---------- | ----------- |\n| \u003cimg width=\"100px\" alt=\"Apache Kafka\" title=\"Apache Kafka\" src=\"https://docs.ylem.co/~gitbook/image?url=https%3A%2F%2F3180830455-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FD0FT8l3QzMrw546vOdHU%252Fuploads%252FFGLlkFHxjsJMN4bfHJFR%252Fkf.png%3Falt%3Dmedia%26token%3D47a37fc7-13df-410e-9760-85525a424fde\u0026width=245\u0026dpr=2\u0026quality=100\u0026sign=b464460b\u0026sv=1\"\u003e  | https://github.com/ylem-co/ylem-kafka-trigger | Containerized Apache Kafka listener to stream data to Ylem |\n| \u003cimg width=\"100px\" alt=\"RabbitMQ\" title=\"RabbitMQ\" src=\"https://docs.ylem.co/~gitbook/image?url=https%3A%2F%2F3180830455-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FD0FT8l3QzMrw546vOdHU%252Fuploads%252FOG2ufh5uxAi54ht1kIBm%252Frabbitmq_logo_icon_170812.png%3Falt%3Dmedia%26token%3D01fb235c-59a3-4a3d-83e3-f6a256f99dd6\u0026width=245\u0026dpr=2\u0026quality=100\u0026sign=d1eda75a\u0026sv=1\"\u003e      | https://github.com/ylem-co/ylem-rabbitmq-consumer | Containerized RabbitMQ consumer to stream data to Ylem |\n| \u003cimg width=\"100px\" alt=\"AWS S3\" title=\"AWS S3\" src=\"https://docs.ylem.co/~gitbook/image?url=https%3A%2F%2F3180830455-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FD0FT8l3QzMrw546vOdHU%252Fuploads%252F9ltBsXtDuKbyl1YlaX1M%252Fs3.png%3Falt%3Dmedia%26token%3Daec101c7-f899-4c6b-ad91-7b230b694a63\u0026width=245\u0026dpr=2\u0026quality=100\u0026sign=b3b158ab\u0026sv=1\"\u003e | https://github.com/ylem-co/s3-lambda-trigger  | AWS Lambda function to stream data from AWS S3 to Ylem |\n| \u003cimg width=\"100px\" alt=\"Tableau\" title=\"Tableau\" src=\"https://docs.ylem.co/~gitbook/image?url=https%3A%2F%2F3180830455-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FD0FT8l3QzMrw546vOdHU%252Fuploads%252Fr5M0FiIfvC7on34tLYNN%252FTB.png%3Falt%3Dmedia%26token%3D3d29f3ad-2df5-454f-ae2c-c1cc4a1f191c\u0026width=245\u0026dpr=2\u0026quality=100\u0026sign=8ee0d519\u0026sv=1\"\u003e       | https://github.com/ylem-co/tableau-http-wrapper | Containerized HTTP wrapper to stream data from Ylem to Tableau |\n\n# Key contributors\n\n* [olschaefer](https://github.com/olschaefer)\n* [schneekatze](https://github.com/schneekatze)\n* [lunoshot](https://github.com/lunoshot)\n* [Ardem](https://github.com/Ardem)\n","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fylem-co%2Fylem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fylem-co%2Fylem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fylem-co%2Fylem/lists"}