{"id":17004630,"url":"https://github.com/emreyalvac/sulfur","last_synced_at":"2025-08-19T16:33:02.386Z","repository":{"id":188917542,"uuid":"679686455","full_name":"emreyalvac/sulfur","owner":"emreyalvac","description":"Shaping, Processing, and Transforming Data with the Power of Sulfur with Rust","archived":false,"fork":false,"pushed_at":"2024-03-11T06:30:25.000Z","size":172,"stargazers_count":11,"open_issues_count":4,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-10-31T10:43:51.786Z","etag":null,"topics":["data","data-analysis","data-flow","database"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/emreyalvac.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-08-17T11:47:55.000Z","updated_at":"2024-03-10T18:54:15.000Z","dependencies_parsed_at":"2024-03-11T07:48:38.004Z","dependency_job_id":null,"html_url":"https://github.com/emreyalvac/sulfur","commit_stats":null,"previous_names":["emreyalvac/sulfur"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emreyalvac%2Fsulfur","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emreyalvac%2Fsulfur/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emreyalvac%2Fsulfur/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emreyalvac%2Fsulfur/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/emreyalvac","download_url":"https://codeload.github.com/emreyalvac/sulfur/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230364005,"owners_count":18214717,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","data-analysis","data-flow","database"],"created_at":"2024-10-14T04:44:04.790Z","updated_at":"2024-12-19T02:09:03.338Z","avatar_url":"https://github.com/emreyalvac.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sulfur\n![alt text](sulfur.jpeg)\n\n#### WIP\n\nhttps://www.youtube.com/watch?v=PAAvNmoqDq0 \u003cbr/\u003e\n\n## \"Shaping, Processing, and Transforming Data with the Power of Sulfur\"\n\nWelcome to the Sulfur project, where we harness the elemental power of data transformation. Just like sulfur can reshape\nits form, our platform reshapes, processes, and transforms data, turning it into valuable insights.\n\nJoin us on this journey of alchemy where data turns into gold through customization and innovation. Unleash the\npotential of Sulfur and turn raw data into refined intelligence.\n\n## Usage\n\n1. Clone the repository:\n\n    ```sh\n    git clone https://github.com/emreyalvac/sulfur.git\n    ```\n\n2. Navigate to the project directory:\n\n    ```sh\n    cd sulfur\n    ```\n\n3. Install dependencies:\n\n    ```sh\n    cargo build\n    ```\n\n4. Configure `config.yml` for your pipelines.\n\n5. To run pipelines, use the terminal:\n\n    ```sh\n    cargo run -- --flow flow.yml\n    ```\n\n## Configuration (`config.yml`)\n\nHere's an example of how to configure your pipelines using the `config.yml` file:\n\n```yaml\nsulfur:\n  - name: \"Mongo to Redis\"\n    cron: \"0 0 * * *\"\n    source:\n      type: \"Mongo\"\n      host: \"example.com\"\n      port: \"5432\"\n      user: \"user\"\n      password: \"password\"\n      database: \"db_name\"\n      collection: \"collection_name\"\n    destination:\n      type: \"Redis\"\n      host: \"redis.example.com\"\n      port: \"6379\"\n      password: \"redis_password\"\n      key: \"data_key\"      \n```\n\n### Configuration Details\n\n- Each pipeline is defined under the `sulfur` key.\n- `name`: A descriptive name for your pipeline.\n- `cron` (optional): The cron expression to schedule pipeline runs (e.g., \"0 0 * * *\" for daily runs).\n- `source`: Specifies the data source configuration.\n- `destination`: Specifies the data destination configuration.\n\nFor the `source` and `destination` configurations, you can specify various parameters based on the type of engine you're\nusing (e.g., \"Database,\" \"Redis,\" \"BigQuery,\" etc.).\n\nRemember to customize the configuration according to your project's specific requirements.\n\n## Transform Data, Shape Intelligence\n\nSulfur is more than a project; it's a catalyst for data alchemy. Whether you're merging, filtering, or aggregating,\nSulfur empowers you to sculpt raw data into refined insights, making your data truly valuable.\n\n## Supported Engines\n\nSulfur currently supports the following data storage engines:\n\n| Engine        | Type                 | Description                                | Parameters                                                       |\n|---------------|----------------------|--------------------------------------------|------------------------------------------------------------------|\n| ElasticSearch | Database         | Use Elasticsearch as a data source or destination. | host, port, user, password, index                                |\n| MongoDB       | Database             | Use MongoDB as a data source or destination. | host, port, user, password, database, collection                 |\n| Redis         | In-Memory Data Store | Use Redis as a data source or destination. | host, port, password, key                                        |\n| BigQuery      | Data Warehouse       | Use Google BigQuery as a data destination. | project_id, dataset_id, table_id, credentials (service_key.json) |\n\nWe're committed to expanding the list of supported engines to give you even more flexibility. Adding new platforms is a\nstraightforward process, allowing you to tailor Sulfur to your evolving data needs.\n\n## Upcoming Storage Possibilities\n\nAt Sulfur, we're committed to expanding the range of supported storage engines to cater to your evolving needs. Here's a\nsneak peek at some potential storage engines that might be added in the future:\n\n1. Amazon S3\n2. Microsoft Azure Blob Storage\n3. PostgreSQL\n4. MySQL\n5. SQLite\n6. Cassandra\n7. Apache Hadoop HDFS\n8. Amazon Redshift\n9. Snowflake\n10. Apache Kafka\n11. Oracle Database\n12. IBM Db2\n13. Microsoft SQL Server\n14. Apache Hive\n15. MongoDB Atlas\n16. Elasticsearch Service\n17. Redis Cloud\n18. Memcached\n19. InfluxDB\n20. Kafka\n21. RabbitMQ\n\nStay tuned as we continue to explore and add more storage engine options to the Sulfur platform. We're excited to\nprovide you with a broader range of choices for your data storage needs!\n\n## Advanced Data Transformation\n\nAt Sulfur, we're dedicated to evolving our platform to meet your needs. We're excited to introduce a feature:\n\n### How It Works\n\nThe Advanced Data Transformation feature will provide a powerful toolkit for crafting precise data transformations. From\nmathematical operations to conditional logic, this feature grants you unparalleled control over your data.\n\nStay tuned as we work diligently to unveil this enhancement. Your data transformation possibilities are about to expand\nlike never before!\n\n## Advanced Data Transformation using Python\n\nSulfur enables you to harness the power of custom Python scripts for advanced data transformation during the pipeline process. By integrating Python scripts, you can perform complex data manipulations, calculations, and enrichments before the data is forwarded to its destination.\n\n### Implementing Advanced Transformation\n\nTo demonstrate the power of custom Python scripts for data transformation, we've provided an example `advanced_transform` function that showcases a basic transformation:\n\n```python\nimport json\n\ndef advanced_transform(*args):\n   # Unpack the arguments tuple\n   data_string = args[0]\n\n   # Load the JSON data\n   data = json.loads(data_string)\n\n   # Perform your advanced transformation here\n   transformed_data = {\n      \"name\": \"TRANSFORMED\",\n      \"original_data\": data\n   }\n\n   # Convert the transformed data back to a JSON string\n   transformed_json = json.dumps(transformed_data)\n\n   return transformed_json\n```\n\n```yaml\nsulfur:\n  - name: \"Pipeline1\"\n    cron: \"0 0 * * *\"\n    transform:\n      file: './transform.py'\n      fn: 'advanced_transform'\n    source:\n      type: \"MongoDB\"\n      host: \"mongodb.example.com\"\n      port: \"27017\"\n      user: \"user\"\n      password: \"password\"\n      database: \"db_name\"\n      collection: \"collection_name\"\n    destination:\n      type: \"Redis\"\n      host: \"redis.example.com\"\n      port: \"6379\"\n      password: \"redis_password\"\n      key: \"data_key\"\n\n```\n\n## Contributing\n\nSee [Contributing Guidelines](CONTRIBUTING.md) for details on how to contribute to this project.\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femreyalvac%2Fsulfur","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femreyalvac%2Fsulfur","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femreyalvac%2Fsulfur/lists"}