{"id":13645867,"url":"https://github.com/yobulkdev/yobulkdev","last_synced_at":"2025-04-21T17:31:31.714Z","repository":{"id":66340644,"uuid":"570121705","full_name":"yobulkdev/yobulkdev","owner":"yobulkdev","description":"🔥 🔥 🔥Open Source \u0026 AI driven Data Onboarding Platform:Free flatfile.com alternative","archived":false,"fork":false,"pushed_at":"2023-08-03T04:51:35.000Z","size":996,"stargazers_count":880,"open_issues_count":59,"forks_count":45,"subscribers_count":12,"default_branch":"main","last_synced_at":"2024-11-18T04:42:20.712Z","etag":null,"topics":["csv-import","csv-parser","csv-reader","data-engineering","datacleaning","embeddable","javascript","languagemodel","mongodb","nextjs","nodejs","open-source","react","stream","streaming"],"latest_commit_sha":null,"homepage":"https://www.yobulk.dev/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yobulkdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-11-24T11:42:19.000Z","updated_at":"2024-11-06T17:18:21.000Z","dependencies_parsed_at":"2024-01-14T10:01:45.114Z","dependency_job_id":"0a017c1e-cf08-43e1-b89f-7806347a11b4","html_url":"https://github.com/yobulkdev/yobulkdev","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yobulkdev%2Fyobulkdev","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yobulkdev%2Fyobulkdev/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yobulkdev%2Fyobulkdev/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yobulkdev%2Fyobulkdev/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yobulkdev","download_url":"https://codeload.github.com/yobulkdev/yobulkdev/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250100444,"owners_count":21374939,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv-import","csv-parser","csv-reader","data-engineering","datacleaning","embeddable","javascript","languagemodel","mongodb","nextjs","nodejs","open-source","react","stream","streaming"],"created_at":"2024-08-02T01:02:43.824Z","updated_at":"2025-04-21T17:31:31.114Z","avatar_url":"https://github.com/yobulkdev.png","language":"JavaScript","readme":"\u003ch1 align=\"center\" style=\"border-bottom: none\"\u003e\n    \u003cb\u003e\n        \u003ca href=\"https://www.YoBulk.dev\"\u003eYoBulk\u003c/a\u003e\u003cbr\u003e\n    \u003c/b\u003e\n    ⭐️ Open Source Alternative To  \u003ca href=\"https://flatfile.com/\"\u003eFlatfile.com\u003c/a\u003e ⭐️ \u003cbr\u003e\n        \n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cbr\u003e\nA \u003cb\u003eData exchange platform\u003c/b\u003e for all businesses that run on \u003cb\u003eCSV files\u003c/b\u003e.\nCreate YoBulk CSV Import button and make CSV importing \u003cb\u003e10x\u003c/b\u003e faster.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"http://www.yobulk.dev\"\u003e\u003cb\u003eWebsite\u003c/b\u003e\u003c/a\u003e •\n    \u003ca href=\"https://twitter.com/yobulkdev\"\u003e\u003cb\u003eTwitter\u003c/b\u003e\u003c/a\u003e  •\n    \u003ca href=\"https://join.slack.com/t/yobulkdev/shared_invite/zt-1kiutrmhx-6z_Mvq17dW0pPYePrwPocg\" target=\"_blank\"\u003e\u003cb\u003eSlack\u003c/b\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch4 align=\"center\"\u003e\n  \u003ca href=\"https://github.com/uptrainai/uptrain/blob/main/LICENSE\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/license-AGPL-blue\" alt=\"YoBulk is released under the AGPL license.\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://doc.yobulk.dev/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Read-Docs-blue\" alt=\"Docs\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://join.slack.com/t/yobulkdev/shared_invite/zt-1kiutrmhx-6z_Mvq17dW0pPYePrwPocg\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/community%20-slack-yellow\" alt=\"Community\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://yobulk.dev/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/website-yobulk-brightgreen\" alt=\"Website\" /\u003e\n  \u003c/a\u003e\n\u003c/h4\u003e\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://user-images.githubusercontent.com/118799976/220358070-e16a3ffb-3091-4a86-91c9-4a3035d7b10e.jpeg\" alt=\"The Open Source Alternative To Flatfile.com.\" width=\"1000px\" /\u003e\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://user-images.githubusercontent.com/118799976/220357610-aa27cbaf-ea8f-4849-a567-c2c7c63db8f5.jpeg\" alt=\"The Open Source Alternative To Flatfile.com.\" width=\"1000px\" /\u003e\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://user-images.githubusercontent.com/118799976/220358954-ce774959-141d-411d-8603-c999458bfee8.jpeg\" alt=\"The Open Source Alternative To Flatfile.com.\" width=\"1000px\" /\u003e\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://user-images.githubusercontent.com/118799976/220360438-5eec9e5c-241c-43e0-bc17-c962f94accf3.jpeg\" alt=\"The Open Source Alternative To Flatfile.com.\" width=\"1000px\" /\u003e\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://user-images.githubusercontent.com/118799976/220362469-f1cd0b51-2d01-479c-9e50-e9d0f947f205.jpeg\" alt=\"The Open Source Alternative To Flatfile.com.\" width=\"1000px\" /\u003e\u003c/p\u003e\n\n\n\n\n\n\u003cdiv align=\"center\"\u003e\n\n[\u003cimg height=\"38\" src=\"https://user-images.githubusercontent.com/61551451/135263434-75fe793d-42af-49e4-b964-d70920e41655.png\"\u003e](markdown/languages/chinese.md)\n[\u003cimg height=\"38\" src=\"https://user-images.githubusercontent.com/61551451/135263770-38e3e79d-11d4-472e-ac27-ae0f17cf65c4.png\"\u003e](markdown/languages/japanese.md)\n[\u003cimg height=\"38\" src=\"https://user-images.githubusercontent.com/61551451/135263531-fae58600-6616-4b43-95a0-5891019dd35d.png\"\u003e](markdown/languages/german.md)\n[\u003cimg height=\"38\" src=\"https://user-images.githubusercontent.com/61551451/135263888-151d4ad1-7084-4943-97c9-56f28cd40b80.png\"\u003e](markdown/languages/russian.md)\n\n\u003cp align=\"center\"\u003e\u003ca href=\"markdown/languages\"\u003e\u003cb\u003eSee other languages »\u003c/b\u003e\u003c/a\u003e\u003c/p\u003e\n\n\u003c/div\u003e\n\n## User Installation\n\n- [Docker](https://doc.yobulk.dev/GetStarted/Installation#yobulk-installation)\n\n## Docker\n### Option:1 Through Docker Compose\nThe docker compose brings both \u003cb\u003eYoBulk Image and Mongodb Image\u003c/b\u003e.\nWe provide docker-compose.yml file under yobulkdev code repository.\n```bash\ngit clone https://github.com/yobulkdev/yobulkdev.git\ncd yobulkdev \ndocker-compose up -d\n```\n- To make use of \u003cb\u003eYoBulk Open AI\u003c/b\u003e feature, please follow the guidelines as below.\n\n\u003e Please open the `docker-compose.yml` file and update the `OPENAI_SECRET_KEY` variable with the key generated from [OpenAI API site](https://platform.openai.com/account/api-keys).\n  \n\nThe YoBulk dashboard is ready to go at [http://localhost:5050/](http://localhost:5050/).\n\n### Option:2 Through Docker Run\n\nIf you are a Docker user, you may try this way!   \n- Prerequisites:\u003cb\u003eYou should have installed mongodb on your local machine and it's running.\u003c/b\u003e\n```bash\ndocker run --rm -it -p 5050:5050/tcp  yobulk/yobulk\n```\n- To make use of \u003cb\u003eYoBulk Open AI\u003c/b\u003e feature, please follow the guidelines as below.  \n\n\u003e Please pass the Open AI key generated from [OpenAI API site](https://platform.openai.com/account/api-keys)\nin --env to explore YoBulk AI features.\n\nExample:\n```bash\ndocker run --rm -it -p 5050:5050/tcp  --env=\"OPENAI_SECRET_KEY=****\" yobulk/yobulk\n```\nYoBulk dashboard is ready to go at [http://localhost:5050/](http://localhost:5050/).\n\n## Building locally.\n\nIf you are a developer and want to build the system locally, you should have Mongodb running locally.  \n\nYoBulk is a fullstack next.js application which uses Mongodb as it's primary Database.\n\n```bash\ngit clone https://github.com/yobulkdev/yobulkdev\ncd yobulkdev\nyarn install\nyarn run dev\n```\n- To make use of \u003cb\u003eYoBulk Open AI\u003c/b\u003e feature, please follow the guidelines as below.\n\n\u003e Please update `OPENAI_SECRET_KEY` variable in .env file with the Open AI key generated from [OpenAI API site](https://platform.openai.com/account/api-keys)\n\nThe dashboard is ready to view at [http://localhost:5050/](http://localhost:5050/).\n\n## YoBulk Cloud\nYoBulk Cloud is the fastest and most reliable way to run YoBulk. You can get started with free credits in minutes.\n\nSign up for [YoBulk Cloud](https://docs.google.com/forms/d/e/1FAIpQLSfpznd0nsnL5IGWIO1f4O4SBJugxAjI8bI12QqhDP_hqYvJ-A/viewform).\n\n## Quick start\n\nPlease view the [documentation](https://doc.yobulk.dev/GetStarted/Quickstart) to explore CSV importing flow.\n\n## Roadmap\n\n- [YoBulk Roadmap ReadMe](https://doc.yobulk.dev/RoadMap/ProductVision)\n\n# Features\n\n### Supported Features\n\n- ⚡ \u0026nbsp;No code Template and Import button creation\n- ⚡ \u0026nbsp;Smart auto matching between the CSV column and Template column\n- ⚡ \u0026nbsp;Custom validation rule setting against a field\n- ⚡ \u0026nbsp;Delightful Data review experience\n- ⚡ \u0026nbsp;Scalable through streaming.You can import a CSV with size of 1GB\n- ⚡ \u0026nbsp;GPT3 Integration\n- ⚡ \u0026nbsp;Template creation through JSON\n- ⚡ \u0026nbsp;Regex creation and addition in custom validation\n- ⚡ \u0026nbsp;Collaboration \u0026 workspace creation.\n- ⚡ \u0026nbsp;AI based auto suggestion for correcting errors\n\nComing Soon:\n- ⚡ \u0026nbsp;Custom LLM models for Auto suggestion and Auto Cleaning of CSV errors\n- ⚡ \u0026nbsp;Data mapping decission knowledge graph for different sectors\n\n\n\n## Demos\n\n### NoCode Template Creation Demo\nhttps://user-images.githubusercontent.com/118799976/220317753-8e540eb0-af1a-47fe-b384-72081674d14e.mp4\n\n### All DataType Validation Demo\nhttps://user-images.githubusercontent.com/118799976/220321287-8cc38555-51ef-412b-885a-b84cba3e5571.mp4\n\n### String Validation Demo\nhttps://user-images.githubusercontent.com/118799976/220321602-6beebf43-b97e-486b-9aa6-aae9b3f68eca.mp4\n\n### GPT Demo\nhttps://user-images.githubusercontent.com/118799976/220322196-38bc6db3-0b4c-4343-9648-3958ab720c8b.mp4\n\n\n## Why are we building this?\n\nMore than 70% of the business data is shared in the form of CSVs and Excel files. Less than 10% have truly adopted API integrations for data exchange.\n\nAs a developer and product manager, we have gone through the pain of building a scalable CSV importer. Everyday thousands of developers go through the same pain and hundreds of companies have to get enormous amount of spreadsheet and CSV data into a database.\n\nYoBulk is building an OSS standard for CSV and flatfile data onboarding where it can help you to build a data importing solution in-house.\n\nEnterprises like Flatfile.com are addressing this problem but it has constraints like weak data security and high pricing. While developers want an embeddable/scalable CSV importer for SaaS, the truth is that, it is not possible to come up with an \"one-size fits all solution\" within such fragmented market.\n\nAll these \u003cb\u003erestrictions necessitate\u003c/b\u003e our mission, to make it possible for anyone to create an data import solution that suits their needs well.\n\nTo Developers, we would like to offer a \"bring your own validations\"(BYOV)  and \"bring your own database\"(BYOD) experience. You can design and modify validations rules on your own way using YoBulk.\n\nTo enterprises , YoBulk is dedicated to bring data security. Moreover, you have 100% control of your data armed with long-term maintainability.\n\nTo be honest, we do not claim to outperform flatfile.com in terms of functionality and design, at least for now. Besides, our priority is to add more functionality at the moment. We would like to cultivate a community to democratise the knowledge and bring all CSV and flatfile open source projects under a single framework.\n\nRead more [Open vs Closed Data Onboarding](https://doc.yobulk.dev/#yobulk-positioning-open-source-vs-closed-source-data-onboarding-platforms)\n\n# Our Mission\n\nOur mission is to provide the most powerful flatfile(CSV,Spreadsheet) importer which is open source to every single internet business in the world.\nAn AI first data exchange for businesses to share flatfiles where you are in-charge of your customer’s data onboarding, validation \u0026 transformation.\nWe stongly support Anderz's vision.\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://user-images.githubusercontent.com/118799976/222167945-01d5f60b-b5d6-4a7e-aac5-b3265721745d.png\" alt=\"The Open Source Alternative To Flatfile.com.\" width=\"1000px\" /\u003e\u003c/p\u003e\n\n# Our Vision\n\nYoBulk's vision is to create an AI first knowledge graph for carrying out over billions of data mapping decisions while exchanging data between businesses across various sectors through flat files such as CSV and Excel.\n\n\n\n## Support\n\nIf you have any questions or problems, please visit our GitHub discussions page, and we'll try to help you as soon as possible.\n\n[https://github.com/yobulkdev/yobulkdev/discussions](https://github.com/yobulkdev/yobulkdev/discussions)\n\n## License\n\nYoBulk is an open-source software with the [AGPL 3.0 license](https://github.com/yobulkdev/yobulkdev/blob/main/LICENSE.md).\n\n## Contributors\nWe welcome contributions to YoBulk.\n\n\u003ca href=\"https://github.com/yobulkdev/yobulkdev/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=yobulkdev/yobulkdev\" /\u003e\n\u003c/a\u003e\n\n","funding_links":[],"categories":["Others","JavaScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyobulkdev%2Fyobulkdev","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyobulkdev%2Fyobulkdev","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyobulkdev%2Fyobulkdev/lists"}