{"id":21684385,"url":"https://github.com/knovator/data-import","last_synced_at":"2026-01-30T10:04:26.618Z","repository":{"id":39866380,"uuid":"431110673","full_name":"knovator/data-import","owner":"knovator","description":"Project built to faster the data import functionality.","archived":false,"fork":false,"pushed_at":"2022-07-29T04:53:06.000Z","size":244,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-03T14:42:39.970Z","etag":null,"topics":["data-import","mongodb","nodejs","rabbitmq","spreadsheet"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/knovator.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-23T13:26:38.000Z","updated_at":"2022-07-29T04:55:11.000Z","dependencies_parsed_at":"2022-08-27T01:40:53.608Z","dependency_job_id":null,"html_url":"https://github.com/knovator/data-import","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/knovator/data-import","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knovator%2Fdata-import","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knovator%2Fdata-import/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knovator%2Fdata-import/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knovator%2Fdata-import/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/knovator","download_url":"https://codeload.github.com/knovator/data-import/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knovator%2Fdata-import/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28910969,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T08:15:08.179Z","status":"ssl_error","status_checked_at":"2026-01-30T08:14:31.507Z","response_time":66,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-import","mongodb","nodejs","rabbitmq","spreadsheet"],"created_at":"2024-11-25T16:15:31.830Z","updated_at":"2026-01-30T10:04:26.599Z","avatar_url":"https://github.com/knovator.png","language":"JavaScript","readme":"\u003cdiv id=\"top\"\u003e\u003c/div\u003e\n\n\u003c!-- PROJECT LOGO --\u003e\n\u003cbr /\u003e\n\u003cdiv align=\"center\"\u003e\n\n\u003ch3 align=\"center\"\u003e@knovator/data-import\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\n    \u003cb\u003e@knovator/data-import\u003c/b\u003e is project built to faster the data import functionality.\n    \u003cbr /\u003e\n    \u003ca href=\"https://github.com/knovator/data-import\"\u003e\u003cstrong\u003eExplore the docs »\u003c/strong\u003e\u003c/a\u003e\n    \u003cbr /\u003e\n    \u003cbr /\u003e\n    \u003ca href=\"https://github.com/knovator/data-import\"\u003eView Demo\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/knovator/data-import/issues\"\u003eReport Bug\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/knovator/data-import/issues\"\u003eRequest Feature\u003c/a\u003e\n  \u003c/b\u003e\n\u003c/div\u003e\n\n\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#about-the-project\"\u003eAbout The Project\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#built-with\"\u003eBuilt With\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#prerequisites\"\u003ePrerequisites\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\n          \u003ca href=\"#routes\"\u003eRoutes\u003c/a\u003e\n          \u003cul\u003e\n            \u003cli\u003e\u003ca href=\"#health-check\"\u003eHealth Check\u003c/a\u003e\u003c/li\u003e\n            \u003cli\u003e\u003ca href=\"#user-routes\"\u003eUser Routes\u003c/a\u003e\u003c/li\u003e\n            \u003cli\u003e\u003ca href=\"#project-routes\"\u003eProject Routes\u003c/a\u003e\u003c/li\u003e\n            \u003cli\u003e\u003ca href=\"#templates-routes\"\u003eTemplates Routes\u003c/a\u003e\u003c/li\u003e\n            \u003cli\u003e\u003ca href=\"#columns-routes\"\u003eColumns Routes\u003c/a\u003e\u003c/li\u003e\n          \u003c/ul\u003e\n        \u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#usage-flow\"\u003eFlow\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contact\"\u003eContact\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n\n\u003c!-- ABOUT THE PROJECT --\u003e\n## About The Project\n\nOften times uploaded spreadsheets in projects are too large to handle at once by backend applications. And to handle large quantity of imported data we need to write same solutions again and again to convert XML-to-JSON, validating data and processing that large data in chunks.  \n\n`data-import` is built with a intention to reduce repetitions of conversion, validation and importing data. It manages data in format of projects, templates and columns.\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n### Built With\n\n* [@knovator/api](https://www.npmjs.com/package/@knovator/api)\n* [axios](https://www.npmjs.com/package/axios)\n* [express](https://www.npmjs.com/package/express)\n* [mongoose](https://www.npmjs.com/package/mongoose)\n* [multer](https://www.npmjs.com/package/multer)\n* [nodemailer](https://www.npmjs.com/package/nodemailer)\n* [passport](https://www.npmjs.com/package/passport)\n* [winston](https://www.npmjs.com/package/winston)\n* [xlsx](https://www.npmjs.com/package/xlsx)\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\u003c!-- GETTING STARTED --\u003e\n## Getting Started\n`data-import` is built to run as standalone application.\n\n### Prerequisites\n`data-import` uses [MongoDB](https://www.mongodb.com/) database and [RabbitMQ](https://www.rabbitmq.com/) Queue to process imported data. So you need to have **MongoDB** and **RabbitMQ** URLS ready\n\n### Installation\n\n1. Clone the repo, and go into it\n   ```sh\n   git clone https://github.com/knovator/data-import.git\n   cd data-import\n   ```\n2. Install NPM packages and install **esm** package\n   ```sh\n   yarn install\n   ```\n3. Copy `.env.example` file to `.env`\n    ```sh\n    cp .env.example .env\n    ```\n4. Edit your credentials in `.env` like **MongoDB** and **RabbitMQ** URLs.\n5. Start the application\n    ```sh\n    yarn start\n    ```\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\u003c!-- USAGE EXAMPLES --\u003e\n## Usage\n\n`data-import` provides set of Routes to Work with.\n\n### Routes\n\n#### Health Check\n| path | Method | Validations | Description | Accepts | Returns |\n| --- | --- | --- | ----------- | --- | --- |\n| `/v1/status` | `GET` | | Returns **OK**, if application is running well | | |\n\n#### User Routes\nUser Routes deals with User schema listed in [user-schema](data-formats.md#user-schema).\n| path | Method | Validations | Description | Accepts | Returns |\n| --- | --- | --- | ----------- | --- | --- |\n| `/v1/users/list` | `GET` | | List all users | | List of registered Users |\n| `/v1/users/` | `POST` | [data-formats/create-user](data-formats.md#create-user) | Creates user | User Data | Created User |\n| `/v1/users/` | `GET` | | Get users with Pagination | | |\n\n#### Project Routes\nProject Routes deals with Project schema listed in [data-formats.md#project-schema](data-formats.md#project-schema).\n| path | Method | Validations | Description | Accepts | Returns |\n| --- | --- | --- | ----------- | --- | --- |\n| `/v1/projects/:projectId` | `GET` | | Get Project Details | `projectId` | List of Projects |\n| `/v1/projects/` | `POST` | [data-formats/create-project](data-formats.md#create-project) | Create Project | Project Data | Created Project |\n| `/v1/projects/` | `GET` | | Get projects with Pagination | | |\n\n\n#### Templates Routes\nTemplates Routes deals with Templates schema listed in [data-formats.md#template-schema](data-formats.md#template-schema).\n| path | Method | Validations | Description | Accepts | Returns |\n| --- | --- | --- | ----------- | ----------- | ----------- |\n| `/v1/templates/:templateId` | `GET` | | Get Template Details | | |\n| `/v1/templates/` | `POST` | | Upload **sample** spreadsheet file as template | | |\n| `/v1/templates/:templateId/process-file` | `POST` | | Upload spreadsheet | **files** in body | Uploaded spreadsheet data details |\n| `/v1/templates/:templateId/map-data` | `POST` | | Map spreadsheet Data | Uploaded spreadsheet data details | |\n\n#### Columns Routes\nColumns Routes deals with Column schema listed in [data-formats.md#column-schema](data-formats.md#column-schema).\n| path | Method | Validations | Description | Accepts | Returns |\n| --- | --- | --- | ----------- | ----------- | ----------- |\n| `/v1/columns/` | `GET` | | Get Paginated Columns | | |\n| `/v1/columns/` | `POST` | | Create Column | | |\n\n### Usage Flow\n1. Create Project\n```cURL\nHTTP POST /v1/projects\n  -d '{ 'nm': '...', 'cd': '...', 'cbUrl': '...' }'\n```\n\n2. Add Template\n```\nHTTP POST /v1/templates\n  -d '{ 'nm': '...', 'cd': '...', 'cb':{ 'm': '...', 'u': '...' }, 'p':{ 'cd': '...', 'id': '...', 'nm': '...' } }'\n```\n\n3. Add Columns to Template\n```\nHTTP POST /v1/projects\n  -d '{ 'tId': '...', 'nm': '...', 'k': '...', 'aK': '...', 'r': false, 'u': false, 'typ': '...', 'rgx': '...', 's': '...', 't': '...', 'seq': 15 }'\n```\n\n4. Process File\n```\nHTTP POST /v1/templates/:templateId/process-file\n  -d [file array as `files` (Max 2)]\n```\n\n5. Map Data\n```\nHTTP POST /v1/templates/:templateId/map-data\n  -d [Response `payload` from \"Step 4\"]\n```\n\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\u003c!-- CONTRIBUTING --\u003e\n## Contributing\n\nContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.\n\nIf you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag \"enhancement\".\nDon't forget to give the project a star! Thanks again!\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\u003c!-- LICENSE --\u003e\n## License\n\nDistributed under the MIT License. See `LICENSE.txt` for more information.\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\u003c!-- CONTACT --\u003e\n## Contact\n\nKnovator Technologies\n- Twitter [@knovator](https://twitter.com/knovator)\n- Web [https://knovator.com/](https://knovator.com/)\n\nProject Link: [https://github.com/knovator/data-import](https://github.com/knovator/data-import)\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknovator%2Fdata-import","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fknovator%2Fdata-import","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknovator%2Fdata-import/lists"}