{"id":45150876,"url":"https://github.com/seryllns/data_porter","last_synced_at":"2026-02-20T03:01:00.563Z","repository":{"id":336937094,"uuid":"1151741430","full_name":"SerylLns/data_porter","owner":"SerylLns","description":"Mountable Rails engine for CSV, XLSX, JSON \u0026 API data imports. Declarative DSL, live preview, dry run, real-time progress via ActionCable.","archived":false,"fork":false,"pushed_at":"2026-02-19T09:05:03.000Z","size":8382,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-19T13:54:28.545Z","etag":null,"topics":["activerecord","api","csv","data-import","engine","gem","json","rails","ruby","rubyonrails","spreadsheet","xlsx"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SerylLns.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-06T20:57:54.000Z","updated_at":"2026-02-19T09:05:06.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/SerylLns/data_porter","commit_stats":null,"previous_names":["seryllns/data_porter"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/SerylLns/data_porter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SerylLns%2Fdata_porter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SerylLns%2Fdata_porter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SerylLns%2Fdata_porter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SerylLns%2Fdata_porter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SerylLns","download_url":"https://codeload.github.com/SerylLns/data_porter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SerylLns%2Fdata_porter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29639808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T22:32:43.237Z","status":"online","status_checked_at":"2026-02-20T02:00:07.535Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activerecord","api","csv","data-import","engine","gem","json","rails","ruby","rubyonrails","spreadsheet","xlsx"],"created_at":"2026-02-20T03:00:59.745Z","updated_at":"2026-02-20T03:01:00.557Z","avatar_url":"https://github.com/SerylLns.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DataPorter\n\n[![Gem Version](https://badge.fury.io/rb/data_porter.svg?icon=si%3Arubygems)](https://badge.fury.io/rb/data_porter)\n[![Docs](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://seryllns.github.io/data_porter/)\n\nA mountable Rails engine for data import workflows: **Upload**, **Map**, **Preview**, **Import**.\n\nSupports CSV, JSON, XLSX, and API sources with a declarative DSL for defining import targets. Business-agnostic by design -- all domain logic lives in your host app.\n\n![DataPorter demo](docs/screenshots/demo_fast.gif)\n\n## Features\n\n- **4 source types** -- CSV, XLSX, JSON, and API with a unified parsing pipeline\n- **Interactive column mapping** -- Drag-free UI to match file headers to target fields ([docs](docs/MAPPING.md))\n- **Mapping templates** -- Save and reuse column mappings across imports ([docs](docs/MAPPING.md#mapping-templates))\n- **Real-time progress** -- JSON polling with animated progress bar, no ActionCable required\n- **Dry run mode** -- Validate against the database without persisting\n- **Standalone UI** -- Self-contained layout with Turbo Drive and Stimulus, no host app dependencies\n- **Import params** -- Declare extra form fields (select, text, number, hidden) per target for scoped imports ([docs](docs/TARGETS.md#params--))\n- **Per-target source filtering** -- Each target declares its allowed sources, the UI filters accordingly\n- **Import deletion \u0026 auto-purge** -- Delete imports from the UI, or schedule `rake data_porter:purge` for automatic cleanup\n- **Reject rows export** -- Download a CSV of failed/errored records with error messages after import\n- **Scoped imports** -- `config.scope` for multi-tenant isolation; each user only sees their own imports\n- **Security validations** -- File size limit, MIME type check, strong parameter whitelisting, IDOR protection via scope\n- **Safety guards** -- Max records limit (`config.max_records`), configurable transaction mode (`:per_record` or `:all`)\n- **Declarative Target DSL** -- One class per import type, zero boilerplate ([docs](docs/TARGETS.md))\n\n## Requirements\n\n- Ruby \u003e= 3.2\n- Rails \u003e= 7.0\n- ActiveStorage (for file uploads)\n\n## Installation\n\n```bash\nbundle add data_porter\nbin/rails generate data_porter:install\nbin/rails db:migrate\n```\n\nThe generator creates:\n- Migrations for `data_porter_imports` and `data_porter_mapping_templates`\n- An initializer at `config/initializers/data_porter.rb`\n- The `app/importers/` directory\n- Engine mount at `/imports`\n\n## Quick Start\n\nGenerate a target:\n\n```bash\nbin/rails generate data_porter:target Product name:string:required price:integer sku:string --sources csv xlsx\n```\n\nImplement `persist` in `app/importers/product_target.rb`:\n\n```ruby\nclass ProductTarget \u003c DataPorter::Target\n  label \"Product\"\n  model_name \"Product\"\n  icon \"fas fa-file-import\"\n  sources :csv\n\n  columns do\n    column :name,  type: :string, required: true\n    column :price, type: :integer\n    column :sku,   type: :string\n  end\n\n  def persist(record, context:)\n    Product.create!(record.attributes)\n  end\nend\n```\n\nVisit `/imports` and start importing.\n\n## Import Workflow\n\n```\nFile-based (CSV/XLSX):\npending -\u003e extracting_headers -\u003e mapping -\u003e parsing -\u003e previewing -\u003e importing -\u003e completed\n\nNon-file (JSON/API):\npending -\u003e parsing -\u003e previewing -\u003e importing -\u003e completed\n```\n\n| Status | Description |\n|---|---|\n| `pending` | Waiting for processing |\n| `extracting_headers` | Reading file headers for column mapping |\n| `mapping` | Waiting for user to map columns |\n| `parsing` | Records being extracted |\n| `previewing` | Records ready for review |\n| `importing` | Records being persisted |\n| `completed` | All records processed |\n| `failed` | Fatal error encountered |\n| `dry_running` | Dry run validation in progress |\n\n## Documentation\n\n**[Full documentation on GitHub Pages](https://seryllns.github.io/data_porter/)**\n\n| Topic | Description |\n|---|---|\n| [Configuration](docs/CONFIGURATION.md) | All options, authentication, context builder, real-time updates |\n| [Targets](docs/TARGETS.md) | DSL reference, columns, hooks, generator |\n| [Sources](docs/SOURCES.md) | CSV, JSON, XLSX, API setup and examples |\n| [Column Mapping](docs/MAPPING.md) | Interactive mapping, templates, priority order |\n| [Roadmap](docs/ROADMAP.md) | v1.0 plan and progress |\n\n## Routes\n\n| Method | Path | Action |\n|---|---|---|\n| GET | `/imports` | List imports |\n| POST | `/imports` | Create import |\n| GET | `/imports/:id` | Show import |\n| DELETE | `/imports/:id` | Delete import |\n| GET | `/imports/:id/status` | JSON progress polling |\n| PATCH | `/imports/:id/update_mapping` | Save column mapping |\n| POST | `/imports/:id/parse` | Parse source |\n| POST | `/imports/:id/confirm` | Run import |\n| POST | `/imports/:id/cancel` | Cancel import |\n| POST | `/imports/:id/back_to_mapping` | Reset to mapping step |\n| POST | `/imports/:id/dry_run` | Dry run validation |\n| GET | `/imports/:id/export_rejects` | Download rejects CSV |\n| | `/mapping_templates` | Full CRUD for templates |\n\n## Development\n\n```bash\ngit clone https://github.com/SerylLns/data_porter.git\ncd data_porter\nbin/setup\nbundle exec rspec     # 423 specs\nbundle exec rubocop   # 0 offenses\n```\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseryllns%2Fdata_porter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseryllns%2Fdata_porter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseryllns%2Fdata_porter/lists"}