{"id":31674979,"url":"https://github.com/appbaseio/importer","last_synced_at":"2026-05-15T01:37:10.992Z","repository":{"id":313559808,"uuid":"1051550399","full_name":"appbaseio/importer","owner":"appbaseio","description":"Brower GUI data importer for OpenSearch and Elasticsearch clusters. Supports uploading CSV, JSON, and NDJSON files.","archived":false,"fork":false,"pushed_at":"2025-11-27T08:23:47.000Z","size":644,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-06T14:22:36.507Z","etag":null,"topics":["data-import","elasticsearch","gui","opensearch"],"latest_commit_sha":null,"homepage":"https://importer.reactivesearch.io","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/appbaseio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-06T08:20:14.000Z","updated_at":"2025-11-27T08:23:50.000Z","dependencies_parsed_at":"2025-09-06T22:44:01.002Z","dependency_job_id":"c6a7e961-ac2c-4bea-b354-6c0407688858","html_url":"https://github.com/appbaseio/importer","commit_stats":null,"previous_names":["appbaseio/importer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/appbaseio/importer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appbaseio%2Fimporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appbaseio%2Fimporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appbaseio%2Fimporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appbaseio%2Fimporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/appbaseio","download_url":"https://codeload.github.com/appbaseio/importer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appbaseio%2Fimporter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33050427,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-14T02:00:06.663Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-import","elasticsearch","gui","opensearch"],"created_at":"2025-10-08T04:58:27.396Z","updated_at":"2026-05-15T01:37:10.980Z","avatar_url":"https://github.com/appbaseio.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Importer\n\nA browser‑based importer UI to index data into Elasticsearch / OpenSearch.\n\n## Highlights\n\n- React 18 + Vite + TypeScript + Tailwind UI\n- Drag \u0026 drop upload with auto format detection (CSV, JSON array, NDJSON)\n- 100 MB file size guard and binary/corruption sniffing\n- Preview table (first rows) with id key/column detection and warning\n- Stepper flow with strict gating and clear statuses\n  - Upload → Cluster → Ingestion → Import\n- Cluster connect with persisted URL, Basic auth support, and HTTP status badge\n- Ingestion: validate/create index, document count, inline JSON settings editor, optional ingest pipeline\n- Import: concurrent \\_bulk with backoff, accurate Sent/OK/Failed counts, per‑batch logs\n- State preserved across steps; import state preserved unless inputs change\n- One‑click “View data in Dejavu” after import, using your cluster URL (with auth)\n\n## Requirements\n\n- Node.js 18+ (recommended)\n- pnpm (recommended) or npm/yarn\n\n## Quick start\n\nUsing pnpm:\n\n```bash\npnpm install\npnpm dev\n```\n\nUsing npm:\n\n```bash\nnpm install\nnpm run dev\n```\n\nBuild \u0026 preview:\n\n```bash\npnpm build\npnpm preview\n```\n\n## Usage\n\n1. Upload\n\n- Drag \u0026 drop a file or click to browse. Supported: CSV, JSON array, NDJSON (≤ 100 MB).\n- A preview renders the first rows. If no id key/column is found, you’ll see a warning about potential duplicates.\n- Optional: use the “Add sample dataset of 18,000 movies” button to try a bundled NDJSON file.\n\n![](https://i.postimg.cc/htX8DCFk/Screenshot-2025-09-06-at-1-39-13-PM.png)\n\n2. Cluster\n\n- Enter your Elasticsearch/OpenSearch URL. If Basic auth is needed, you can include it when prompted; the app persists the auth header.\n- The UI shows the HTTP status and product info if reachable.\n\n![](https://i.postimg.cc/yY4WgvZq/Screenshot-2025-09-06-at-1-57-37-PM.png)\n\n3. Ingestion\n\n- Enter an index name. Validate will check existence or create a new index with an “all‑as‑string” mapping template if it doesn’t exist.\n- JSON settings editor (tree/text) lets you edit index settings.\n- If you have an ingestion pipeline configured, you can configure its id. This is optional.\n\n![](https://i.postimg.cc/0NmpmbyZ/Screenshot-2025-09-06-at-1-45-24-PM.png)\n\n4. Import\n\n- Import starts automatically when you enter this step (from Ingestion) or via the button.\n- Progress shows Sent, OK, Failed and a bar based on total rows from parsing.\n- Logs show per‑batch stats only. Failures include a sample payload list.\n- When done, use “View data in Dejavu” to browse your index data.\n\n![](https://i.postimg.cc/LX3LCwf1/Screenshot-2025-09-06-at-1-48-41-PM.png)\n\n## Data formats\n\n- CSV: First row as headers. Each row becomes a document. If a column named `_id` or `id` exists, it’s used as the document id.\n- JSON array: A top‑level array of objects.\n- NDJSON: One JSON object per line.\n\n## Authentication\n\n- Basic auth is captured during cluster connect and reused for index/settings/bulk operations.\n- The Dejavu link includes user:password@host in the URL when Basic auth is present (be mindful of URL exposure in history/address bar).\n\n## Accuracy \u0026 logging\n\n- Sent increments as soon as a batch is dispatched.\n- OK/Failed update when the bulk response returns; partial successes within a batch are accounted for per item.\n- Batch numbers increment on completion (1..N), even with concurrency/retries.\n\n## State \u0026 resets\n\n- Ingestion screen state is cached per (cluster URL + index) and restored when revisiting.\n- Import state (progress/logs/workers) is preserved only if the run “signature” (cluster + index + file name/size + settings baseline) is unchanged. Any change resets the import state.\n\n## Troubleshooting\n\n- CORS or network errors during connect may hide the exact HTTP code; consider a dev proxy if needed.\n- Very large files can be memory‑intensive in the browser; keep to ≤ 100 MB as enforced.\n- If logs double or batches repeat, ensure only one tab/instance is running; the app reuses workers and guards duplicate starts.\n\n## Scripts\n\n- `dev` – start Vite dev server\n- `build` – production build\n- `preview` – preview the production build locally\n\n## Embedding in your dashboard\n\nInstall the package and render the Importer component in your React app:\n\n```bash\npnpm add @appbaseio/importer\n# or\nnpm install @appbaseio/importer\n```\n\n```tsx\nimport { Importer } from \"@appbaseio/importer\";\n\nexport default function DataImportPage() {\n  return (\n    \u003cdiv style={{ padding: 16 }}\u003e\n      \u003cImporter\n        config={{\n          sampleDataset: {\n            url: \"/samples/movies.json\", // any JSON/NDJSON/JSON array URL\n            label: \"Load sample movies\",\n            filename: \"movies.json\",\n          },\n        }}\n      /\u003e\n    \u003c/div\u003e\n  );\n}\n```\n\nNotes:\n\n- No props are required. You can optionally provide `config.sampleDataset` to show a custom sample button.\n- The package expects React 18+ in your host app (declared as peer dependency).\n- Styles are bundled from the component; if you tree‑shake CSS, ensure component CSS is included.\n\n## License\n\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fappbaseio%2Fimporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fappbaseio%2Fimporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fappbaseio%2Fimporter/lists"}