{"id":31542588,"url":"https://github.com/baikho/drupal-migrate_batch","last_synced_at":"2026-05-15T22:32:12.228Z","repository":{"id":316956934,"uuid":"1064864318","full_name":"baikho/drupal-migrate_batch","owner":"baikho","description":"A Drupal module that provides batch migration processing with automatic offset tracking.","archived":false,"fork":false,"pushed_at":"2025-10-03T15:15:53.000Z","size":34,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"1.x","last_synced_at":"2026-01-15T15:44:43.607Z","etag":null,"topics":["drupal","drupal-10","drupal-10-module","drupal-migrate","drupal-migration","drupal-module","drupal-project","drush-commands","import","importer"],"latest_commit_sha":null,"homepage":"https://www.drupal.org/project/migrate_batch","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/baikho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-26T17:10:08.000Z","updated_at":"2025-10-03T15:12:40.000Z","dependencies_parsed_at":"2025-09-27T21:21:24.482Z","dependency_job_id":"c585594d-c10d-4edd-8362-b7da91dab976","html_url":"https://github.com/baikho/drupal-migrate_batch","commit_stats":null,"previous_names":["baikho/drupal-migrate_batch"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/baikho/drupal-migrate_batch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baikho%2Fdrupal-migrate_batch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baikho%2Fdrupal-migrate_batch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baikho%2Fdrupal-migrate_batch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baikho%2Fdrupal-migrate_batch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/baikho","download_url":"https://codeload.github.com/baikho/drupal-migrate_batch/tar.gz/refs/heads/1.x","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baikho%2Fdrupal-migrate_batch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33082002,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T20:25:35.270Z","status":"ssl_error","status_checked_at":"2026-05-15T20:25:34.732Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["drupal","drupal-10","drupal-10-module","drupal-migrate","drupal-migration","drupal-module","drupal-project","drush-commands","import","importer"],"created_at":"2025-10-04T11:59:22.641Z","updated_at":"2026-05-15T22:32:12.213Z","avatar_url":"https://github.com/baikho.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Migrate Batch\n\nA Drupal module that provides batch migration processing with automatic offset tracking.\n\n## Overview\n\nThis module extends Drupal's migration system by providing commands to process migration items in configurable batches with automatic progress tracking. Unlike standard `drush migrate:import --limit`, these commands maintain state between runs and can cycle through all source items continuously.\n\n## Installation\n\n### Standard Drupal Installation\n\n1. Download and install the module from [Drupal.org](https://www.drupal.org/project/migrate_batch)\n2. Enable the module: `drush en migrate_batch`\n\n**Requirements:**\n- **PHP 8.1+**\n- **Drupal 10+** with the `migrate` module enabled\n- **Drush 11+** (optional, required only for command-line usage - uses modern attribute-based commands)\n\n## Usage\n\n### Programmatic Usage\n\nYou can use the `migrate_batch` service directly in your custom modules, hooks, or other Drupal code:\n\n```php\n/** @var \\Drupal\\migrate_batch\\Service\\MigrateBatchService $batch */\n$batch = \\Drupal::service('migrate_batch');\n\n// Process a batch of default item amount (20).\n$batch-\u003enext('my_migration');\n\n// Process next batch of 50 items.\n$batch-\u003enext('my_migration', 50);\n\n// Process 50 items starting from offset 100.\n$batch-\u003enext('my_migration', 50, 100);\n\n// Check current offset.\n$offset = $batch-\u003egetOffset('my_migration');\necho \"Current offset: $offset\";\n\n// Set offset to a specific value.\n$batch-\u003esetOffset('my_migration', 100);\n\n// Reset offset back to 0.\n$batch-\u003eresetOffset('my_migration');\n```\n\n**Note:** The service automatically tracks progress using Drupal's State API. Each call to `next()` processes the next batch and advances the offset.\n\n#### Service API Reference\n\n- **`next(string $migrationId, ?int $limit = NULL, ?int $offset = NULL): void`**\n  Processes the next batch of items for the specified migration. If offset is provided, starts from that offset instead of the stored offset.\n\n- **`getOffset(string $migrationId): int`**\n  Returns the current offset for a migration.\n\n- **`setOffset(string $migrationId, int $offset): MigrateBatchService`**\n  Sets the offset for a migration to a specific value.\n\n- **`resetOffset(string $migrationId): MigrateBatchService`**\n  Resets the offset for a migration back to 0.\n\n- **`getDefaultLimit(): int`**\n  Returns the default batch size limit.\n\n#### Usage Examples\n\n**In a custom module:**\n```php\n/**\n * Implements hook_cron().\n */\n#[Hook('cron')]\npublic function cron(): void {\n  /** @var \\Drupal\\migrate_batch\\Service\\MigrateBatchService $batch */\n  $batch = \\Drupal::service('migrate_batch');\n  // Process 100 items per cron run.\n  $batch-\u003enext('my_custom_migration', 100);\n}\n```\n\n**In a controller or form submit:**\n```php\npublic function processBatch(): void {\n  /** @var \\Drupal\\migrate_batch\\Service\\MigrateBatchService $batch */\n  $batch = \\Drupal::service('migrate_batch');\n  $batch-\u003enext('user_import', 50);\n  \\Drupal::messenger()-\u003eaddMessage('Processed next 50 users.');\n}\n```\n\n### Admin Interface\n\nThe module provides an admin interface at **Administration → Content → Migrate Batch States** (`/admin/content/migrate/batch-states`) where you can:\n\n- **View all migrations** organized by group with their current batch offsets\n- **Reset batch offsets** back to 0 for any migration\n- **Run individual batches** directly from the UI (50 items per batch)\n- **Monitor batch progress** through offset tracking\n\n**Note:** Access to this interface requires the `administer migrate batch states` permission. The admin interface is optimized for performance and only loads migration status when needed, making it fast even with many migrations.\n\n### Drush Commands\n\n**Note:** These commands require Drush 11+ due to the use of modern PHP 8.1+ attributes for command definition. The core functionality works without Drush by calling the service directly.\n\n#### Basic Batch Processing\n\nProcess items in batches with automatic offset tracking:\n\n```bash\n# Process the next default amount of items, automatically tracking progress\ndrush migrate:batch-next my_migration\n\n# Run again to process the next 50 items in sequence\ndrush migrate:batch-next my_migration --limit=50\n\n# Process 500 items starting from the current offset\ndrush migrate:batch-next my_migration --limit=500\n\n# Process 500 items starting from offset 20\ndrush migrate:batch-next my_migration --limit=500 --offset=20\n```\n\n### Manual Offset Control\n\nOverride the stored offset when needed by resetting and running multiple batches:\n\n```bash\n# Reset offset to 0, then process first 25 items\ndrush migrate:batch-offset:reset my_migration\ndrush migrate:batch-next my_migration --limit=25\n\n# Process next 25 items (items 25-50)\ndrush migrate:batch-next my_migration --limit=25\n```\n\n### Offset Management\n\n```bash\n# Check current batch offset for a migration\ndrush migrate:batch-offset my_migration\n\n# Set batch offset to a specific value\ndrush migrate:batch-offset:set my_migration 100\n\n# Reset batch offset back to 0\ndrush migrate:batch-offset:reset my_migration\n```\n\n## Commands\n\n### `migrate:batch-next` (alias: `mbn`)\n\nMain command for processing the next batch of migration items.\n\n### `migrate:batch-offset` (alias: `mbo`)\n\nCheck the current batch offset for a migration.\n\n### `migrate:batch-offset:set` (alias: `mbos`)\n\nSet the batch offset for a migration to a specific value.\n\n### `migrate:batch-offset:reset` (alias: `mbor`)\n\nReset the batch offset for a migration back to 0.\n\n## How It Works\n\n1. **State Tracking**: Uses Drupal's State API to store the current offset for each migration\n2. **Direct API Execution**: Uses `MigrateExecutable` directly (like Drush's `migrate:import`) instead of shell execution for better performance\n3. **Source Limiting**: Passes LIMIT/OFFSET directly to the migration source configuration\n4. **Automatic Advancement**: After successful processing, automatically increments the offset\n5. **Cyclic Behavior**: When reaching the end of available items, wraps back to offset 0\n\n## Technical Details\n\n### State Storage\n\nProgress is stored using Drupal's State API with keys in the format:\n`migrate_batch.offset.{migration_id}`\n\n### Offset Calculation\n\n- After successful processing: `new_offset = (current_offset + limit) % total_items`\n- When no items found at current offset: Automatically resets to 0\n\n## Integration\n\nThis module works with any Drupal migration. For optimal performance with large datasets, source plugins should use the `BatchableSourceTrait` to support batch processing:\n\n### Source Plugin Integration\n\nThe `BatchableSourceTrait` provides three key methods for batch processing:\n\n- **`isBatchRequest()`**: Returns TRUE when the migration is running in batch mode\n- **`getBatchLimit()`**: Returns the number of items to process in this batch\n- **`getBatchOffset()`**: Returns the starting position for this batch\n\nUse these methods in your source plugin's `initializeIterator()` method to apply LIMIT and OFFSET to your data retrieval:\n\n```php\nuse Drupal\\migrate_batch\\Traits\\BatchableSourceTrait;\n\nclass MySourcePlugin extends SomeBaseClass {\n\n  use BatchableSourceTrait;\n\n  /**\n   * {@inheritDoc}\n   */\n  protected function initializeIterator(): DataParserPluginInterface {\n    // Apply batch parameters to your data source here\n    if ($this-\u003eisBatchRequest()) {\n      // Modify your data source to use batch limit and offset\n      $limit = $this-\u003egetBatchLimit();\n      $offset = $this-\u003egetBatchOffset();\n\n      // Apply to your specific data source (API, database, files, etc.)\n      $this-\u003eapplyBatchParameters($limit, $offset);\n    }\n\n    return parent::initializeIterator();\n  }\n\n  /**\n   * Apply batch parameters to your specific data source.\n   */\n  protected function applyBatchParameters(?int $limit, int $offset): void {\n    // Implementation depends on your data source type\n    // Examples:\n    // - SQL: Add LIMIT/OFFSET to query\n    // - API: Add to request parameters\n    // - Files: Slice the file list\n  }\n}\n```\n\n### SQL-Based Sources\n\nFor Drupal's `SqlBase` source plugins, override the `query()` method to apply batch parameters:\n\n```php\nuse Drupal\\migrate_batch\\Traits\\BatchableSourceTrait;\n\nclass MySqlSource extends SqlBase {\n\n  use BatchableSourceTrait;\n\n  public function query() {\n    $query = parent::query();\n\n    // Apply batch parameters using Drupal's query range method\n    // range($offset, $limit) automatically handles LIMIT and OFFSET\n    if ($this-\u003eisBatchRequest()) {\n      if ($limit = $this-\u003egetBatchLimit()) {\n        $query-\u003erange($this-\u003egetBatchOffset(), $limit);\n      }\n    }\n\n    return $query;\n  }\n}\n```\n\nFor other source types, apply batch parameters in the appropriate method where your data source is initialized or queried.\n\n### Why Use the Trait?\n\n**Performance Benefits:**\n- Large datasets are processed in manageable chunks\n- Reduces memory usage and execution time\n- Allows resuming interrupted migrations\n\n**Flexibility:**\n- Batch parameters are applied at runtime, not stored in config\n- Works with any data source (SQL, APIs, files, etc.)\n- Compatible with existing migration configurations\n\n**Integration:**\n- The trait works seamlessly with the `MigrateBatchService`\n- No changes needed to migration YAML files\n- Maintains full compatibility with standard Drupal migrations\n\n## Permissions\n\nThe module defines the following permission:\n\n- **`administer migrate batch`** - Access the migrate batch states admin interface and manage batch processing operations. This permission is restricted and should only be granted to trusted administrators.\n\n## Compatibility\n\n- **Drupal**: \u003e=10.x\n- **PHP**: 8.1+\n- **Drush**: 11+ (optional, enhances with CLI commands)\n- **Migration Framework**: Any Drupal migration using the core migrate API\n\n## Maintainers\n\n* Sang Lostrie (baikho) - https://www.drupal.org/u/baikho\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaikho%2Fdrupal-migrate_batch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbaikho%2Fdrupal-migrate_batch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaikho%2Fdrupal-migrate_batch/lists"}