{"id":50986792,"url":"https://github.com/kubeservice-stack/hf-sync-action","last_synced_at":"2026-06-19T20:04:03.621Z","repository":{"id":361085497,"uuid":"1252970671","full_name":"kubeservice-stack/hf-sync-action","owner":"kubeservice-stack","description":"Realize bidirectional synchronization of models and datasets across Hugging Face and ModelScope, with schedule tasks and auto sync capabilities.","archived":false,"fork":false,"pushed_at":"2026-05-29T05:50:43.000Z","size":102,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-29T07:21:13.068Z","etag":null,"topics":["github-actions","hf-ms-transfer","tools"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kubeservice-stack.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-29T03:36:19.000Z","updated_at":"2026-05-29T05:52:53.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kubeservice-stack/hf-sync-action","commit_stats":null,"previous_names":["kubeservice-stack/hf-sync-action"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/kubeservice-stack/hf-sync-action","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeservice-stack%2Fhf-sync-action","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeservice-stack%2Fhf-sync-action/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeservice-stack%2Fhf-sync-action/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeservice-stack%2Fhf-sync-action/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kubeservice-stack","download_url":"https://codeload.github.com/kubeservice-stack/hf-sync-action/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeservice-stack%2Fhf-sync-action/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34546207,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-19T02:00:06.005Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["github-actions","hf-ms-transfer","tools"],"created_at":"2026-06-19T20:04:02.692Z","updated_at":"2026-06-19T20:04:03.610Z","avatar_url":"https://github.com/kubeservice-stack.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ch1 align=\"center\"\u003eHF-MS Sync\u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\n    \u003cstrong\u003eBidirectional sync of AI models and datasets between HuggingFace and ModelScope\u003c/strong\u003e\n  \u003c/p\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/marketplace/actions/hf-ms-sync\"\u003e\n      \u003cimg src=\"https://img.shields.io/badge/GitHub%20Marketplace-Available-blue?style=flat-square\" alt=\"Marketplace\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/kubeservice-stack/hf-sync-action/actions\"\u003e\n      \u003cimg src=\"https://img.shields.io/github/actions/workflow/status/kubeservice-stack/hf-sync-action/sync.yml?style=flat-square\" alt=\"CI\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/kubeservice-stack/hf-sync-action/blob/master/LICENSE\"\u003e\n      \u003cimg src=\"https://img.shields.io/github/license/kubeservice-stack/hf-sync-action?style=flat-square\" alt=\"License\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/kubeservice-stack/hf-sync-action/releases\"\u003e\n      \u003cimg src=\"https://img.shields.io/github/v/release/kubeservice-stack/hf-sync-action?style=flat-square\" alt=\"Release\"\u003e\n    \u003c/a\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\n---\n\n## Overview\n\n**HF-MS Sync** is a GitHub Action that automatically synchronizes AI models and datasets between [HuggingFace Hub](https://huggingface.co) and [ModelScope](https://modelscope.cn). It provides a flexible, configuration-driven approach to keeping your model and dataset repositories in sync across both platforms.\n\n### Key Features\n\n- **Bidirectional Sync** - Sync from HuggingFace to ModelScope, ModelScope to HuggingFace, or both directions\n- **Model \u0026 Dataset Support** - Sync any type of repository: models, datasets, and more\n- **Smart Change Detection** - Only transfers files that have actually changed, saving bandwidth and time\n- **Conflict Resolution** - Configurable strategies: newer-wins, platform priority, or skip\n- **Multiple Triggers** - Scheduled (cron), manual (workflow_dispatch), and webhook (repository_dispatch)\n- **Dry Run Mode** - Preview what would be synced without actually transferring files\n- **Large File Support** - Git LFS compatible, handles multi-GB model weights efficiently\n- **State Persistence** - Tracks sync history across runs via GitHub Actions artifacts\n- **GitHub Job Summary** - Generates detailed Markdown reports in your workflow run summary\n\n## Quick Start\n\n### 1. Create Configuration\n\nCreate `config/sync_config.yaml` in your repository:\n\n```yaml\nglobal:\n  sync_direction: \"hf_to_ms\"       # hf_to_ms | ms_to_hf | bidirectional\n  conflict_strategy: \"newer_wins\"  # newer_wins | hf_priority | ms_priority | skip\n  max_file_size_gb: 50\n  max_parallel_downloads: 4\n\nmodels:\n  - name: \"qwen2.5-7b\"\n    hf_repo_id: \"Qwen/Qwen2.5-7B-Instruct\"\n    ms_repo_id: \"Qwen/Qwen2.5-7B-Instruct\"\n    direction: \"hf_to_ms\"\n    include_patterns:\n      - \"*.safetensors\"\n      - \"*.json\"\n      - \"tokenizer*\"\n    exclude_patterns:\n      - \"*.msgpack\"\n    enabled: true\n\ndatasets:\n  - name: \"my-dataset\"\n    hf_repo_id: \"my-org/my-dataset\"\n    ms_repo_id: \"my-org/my-dataset\"\n    direction: \"hf_to_ms\"\n    enabled: true\n```\n\n### 2. Set Up Secrets\n\nAdd these secrets to your GitHub repository (**Settings \u003e Secrets and variables \u003e Actions**):\n\n| Secret | Description |\n|--------|-------------|\n| `HF_TOKEN` | HuggingFace API token with write access |\n| `MODELSCOPE_TOKEN` | ModelScope API token with write access |\n\n### 3. Create Workflow\n\nCreate `.github/workflows/sync.yml`:\n\n```yaml\nname: Sync HF \u003c-\u003e ModelScope\n\non:\n  schedule:\n    - cron: '0 */6 * * *'    # Every 6 hours\n  workflow_dispatch:\n    inputs:\n      sync_target:\n        description: 'Specific item to sync (empty = all)'\n        required: false\n      direction:\n        description: 'Override direction'\n        type: choice\n        options: [config, hf_to_ms, ms_to_hf, bidirectional]\n      dry_run:\n        description: 'Dry run'\n        type: boolean\n        default: false\n\njobs:\n  sync:\n    runs-on: ubuntu-latest\n    timeout-minutes: 360\n    steps:\n      - uses: actions/checkout@v4\n\n      - uses: kubeservice-stack/hf-sync-action@v1\n        with:\n          config: config/sync_config.yaml\n          direction: ${{ inputs.direction || 'config' }}\n          dry_run: ${{ inputs.dry_run || 'false' }}\n          target: ${{ inputs.sync_target || '' }}\n        env:\n          HF_TOKEN: ${{ secrets.HF_TOKEN }}\n          MODELSCOPE_TOKEN: ${{ secrets.MODELSCOPE_TOKEN }}\n```\n\n## Usage\n\n### As a GitHub Action (Recommended)\n\n```yaml\n- uses: kubeservice-stack/hf-sync-action@v1\n  with:\n    config: 'config/sync_config.yaml'\n    direction: 'hf_to_ms'\n    dry_run: 'false'\n    target: ''\n    log_level: 'INFO'\n  env:\n    HF_TOKEN: ${{ secrets.HF_TOKEN }}\n    MODELSCOPE_TOKEN: ${{ secrets.MODELSCOPE_TOKEN }}\n```\n\n### Standalone (Python CLI)\n\n```bash\n# Install dependencies\npip install -r requirements.txt\n\n# Run sync\nHF_TOKEN=hf_xxx MODELSCOPE_TOKEN=ms_xxx \\\n  python -m src.sync_engine \\\n    --config config/sync_config.yaml \\\n    --state-dir .sync_state/\n\n# Dry run\npython -m src.sync_engine --config config/sync_config.yaml --dry-run true\n```\n\n### Webhook Trigger\n\nTrigger sync from an external service:\n\n```bash\ncurl -X POST \\\n  -H \"Accept: application/vnd.github+json\" \\\n  -H \"Authorization: Bearer $GITHUB_TOKEN\" \\\n  https://api.github.com/repos/{owner}/{repo}/dispatches \\\n  -d '{\n    \"event_type\": \"sync-trigger\",\n    \"client_payload\": {\n      \"target\": \"qwen2.5-7b\",\n      \"direction\": \"hf_to_ms\",\n      \"platform\": \"hf\"\n    }\n  }'\n```\n\n## Configuration Reference\n\n### Global Settings\n\n| Setting | Type | Default | Description |\n|---------|------|---------|-------------|\n| `sync_direction` | string | `bidirectional` | Default sync direction (`hf_to_ms`, `ms_to_hf`, `bidirectional`) |\n| `conflict_strategy` | string | `newer_wins` | How to resolve conflicts (`newer_wins`, `hf_priority`, `ms_priority`, `skip`) |\n| `max_file_size_gb` | float | `50` | Skip files larger than this size |\n| `retry_attempts` | int | `3` | Number of retry attempts for failed transfers |\n| `retry_delay_seconds` | int | `30` | Delay between retries |\n| `max_parallel_downloads` | int | `4` | Maximum concurrent downloads |\n| `max_parallel_uploads` | int | `2` | Maximum concurrent uploads |\n| `delete_orphaned` | bool | `false` | Delete files on target that don't exist on source |\n\n### Item Settings (per model/dataset)\n\n| Setting | Type | Default | Description |\n|---------|------|---------|-------------|\n| `name` | string | *required* | Unique identifier for this sync item |\n| `hf_repo_id` | string | *required* | HuggingFace repository ID (e.g., `org/model-name`) |\n| `ms_repo_id` | string | *required* | ModelScope repository ID |\n| `direction` | string | global | Override global direction for this item |\n| `include_patterns` | list | `[\"*\"]` | Glob patterns for files to include |\n| `exclude_patterns` | list | `[]` | Glob patterns for files to exclude |\n| `enabled` | bool | `true` | Enable/disable this sync item |\n\n### Conflict Resolution\n\n| Strategy | Behavior |\n|----------|----------|\n| `newer_wins` | The file with the more recent modification timestamp wins |\n| `hf_priority` | HuggingFace version always wins on conflict |\n| `ms_priority` | ModelScope version always wins on conflict |\n| `skip` | Skip conflicting files (keep both versions as-is) |\n\n## Action Inputs\n\n| Input | Required | Default | Description |\n|-------|----------|---------|-------------|\n| `config` | No | `config/sync_config.yaml` | Path to configuration file |\n| `direction` | No | `config` | Override sync direction |\n| `dry_run` | No | `false` | Preview changes without transferring |\n| `target` | No | *(all)* | Sync only a specific item by name |\n| `log_level` | No | `INFO` | Logging verbosity |\n\n## Architecture\n\n```\nTriggers (cron / manual / webhook)\n         │\n         ▼\n   Sync Engine (Python)\n   ├── Config Loader      ← YAML configuration\n   ├── Change Detector    ← Compares file snapshots between platforms\n   └── Transfer Executor  ← Downloads from source, uploads to target\n         │\n    ┌────┴────┐\n    ▼         ▼\nHF Adapter  MS Adapter    ← Platform-specific API wrappers\n    │         │\n    ▼         ▼\nHuggingFace  ModelScope   ← Remote platforms\n```\n\n### How It Works\n\n1. **Snapshot** - Fetches file lists and metadata from both platforms\n2. **Detect** - Compares snapshots to identify new, updated, and deleted files\n3. **Transfer** - Downloads changed files from source and uploads to target\n4. **State** - Persists sync state as GitHub Actions artifacts for next run\n5. **Report** - Generates a Markdown summary in the GitHub Actions run\n\n## Examples\n\nThe [`examples/`](examples/) directory contains ready-to-use configurations and workflows for common scenarios:\n\n### Configuration Examples\n\n| Example | Scenario |\n|---------|----------|\n| [01-hf-to-ms-mirror.yaml](examples/01-hf-to-ms-mirror.yaml) | One-way HF -\u003e MS mirror for popular models |\n| [02-bidirectional-sync.yaml](examples/02-bidirectional-sync.yaml) | Bidirectional sync with conflict resolution |\n| [03-multi-model-batch.yaml](examples/03-multi-model-batch.yaml) | Batch sync many models at once |\n| [04-dataset-sync.yaml](examples/04-dataset-sync.yaml) | Dataset sync (parquet, jsonl, images) |\n| [05-minimal-external.yaml](examples/05-minimal-external.yaml) | Minimal config for external projects |\n| [06-selective-patterns.yaml](examples/06-selective-patterns.yaml) | Selective file patterns (inference-ready only) |\n\n### Workflow Examples\n\n| Example | Scenario |\n|---------|----------|\n| [simple-mirror.yml](examples/workflows/simple-mirror.yml) | Scheduled one-way sync with manual trigger |\n| [bidirectional-with-notify.yml](examples/workflows/bidirectional-with-notify.yml) | Bidirectional + DingTalk/Feishu notifications |\n| [webhook-triggered.yml](examples/workflows/webhook-triggered.yml) | Event-driven sync via webhook |\n| [matrix-sync.yml](examples/workflows/matrix-sync.yml) | Parallel matrix sync (one job per model) |\n\n### Using in Your Own Project\n\n```bash\n# 1. Copy config\ncp examples/05-minimal-external.yaml config/sync_config.yaml\n# Edit with your repo IDs\n\n# 2. Copy workflow\ncp examples/workflows/simple-mirror.yml .github/workflows/sync.yml\n\n# 3. Add secrets HF_TOKEN and MODELSCOPE_TOKEN in GitHub Settings\n\n# 4. Push\ngit add config/ .github/workflows/sync.yml\ngit commit -m \"Add HF-MS sync\"\ngit push\n```\n\nSee [examples/README.md](examples/README.md) for full details.\n\n## E2E Testing\n\nThe project includes an end-to-end test workflow that performs a real sync with a tiny model and verifies file consistency.\n\n### Run E2E Test\n\n1. Go to **Actions** \u003e **E2E Test - Bidirectional Sync**\n2. Click **Run workflow**\n3. Fill in:\n   - **HF Repo**: `sshleifer/tiny-gpt2` (default, ~60MB)\n   - **MS Repo**: your test repo ID (e.g., `your-org/e2e-test`)\n   - **Direction**: `hf_to_ms`, `ms_to_hf`, or `bidirectional`\n   - **Cleanup**: `true` to delete the MS test repo after verification\n\n### What the E2E Test Validates\n\n| Stage | Check |\n|-------|-------|\n| Unit tests | All 48 unit tests pass |\n| Sync execution | Real file transfer between HF and MS |\n| File verification | File lists match on both platforms |\n| Idempotency | Second sync detects zero changes |\n| Cleanup | Optionally deletes test repo |\n\n### PR Validation\n\nOn pull requests, the E2E workflow also runs a lightweight validation (no secrets needed):\n- Config parsing and validation\n- Module import checks for all adapters\n\n## Development\n\n### Setup\n\n```bash\ngit clone https://github.com/kubeservice-stack/hf-sync-action.git\ncd hf-sync-action\npython -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -e \".[dev]\"\n```\n\n### Test\n\n```bash\npytest tests/ -v\n```\n\n### Project Structure\n\n```\n├── .github/workflows/     # CI/CD workflows (sync, e2e-test, webhook)\n├── src/\n│   ├── adapters/          # Platform adapters (HF, MS)\n│   ├── config.py          # Configuration loading\n│   ├── change_detector.py # Change detection logic\n│   ├── sync_engine.py     # Main sync orchestrator\n│   ├── models.py          # Data models\n│   ├── report.py          # Report generation\n│   └── utils.py           # Utilities\n├── config/                # Default sync configuration\n├── examples/              # Ready-to-use configs and workflows\n│   ├── workflows/         # Example GitHub Action workflows\n│   └── *.yaml             # Scenario-based config examples\n├── tests/                 # Test suite\n│   ├── e2e/               # End-to-end test scripts\n│   └── test_*.py          # Unit tests\n├── action.yml             # GitHub Action definition\n├── Dockerfile             # Action runtime\n└── pyproject.toml         # Python project config\n```\n\n## FAQ\n\n**Q: How long does syncing take?**\nA: Depends on the number and size of files. Initial sync of a 7B model (~14GB) typically takes 20-40 minutes. Subsequent syncs only transfer changed files.\n\n**Q: Does it support private repositories?**\nA: Yes, as long as your API tokens have the necessary permissions for both platforms.\n\n**Q: What happens if a sync is interrupted?**\nA: The sync state is persisted after each successful run. On the next run, it will only transfer files that have changed since the last successful sync.\n\n**Q: Can I sync only specific file types?**\nA: Yes, use `include_patterns` and `exclude_patterns` in your configuration. For example, sync only `*.safetensors` files.\n\n**Q: What if both platforms have different versions of a file?**\nA: The `conflict_strategy` setting determines the behavior. `newer_wins` uses timestamps, or you can set platform priority.\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubeservice-stack%2Fhf-sync-action","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkubeservice-stack%2Fhf-sync-action","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubeservice-stack%2Fhf-sync-action/lists"}