{"id":50126181,"url":"https://github.com/olibomby/cm3p","last_synced_at":"2026-05-23T20:03:24.823Z","repository":{"id":325999265,"uuid":"1035212274","full_name":"OliBomby/CM3P","owner":"OliBomby","description":"CM3P (Contrastive Metadata-Map Masked Pre-training) multi-modal representation learning framework for osu! beatmaps","archived":false,"fork":false,"pushed_at":"2026-01-30T20:45:23.000Z","size":10963,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-01-31T12:35:07.983Z","etag":null,"topics":["beatmaps","classifier-model","contrastive-learning","embedding-models","huggingface-transformers","osugame"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OliBomby.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":"OliBomby","tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"lfx_crowdfunding":null,"polar":null,"buy_me_a_coffee":null,"thanks_dev":null,"custom":null}},"created_at":"2025-08-09T22:31:17.000Z","updated_at":"2026-01-30T20:45:27.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/OliBomby/CM3P","commit_stats":null,"previous_names":["olibomby/cm3p"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OliBomby/CM3P","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OliBomby%2FCM3P","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OliBomby%2FCM3P/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OliBomby%2FCM3P/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OliBomby%2FCM3P/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OliBomby","download_url":"https://codeload.github.com/OliBomby/CM3P/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OliBomby%2FCM3P/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33410346,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-23T18:09:33.147Z","status":"ssl_error","status_checked_at":"2026-05-23T18:09:31.380Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beatmaps","classifier-model","contrastive-learning","embedding-models","huggingface-transformers","osugame"],"created_at":"2026-05-23T20:03:19.251Z","updated_at":"2026-05-23T20:03:24.818Z","avatar_url":"https://github.com/OliBomby.png","language":"Python","funding_links":["https://ko-fi.com/OliBomby"],"categories":[],"sub_categories":[],"readme":"# CM3P: Contrastive Metadata-Map Masked Pre-training\n\nCM3P (Contrastive Metadata-Map Masked Pre-training) is a multi-modal representation learning framework for osu! beatmaps. It learns high-quality embeddings for both beatmap structure (events, timing, positions, hitsounds, scroll speed, etc.) and beatmap metadata (difficulty, year, mapper, tags, mode, etc.), optionally conditioned on audio. These embeddings serve as a foundation for downstream tasks such as beatmap retrieval, recommendation, classification (e.g. ranked vs unranked), masked modeling, and transfer to fine-tuned generative or discriminative models.\n\nCM3P provides:\n- Unified multi-modal processor: parses raw `.osu` files + metadata + audio into token \u0026 feature tensors.\n- Dual-tower ModernBERT encoders (beatmap + metadata) with optional fused audio embeddings via placeholder audio tokens.\n- Contrastive embedding pretraining with structured metadata variations (robust in-batch negatives).\n- Optional masked beatmap language modeling and downstream classification heads.\n- High-quality embeddings for retrieval, recommendation, filtering, and fine-tuning bases.\n- Flexible Hydra configuration \u0026 Hugging Face Trainer integration (freeze/unfreeze, Muon optimizer, WandB \u0026 Hub push).\n- Efficient long sequence handling (Flash Attention 2 support) and mixed precision.\n\n---\n## 1. Quick Start (Inference)\n\nTo use a CM3P model in your project, you can simply load it from [Hugging Face Hub](https://huggingface.co/OliBomby/CM3P) and start extracting embeddings:\n```python\nimport torch\nfrom transformers import AutoProcessor, AutoModel\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nrepo_id = \"OliBomby/CM3P\"\n\nprocessor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True, revision=\"main\")\nmodel = AutoModel.from_pretrained(repo_id, device_map=device, dtype=torch.bfloat16, trust_remote_code=True, revision=\"main\")\n\ninputs = processor(beatmap=\"path/to/beatmap.osu\", audio=\"path/to/audio.mp3\")\ninputs = inputs.to(device, dtype=torch.bfloat16)\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n\nbeatmap_embeds = outputs.beatmap_embeds  # (beatmap_length_seconds / 16, projection_dim)\n```\n---\n\n## 2. Beatmap Embedding Explorer\n\n[The osu! Map Explorer](https://olibomby.github.io/CM3P/) is a static visualizer website that allows you to explore a dataset of beatmap embeddings in 2D space. You can search for beatmaps by title, artist, or mapper, and see the nearest neighbors.\n\nYou can use [this Colab notebook](https://colab.research.google.com/github/OliBomby/CM3P/blob/master/colab/CM3P_embeddings.ipynb) or `extract_beatmap_embeddings.py` to easily make a dataset of embeddings from your own beatmaps for visualization.\nA dataset with a ton of precomputed beatmap embeddings can be found [here on Hugging Face Hub](https://huggingface.co/datasets/OliBomby/CM3P-Embeddings-244K).\n\n\u003cimg width=\"1715\" height=\"1273\" alt=\"image\" src=\"https://github.com/user-attachments/assets/dbbc6354-8fce-4be0-aa17-26294d85d948\" /\u003e\n\n---\n\n## 3. Installation\n### Prerequisites\n- Python 3.12\n- [Git](https://git-scm.com/downloads)\n- [ffmpeg](http://www.ffmpeg.org/)\n- [CUDA](https://developer.nvidia.com/cuda-zone) (For NVIDIA GPUs) or [ROCm](https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html) (For AMD GPUs on linux)\n- [PyTorch](https://pytorch.org/get-started/locally/): Make sure to follow the Get Started guide so you install `torch` and `torchaudio` with GPU support. Select the correct Compute Platform version that you have installed in the previous step.\n- A GPU for efficient training (Flash Attention 2 support recommended). For CPU-only or unsupported GPUs, set `attn_implementation: sdpa`\n\n### Steps\n```bash\n# Clone the repository\ngit clone https://github.com/OliBomby/CM3P.git\ncd CM3P\n\n# (Optional) Create and activate a virtual environment\npython -m venv .venv\n\n# In cmd.exe\n.venv\\Scripts\\activate.bat\n# In PowerShell\n.venv\\Scripts\\Activate.ps1\n# In Linux or MacOS\nsource .venv/bin/activate\n\n# Install dependencies\npip install -r requirements.txt\n\n# (Optional) Install Flash Attention 2 if your GPU supports it (A100/H100/RTX 40xx, etc.)\n# Follow official instructions; otherwise switch attn implementation in config.\n```\n\n### Docker Steps (Optional)\n```bash\n# Clone the repository\ngit clone https://github.com/OliBomby/CM3P.git\ncd CM3P\n\n# Build and run the Docker container\ndocker compose up -d\ndocker attach cm3p_space\ncd cm3p\n```\nThe Docker compose is configured to mount the `datasets` directory next to the `CM3P` directory, so place your datasets there to use them in the container.\n\n---\n## 4. Data Preparation\n\nCreate your own dataset using the [Mapperator console app](https://github.com/mappingtools/Mapperator/blob/master/README.md#create-a-high-quality-dataset). It requires an [osu! OAuth client token](https://osu.ppy.sh/home/account/edit) to verify beatmaps and get additional metadata.\n\n```sh\nMapperator.ConsoleApp.exe dataset2 -t \"/Mapperatorinator/datasets/beatmap_descriptors.csv\" -i \"path/to/osz/files\" -o \"/datasets/cool_dataset\"\n```\n\nWhen training CM3P, you can provide multiple dataset roots (list of paths) in `configs/train/default.yaml` under `dataset.train_dataset_paths` and `dataset.test_dataset_paths`.\n\nFiltering knobs (see `dataset` section in config):\n- Year range (`min_year`, `max_year`)\n- Difficulty range (`min_difficulty`, `max_difficulty`)\n- Gamemodes filter (`gamemodes` list)\n- Splitting via indices (`train_dataset_start`, `train_dataset_end`, etc.)\n\n---\n## 5. Model Architecture\nCM3P consists of three main transformer-based components built on ModernBERT:\n- **Metadata Tower** (`CM3PMetadataTransformer`): Encodes metadata token sequences; pools either CLS token or mean over valid tokens.\n- **Beatmap Tower** (`CM3PBeatmapTransformer`): Encodes beatmap token sequences; internally can fuse audio embeddings by replacing audio placeholder tokens with projected audio features produced by:\n  - **Audio Encoder** (`CM3PAudioEncoder`): Two 1D convolutional layers (inspired by Whisper) + ModernBERT + projection MLP (`CM3PMultiModalProjector`) to reach the same embedding dimensionality as beatmap token embeddings.\n- **Projection Heads**: Linear layers map pooled outputs of both towers into a shared `projection_dim` embedding space.\n\nOptional components:\n- **Masked LM Head** (`CM3PPredictionHead + decoder`): When `has_decoder_head=True` in config, produces logits over beatmap vocabulary for MLM training/inference.\n\n### Objectives\n- **Contrastive Loss** (`cm3p_loss`): Symmetric cross-entropy over similarity matrices between beatmap embeddings and metadata embeddings. If metadata variations are present, the original metadata acts as the positive; others as structured negatives.\n- **Masked LM Loss** (if enabled): Standard token-level cross-entropy over masked positions.\n- **Classification Loss** (downstream fine-tunes): For tasks like ranked vs unranked beatmap classification (`CM3PForBeatmapClassification`).\n\n### Attention Implementation\n\n`attn_implementation` can be set to `flash_attention_2` (for supported GPUs with Flash Attention 2 installed), `sdpa` (standard PyTorch attention), or `eager` (fallback implementation).\nFlash Attention 2 offers significant speed and memory benefits for long sequences. CM3P + Flash Attention also supports unpadding batched input sequences for token efficiency.\n\n---\n## 6. Training (From Scratch / Fine-tuning)\n### Base Command\nHydra is used for config composition:\n```bash\npython train.py --config-name v1              # Uses configs/train/v1.yaml -\u003e defaults chain\npython train.py --config-name v7              # Swap to another experiment variant\n```\nOverride any field inline:\n\n```bash\npython train.py -cn v7 training.learning_rate=5e-5 dataset.labels=masked_lm model_cls=CM3PForMaskedLM\n```\n\nFor all overridable configurations see `configs/train/default.yaml`.\nI recommend making a copy of an existing config (e.g., `v7.yaml`) and modifying it for your experiments.\n\n### Fine-tuning From Pretrained\nProvide a checkpoint path or load a Hub model:\n\n```bash\npython train.py -cn \"v7_classifier\" from_pretrained=\"OliBomby/CM3P\" 'dataset={train_dataset_paths:[\"/workspace/datasets/MMRS39389\"],test_dataset_paths:[\"/workspace/datasets/MMRS39389\"],train_dataset_end:39000,test_dataset_start:39000,test_dataset_end:39389}' 'training={dataloader_num_workers:8}' wandb_entity=mappingtools\n```\n\n### Resume Training\nIf `output_dir` has checkpoints and `overwrite_output_dir=false`, the script auto-resumes (unless overridden by `training.resume_from_checkpoint`).\n\n### WandB Logging\nSet:\n```bash\nwandb_project=CM3P wandb_entity=your_entity wandb_mode=online\n```\nDisable (offline) by `wandb_mode=disabled` or remove variables.\n\n### Pushing to Hugging Face Hub\nMake sure you are logged in (`huggingface-cli login`).\nEnable:\n```bash\ntraining.push_to_hub=true\n```\nOr  use `push_to_hub.py` after training with a path to the saved checkpoint.\n\n---\n## 7. Evaluation \u0026 Metrics\n`compute_metrics` (in `train.py`) aggregates metrics across evaluation steps:\n- Zero-shot classification accuracy per variation class: original vs altered year/status/tags/mapper.\n- Masked LM accuracy (if MLM labels present).\n- Classification accuracy + top-5 for beatmap-level tasks.\n\nDuring evaluation, metadata variation groups are resolved to check whether the highest-scoring metadata among variations is the original.\n\nMetrics logged to console, saved to `eval_results.json` style files in `output_dir`, and optionally to WandB.\n\n---\n## 8. Configuration Overview\n`configs/train/default.yaml` controls training, processor parameters, dataset filtering, and Hydra output directory.\n`configs/model/` contains model-level defaults (can extend for different projection dims, hidden sizes, enabling decoder heads, etc.).\n\nTo inspect active config at runtime, print or log `OmegaConf.to_yaml(args)` (you can add a line in `train.py`).\n\n---\n## 9. Advanced Topics\n### Audio Fusion Details\nAudio features are extracted (log-mel), chunked to `max_source_positions`, passed through conv + ModernBERT encoder, then projected. The resulting dense embeddings replace placeholder audio tokens in the beatmap embedding sequence before the beatmap transformer processes them.\n\n### Metadata Variations\nMultiple metadata sequences per beatmap allow structured negatives (e.g., one with altered tags/year). Loss only treats the original (`variation class 0`) as positive; others increase robustness.\n\n---\n## 10. Troubleshooting\n- OOM: Reduce `per_device_train_batch_size`, increase `gradient_accumulation_steps`, lower sequence/window lengths (`processor.default_kwargs.beatmap_kwargs.max_length`).\n- Slow data loading: Increase `training.dataloader_num_workers` or reduce `cycle_length`.\n\n---\n## 11. Roadmap / Next Steps\n- Find some way to use PyTorch compilation during training.\n- Colab notebook examples for inference \u0026 embedding extraction.\n- Evaluate beatmap generative models using distributions of CM3P embeddings.\n\n## Related works\n\n1. [osu! Beatmap Generator](https://github.com/Syps/osu_beatmap_generator) by Syps (Nick Sypteras)\n2. [osumapper](https://github.com/kotritrona/osumapper) by kotritrona, jyvden, Yoyolick (Ryan Zmuda)\n3. [osu-diffusion](https://github.com/OliBomby/osu-diffusion) by OliBomby (Olivier Schipper), NiceAesth (Andrei Baciu)\n4. [osuT5](https://github.com/gyataro/osuT5) by gyataro (Xiwen Teoh)\n5. [Beat Learning](https://github.com/sedthh/BeatLearning) by sedthh (Richard Nagyfi)\n6. [osu!dreamer](https://github.com/jaswon/osu-dreamer) by jaswon (Jason Won)\n7. [Mapperatorinator](https://github.com/OliBomby/Mapperatorinator) by OliBomby (Olivier Schipper)\n8. [osuBERT](https://github.com/khang06/osuBERT) by Khangaroo\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folibomby%2Fcm3p","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Folibomby%2Fcm3p","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folibomby%2Fcm3p/lists"}