{"id":51004085,"url":"https://github.com/firstbatchxyz/constellation","last_synced_at":"2026-06-20T18:02:58.165Z","repository":{"id":356338430,"uuid":"1231709687","full_name":"firstbatchxyz/constellation","owner":"firstbatchxyz","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-07T16:47:04.000Z","size":207,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-07T17:32:50.173Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/firstbatchxyz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-07T08:00:55.000Z","updated_at":"2026-05-07T16:47:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/firstbatchxyz/constellation","commit_stats":null,"previous_names":["firstbatchxyz/constellation"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/firstbatchxyz/constellation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fconstellation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fconstellation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fconstellation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fconstellation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/firstbatchxyz","download_url":"https://codeload.github.com/firstbatchxyz/constellation/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fconstellation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34580041,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-20T02:00:06.407Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-20T18:02:54.629Z","updated_at":"2026-06-20T18:02:58.160Z","avatar_url":"https://github.com/firstbatchxyz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Constellation](docs/constellation.png)\n\n# Constellation\n\nCapability-distilled specialist network experiments for Qwen3-4B descendants.\n\nThe initial target is not routing or serving. The first target is proving that a\nsingle capability specialist beats a matched general-agent distilled control on\nheld-out tasks from the same broad distribution.\n\n## Current Scope\n\n- Canonical JSONL schema for agent and reasoning trajectories.\n- Parsers for AgentTrove-style and Hermes-style traces.\n- Capability labeling helpers.\n- Basic malformed/repetitive trace filters.\n- A lightweight quality scorer for pilot data curation.\n\nNo model weights, datasets, or training artifacts are checked into this repo.\n\n## Commands\n\nUse `uv` for local and GPU-machine commands:\n\n```bash\nuv run python -m unittest discover -s tests\nuv run python -m constellation.cli --help\n```\n\nGPU-only training dependencies are intentionally not required for local tests.\nInstall them on the GPU machine when you are ready to run streaming/training:\n\n```bash\nuv pip install -r requirements/train.txt\n```\n\nThe default SFT backend uses Hugging Face `Trainer` on Accelerate with custom\nlabels so the canonical turn mask is exact. `TRL` is included in the GPU\nrequirements and can be enabled with `\"use_trl_sft_trainer\": true` after a smoke\nrun confirms the installed TRL version accepts the custom masked dataset.\n\n## Data Access Policy\n\nDo not download full datasets locally into this workspace.\n\nUse one of:\n\n- Hugging Face streaming / iterable datasets\n- remote object storage\n- remote training-node scratch space\n- tiny synthetic fixtures for tests\n\nLocal files in this repo should be code, configs, docs, schemas, and small test\nfixtures only.\n\n## Dataset Labeling\n\nAll rollout sources should be normalized into shared capability and domain\ntaxonomies:\n\n- [configs/capability_taxonomy.json](configs/capability_taxonomy.json)\n- [configs/domain_taxonomy.json](configs/domain_taxonomy.json)\n\nUse [docs/DATASET.md](docs/DATASET.md) for the dataset-first relabeling and\nprompt/ICL labeling flow. The first distillation registry supports up to 20\nspecialists in [configs/specialist_targets.json](configs/specialist_targets.json).\nModernBERT can still be useful as an encoder scorer, but not as a decoder-style\nprompt model that emits labels.\n\nFor the main dataset build, use `llm-label` with a small instruction model. The\ndefault is `Qwen/Qwen3.5-0.8B` after the Qwen3-0.6B probe showed too much\ncoding-domain bias. It reads the taxonomy and emits strict JSON labels. The\nolder `model-label` NLI scorer is still useful as a very fast baseline/audit\npath. Weak relabeling is only a fallback/audit path.\n\nFor speed, run the labeler behind SGLang and call `llm-label --backend sglang`\nso model weights stay resident while Constellation streams JSONL rows through\nthe OpenAI-compatible API.\n\nThe SGLang/OpenAI-compatible path uses JSON Schema structured output by default\nto constrain label arrays to known taxonomy enums.\n\n`llm-label` also records post-LLM guardrail edits in metadata so calibration\ncleanup is auditable.\n\nFor production-sized dataset builds on the GPU node, use the systemd helper so\nSGLang and the sharded canonical/labeling job survive SSH disconnects:\n\n```bash\ngit pull\nscripts/final_dataset_systemd.sh setup\n\njournalctl --user -u constellation-final-dataset -f\n```\n\nThe helper writes canonical shards, labels them through the local SGLang server,\nand produces final label/target reports under `CONSTELLATION_RUNS_DIR/final`.\nIt also reapplies `nvidia-cudnn-cu12==9.16.0.29` before SGLang starts, because\nsome `torch`/`sglang` installs resolve back to CuDNN 9.10 while SGLang rejects\nthat PyTorch/CuDNN combination.\nIf a source stream is still writing a `.tmp` canonical directory, completed\nshards can be labeled in parallel with:\n\n```bash\nscripts/final_dataset_systemd.sh label-available-shards\n```\n\nFor a fresh GPU node that should resume from already-uploaded labeled\nAgentTrove shards and continue Hermes formatting/labeling, use the gold-path\nresume helper:\n\n```bash\ngit clone https://github.com/firstbatchxyz/constellation.git /home/ubuntu/constellation\ncd /home/ubuntu/constellation\nscripts/resume_labeling_node.sh start\n```\n\nIt downloads uploaded AgentTrove labels from Hugging Face, marks them done,\nstarts SGLang, streams Hermes Kimi/GLM into canonical shards, and keeps a\nbackground label loop running over closed shards. Check progress with:\n\n```bash\nscripts/resume_labeling_node.sh status\n```\n\nOn RTX 6000 / Blackwell CUDA 13 nodes, prefer the Docker launcher to avoid local\nCUDA toolkit and JIT kernel drift:\n\n```bash\nscripts/run_sglang_rtx6000.sh --detach --pull --stop-existing\n```\n\n`resume_labeling_node.sh` automatically uses this Docker path when it detects an\nRTX 6000/Blackwell GPU and Docker is available.\n\nFor CPU-only formatter nodes, install `uv`, stream Hermes into canonical shards,\nand optionally upload complete canonical shards with:\n\n```bash\ngit clone https://github.com/firstbatchxyz/constellation.git /home/ubuntu/constellation\ncd /home/ubuntu/constellation\nscripts/format_cpu_node.sh start\n```\n\nThe formatter does not use LLMs or GPUs. It only streams, parses, filters, and\nwrites canonical JSONL shards. On high-core CPU nodes it starts multiple HF\nstream shards per Hermes source; tune with `FORMAT_PARALLELISM_PER_SOURCE`.\nCheck progress with:\n\n```bash\nscripts/format_cpu_node.sh status\n```\n\n## Core Training Rule\n\nTool observations are context, not targets.\n\nDuring SFT, loss should be applied only to assistant-generated spans:\n\n- reasoning / scratchpad spans, if included for the experiment\n- tool calls\n- final answers\n\nLoss should be masked for:\n\n- system and user prompts\n- tool observations\n- environment output\n\nThis matters because predicting observations teaches the model to hallucinate\nthe environment instead of using it.\n\n## First Experiment\n\nUse one H100 for a small matched-control pilot:\n\n1. Stream a slice of AgentTrove and Hermes traces into canonical records.\n2. Filter to successful, coherent, non-repetitive trajectories.\n3. Build two training sets with matched token budgets:\n   - `general_agentic_mix`\n   - one narrow specialist, probably `DEBUGGING` or `TERMINAL_WORKFLOW`\n4. Full-SFT both descendants from the same Qwen3-4B checkpoint.\n5. Evaluate against base Qwen3-4B, the general distilled control, and the\n   specialist on held-out tasks grouped by task/repo/source to avoid leakage.\n\nSee [docs/PILOT.md](docs/PILOT.md) for the concrete pilot shape.\n\n## Pilot Commands\n\nRun these on the GPU machine. They stream only the requested row count and write\ncurated artifacts under `CONSTELLATION_RUNS_DIR` or `~/constellation-runs`.\n\n```bash\nexport CONSTELLATION_RUNS_DIR=~/constellation-runs\n\nuv run python -m constellation.cli stream-convert \\\n  --source agenttrove \\\n  --max-rows 10 \\\n  --output '{runs_dir}/canonical/agenttrove.debugging_probe.jsonl' \\\n  --skip-errors\n\nuv run python -m constellation.cli stream-convert \\\n  --source hermes-kimi \\\n  --max-rows 10 \\\n  --output '{runs_dir}/canonical/hermes_kimi.debugging_probe.jsonl' \\\n  --skip-errors\n\nuv run python -m constellation.cli build-subsets \\\n  --input ~/constellation-runs/canonical/agenttrove.debugging_probe.jsonl \\\n          ~/constellation-runs/canonical/hermes_kimi.debugging_probe.jsonl \\\n  --output-dir '{runs_dir}/subsets' \\\n  --max-train-tokens 200000\n\nuv run python -m constellation.cli train-sft --config configs/train_debugger_sft.json\nuv run python -m constellation.cli train-sft --config configs/train_general_agent_sft.json\nuv run python -m constellation.cli eval --config configs/eval_debugging.json\n```\n\n## External Data Notes\n\nAs of May 7, 2026:\n\n- [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) is listed as\n  Apache-2.0 on Hugging Face.\n- [open-thoughts/AgentTrove](https://huggingface.co/datasets/open-thoughts/AgentTrove)\n  is a large open agent-trajectory corpus using a ShareGPT/terminus-style\n  message layout.\n- [lambda/hermes-agent-reasoning-traces](https://huggingface.co/datasets/lambda/hermes-agent-reasoning-traces)\n  contains multi-turn tool-calling traces with real tool execution results.\n\nReasoning-distill datasets generated from commercial APIs should stay in a\nquarantine bucket until provenance, license, and provider terms are reviewed.\nThe HF dataset license field alone is not enough for a clean training decision.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirstbatchxyz%2Fconstellation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffirstbatchxyz%2Fconstellation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirstbatchxyz%2Fconstellation/lists"}