https://github.com/ranxi2001/kaggle-auto-research

AI Agent 驱动的 Kaggle 竞赛自动研究框架，支持调研、EDA、特征工程、模型训练、迭代优化、集成与安全提交管理。
https://github.com/ranxi2001/kaggle-auto-research

ai-agent automation cli competition-framework data-science eda feature-engineering kaggle kaggle-competition typer

Last synced: 18 days ago
JSON representation

AI Agent 驱动的 Kaggle 竞赛自动研究框架，支持调研、EDA、特征工程、模型训练、迭代优化、集成与安全提交管理。

• Host: GitHub
• URL: https://github.com/ranxi2001/kaggle-auto-research
• Owner: ranxi2001
• License: mit
• Created: 2026-06-08T14:26:35.000Z (about 1 month ago)
• Default Branch: main
• Last Pushed: 2026-06-08T20:06:36.000Z (about 1 month ago)
• Last Synced: 2026-06-08T20:10:06.129Z (about 1 month ago)
• Topics: ai-agent, automation, cli, competition-framework, data-science, eda, feature-engineering, kaggle, kaggle-competition, typer
• Language: Python
• Homepage: https://github.com/ranxi2001/kaggel-auto-research
• Size: 666 KB
• Stars: 1
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README-EN.md
  • Changelog: CHANGELOG.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Security: SECURITY.md
  • Agents: AGENTS.md

Awesome Lists containing this project

README

          


  





  Agent-oriented automation framework for Kaggle competitions





  简体中文

  ·

  English

  ·

  Agent Rules

  ·

  Contributing

  ·

  Security





  

  

  

  





  

  

  

  

  

  

  

  





  

  

  

  



Kaggle Auto Research turns competition research, data download, EDA, feature generation, model training, iterative improvement, ensembling, submission validation, and leaderboard feedback into reusable CLI stages and versioned workspace artifacts. The goal is not to write one-off competition scripts, but to give humans and coding agents a safe, reproducible way to keep improving results.

> This project is experimental. It is designed for research, prototyping, and internal competition workflows. Real Kaggle submissions are not executed automatically by default.

## Project Positioning

This project focuses on the engineering problems that make real agent-driven Kaggle work fragile:

- Short commands hide virtualenv and Kaggle CLI details, for example `kar auth` and `kar data drw-crypto`.

- Workspaces isolate each competition's config, data, models, submissions, and experiment logs.

- Versioned artifacts preserve models, OOF predictions, test predictions, and submission files.

- Submission budgets and dry-run validation protect scarce Kaggle daily submissions.

- Public notebooks, feature selection, CV, metrics, ensembling, and leaderboard feedback are meant to become reusable tools instead of one-off scripts.

At this stage, Kaggle Auto Research is a runnable open-source research toolkit, not a polished AutoML product. Contributions should prioritize agent operability, reproducibility, and safety boundaries.

## Project Status

- Current version: `0.1.0`

- Python: `>=3.11`

- Main use cases: Kaggle tabular / time-series / low-signal competition automation

- Default behavior: generate, validate, and log locally; never submit by default

- Open-source entrypoints: [README.md](README.md), [AGENTS.md](AGENTS.md), [CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md), [CHANGELOG.md](CHANGELOG.md)

## Maturity

| Module | Status | Notes |

| --- | --- | --- |

| Workspace / config | Usable | Supports multiple competition directories, templates, and basic config. |

| Kaggle OAuth / data / leaderboard | Usable | `kar auth`, `kar data`, and `kar leaderboard` work directly. |

| Baseline pipeline | Usable but early | Generic tabular flow runs; hard competitions still need recipes. |

| DRW Crypto recipe | In active validation | Strong local OOF exists, but real LB calibration is still needed. |

| Submit safety | Usable | Supports dry-run, budget status, and submission history. |

| Experiment registry | Planned | Needs immutable run IDs, params, commands, data fingerprints, and runtime metadata. |

| Notebook mining | Planned | Public notebook ideas can be absorbed manually; structured mining is not built yet. |

| JSON output | Planned | Agent parsing still needs stable `--json` output. |

## Highlights

| Capability | Description |

| --- | --- |

|  | Research, EDA, Feature, Train, Iteration, and Submit stages communicate through filesystem artifacts. |

|  | Every competition keeps config, data, models, reports, and submissions under `workspaces//`. |

|  | `config.yaml` stores data paths, target columns, submission format, metrics, CV strategy, models, and submission budget. |

|  | Features, models, OOF predictions, test predictions, and submissions are saved by version. |

|  | The default flow generates and dry-run validates submission files; real submissions require explicit control. |

|  | Public notebooks can be pulled locally; future work will mine feature lists, model params, CV strategy, and ensemble formulas. |

|  | See [docs/agent-tooling-roadmap.md](docs/agent-tooling-roadmap.md) for the Agent tooling roadmap. |

## Current DRW Crypto Progress

This repository is using DRW Crypto Market Prediction as the first real tooling testbed.

- Initial auto-feature baseline: CV R2 `-0.002859`

- Clean feature selection + LightGBM: best single-model Pearson around `0.0724`

- Public-notebook-style XGB time-slice: Pearson `0.053338`, but useful as an ensemble-diversity signal

- Closed-form Ridge on top-correlation features: TimeSeriesSplit Pearson `0.143261`

- Pearson OOF rank-normalized simplex ensemble: common-valid-OOF Pearson `0.149746`

- Current best Kaggle submission: `sub_ensemble_ranknorm_v005_v010_v012_v015_v017_v018_v019_v020_v021_v022_v023_v024_v025_v026.csv`, public LB `0.08199`

- Failed recency-proxy submission: `sub_calibrated_tail_cli_full.csv`, public LB `0.07184`

- Next offline candidate: `sub_anchor_blend_conservative.csv`, a small rank blend around the best submitted anchor

- Public leaderboard top is around `0.11 - 0.14`

Conclusion: the toolchain can run end-to-end and improve, but DRW exposed a clear CV/LB gap. The next submission should be anchor-calibrated against the best real LB result instead of blindly maximizing local tail proxies.

## Tooling Gaps Exposed By The DRW Run

The recent DRW Crypto work showed that model code was not the main bottleneck. The missing infrastructure was:

| Gap | Why it matters | Planned tool |

| --- | --- | --- |

| schema/config inspection | DRW's real target column is `label`, so templates cannot be trusted blindly. | `kar inspect  --fix-config` |

| metric contract | RMSE, R2, and Pearson confusion can mislead low-signal search. | `MetricSpec` + trainer validation |

| experiment registry | Comparing runs currently requires scanning `models/vNNN` and metadata files. | `kar experiments` / immutable run registry |

| submission metadata | Manual file submissions still need CV, model versions, and ensemble weights. | normalized submission `.json` files |

| LB sync | Kaggle scores should be written back into local history after submission. | `kar sync-lb ` |

| LB-anchor calibration | Once real LB feedback exists, next candidates should stay close to the best submitted anchor and avoid known failed directions. | `kar drw-anchor-blend ` |

| anti-failed feedback probes | Failed submissions should become explicit negative directions, with candidate families that preserve anchor similarity while moving away from known bad geometry. | `kar drw-anti-failed ` |

| submission-geometry scoring | Candidate files can be ranked against real submitted anchors and failed directions before spending more Kaggle budget. | `kar drw-score-candidates ` |

| notebook miner | Public feature lists, CV schemes, and ensembles should become structured ideas. | `kar notebooks mine` |

| recipe system | Commands like `drw-ridge` should graduate from hardcoded scripts into templates. | `kar run --template crypto/low_signal_feature_select` |

| agent-friendly output | Long tables and styled output are not stable machine interfaces. | global `--json` + stable exit codes |

See [docs/agent-tooling-roadmap.md](docs/agent-tooling-roadmap.md) for the full roadmap.

## Installation

```bash

git clone https://github.com/ranxi2001/kaggel-auto-research.git

cd kaggel-auto-research

python -m venv .venv

.venv\Scripts\activate

pip install -e .

```

On Windows, the repository includes a short launcher:

```bash

.\kar auth

.\kar ls

```

In Git Bash:

```bash

./kar.cmd auth

```

Development dependencies:

```bash

pip install -e ".[dev]"

```

## Quick Start

```bash

# 1. Authenticate with Kaggle

kar auth

# 2. Create a workspace

kar init titanic --type tabular --url https://www.kaggle.com/competitions/titanic

# 3. Download and extract data

kar data titanic

# 4. Run stages

kar research titanic

kar eda titanic

kar train titanic

# 5. Validate a submission without sending it to Kaggle

kar submit titanic --dry-run -f submissions/sub_001.csv

```

Full pipeline:

```bash

kar pipeline titanic --full

```

Improvement loop:

```bash

kar pipeline titanic --iterate 5

kar analyze titanic

kar ensemble titanic

```

## Common CLI

```bash

kar ls

kar auth

kar init  --type tabular --url 

kar data 

kar research 

kar eda 

kar eda  --features-only

kar train 

kar train  --trials 30

kar pipeline  --full

kar pipeline  --from train

kar pipeline  --iterate 5

kar status 

kar analyze 

kar ensemble  --top 5

kar leaderboard 

kar leaderboard  --top -n 10

kar sync-lb 

kar submit  --status

kar submit  --history

kar submit  --dry-run -f submissions/sub_001.csv

kar submit  --flush

```

Note: `kar.cmd` is a short Windows launcher at the repository root, so users do not need to type `.\.venv\Scripts\kar.exe`. After editable installation, `kar` also works directly.

`kar submit  --status` shows the local budget date, next local reset date, and full reserve-queue filenames. Use it before `--flush` to confirm exactly which candidate will be submitted.

If the same submission file is queued more than once, the reserve entry is updated in place instead of duplicated.

DRW Crypto research helpers:

```bash

kar drw-clean drw-crypto --top-k 130 --n-estimators 1200 --learning-rate 0.015

kar drw-ridge drw-crypto --top-k 180 --alphas 300,1000,3000,10000 --cv timeseries

kar drw-public drw-crypto --model lgbm --folds 3

kar drw-ensemble drw-crypto --models v010,v011,v007,v003

kar drw-ensemble drw-crypto --models v005,v010,v012,v015,v017,v018,v019,v020,v021,v022,v023,v024,v025,v026 --method optimize --transform ranknorm

kar drw-tail-ensemble drw-crypto --samples 12000 --seed 47 --output-tag tail_cli

kar drw-anchor-blend drw-crypto --output-tag anchor_blend_safe

kar drw-anchor-blend drw-crypto --groups safe:v016+v017+v031+v032 --alpha-grid 0.15,0.18,0.19,0.20,0.21 --min-spearman 0.994 --max-rank-delta 0.025 --output-tag anchor_blend_conservative

kar drw-anchor-blend drw-crypto --selection-metric utility --failed-threshold 0.935 --risk-penalty 0.60 --output-tag anchor_blend_utility_scan

kar drw-anti-failed drw-crypto --beta-grid 0.04,0.06,0.08,0.10,0.12,0.15

kar drw-score-candidates drw-crypto --output-tag candidate_geometry_score

kar drw-score-candidates drw-crypto --output-tag candidate_geometry_score --json-output

kar drw-compare-submissions drw-crypto --files sub_anchor_blend_utility_scan.csv,sub_anchor_blend_conservative.csv,sub_anti_failed_rank_beta020.csv

```

## Workspace Layout

```text

workspaces//

|-- config.yaml

|-- data/

|   |-- raw/

|   |-- processed/

|   `-- features/

|-- models/

|   `-- v001/

|       |-- model.pkl

|       |-- cv_scores.json

|       |-- oof_preds.npy

|       |-- test_preds.npy

|       `-- importance.csv

|-- reports/

|   |-- research_notes.md

|   `-- eda_summary.md

|-- submissions/

|-- journal.json

|-- idea_pool.json

`-- .state/

```

Large data, models, submissions, and caches are ignored by git.

## Configuration Example

```yaml

competition:

  name: "titanic"

  url: "https://www.kaggle.com/competitions/titanic"

  type: "tabular"

  metric: "accuracy"

  metric_direction: "maximize"

data:

  train: "data/raw/train.csv"

  test: "data/raw/test.csv"

  sample_submission: "data/raw/sample_submission.csv"

  target_column: "Survived"

  id_column: "PassengerId"

model:

  primary: "lightgbm"

  cv_strategy: "stratified_kfold"

  cv_folds: 5

  seed: 42

submission:

  auto_submit: false

  best_threshold: 0.001

  max_daily: 2

```

For time-series, finance, and trading competitions, prefer time-based CV:

```yaml

model:

  cv_strategy: "time_series_split"

```

## Architecture

```text

User / Agent

    |

    v

kar CLI

    |

    v

Pipeline Runner

    |

    +--> Research Agent  --> reports/research_notes.md

    +--> EDA Agent       --> reports/eda_summary.md

    +--> Feature Agent   --> data/features/v*.parquet

    +--> Train Agent     --> models/v*/

    +--> Iteration Agent --> journal.json, idea_pool.json

    +--> Submit Agent    --> submissions/, .state/submission_budget.json

```

State machine:

```text

RESEARCH -> EDA -> FEATURES -> TRAIN -> EVALUATE -> CANDIDATE_READY

                                ^                       |

                                |                       v

                             ITERATE              USER DECISION

                                                        |

                                                        v

                                                     SUBMIT

```

Detailed agent rules are in [AGENTS.md](AGENTS.md).

## Why Agent-Oriented Tooling

Agents usually fail Kaggle workflows because of operational issues, not because they cannot write model code:

- wrong metric or CV setup;

- mismatched config and data schema;

- loading huge parquet files without sampling;

- missing experiment metadata;

- public notebook ideas not converted into structured candidates;

- wasted submission budget;

- CLI output that is hard to parse.

This project prioritizes those tooling problems before adding more model scripts.

## Roadmap

Near-term priorities:

- `kar inspect  --fix-config`: verify and patch schema/config mismatch.

- Experiment registry: command, params, features, metric, CV, runtime, and status for every run.

- Notebook miner: extract public feature lists, params, CV strategy, and ensemble formulas.

- Recipe system: turn one-off commands like `drw-clean` into reusable templates.

- OOF ensemble builder: grid/ridge/rank-average blends with metadata.

- `--json` output for agent parsing.

- `kar sync-lb`: sync Kaggle leaderboard scores and ranks into local history.

Full roadmap: [docs/agent-tooling-roadmap.md](docs/agent-tooling-roadmap.md).

## Safety

- Kaggle submissions are never automatic by default.

- Real submission, data/model deletion, credential edits, and remote pushes require explicit user intent.

- Training, feature generation, submission file generation, dry-run validation, and experiment logging can run automatically.

- Kaggle data, models, submissions, notebook caches, and local credentials should not be committed to git.

## Contributing

Contributions are welcome, especially around agent reliability and open-source

tooling:

- `kar inspect ` schema/config checks;

- experiment registry and run metadata;

- metric/CV contracts;

- public notebook mining;

- reusable recipe system;

- OOF ensemble builders;

- `--json` output and stable exit codes.

Read [CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request. Coding

agents should read [AGENTS.md](AGENTS.md).

## License

MIT, see [LICENSE](LICENSE).