{"id":21667877,"url":"https://github.com/jilljenn/ktm","last_synced_at":"2025-08-21T01:32:08.925Z","repository":{"id":147558521,"uuid":"124039302","full_name":"jilljenn/ktm","owner":"jilljenn","description":"Knowledge Tracing Machines: Factorization Machines for Knowledge Tracing","archived":false,"fork":false,"pushed_at":"2024-04-05T15:50:37.000Z","size":8081,"stargazers_count":134,"open_issues_count":7,"forks_count":34,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-08T01:41:57.786Z","etag":null,"topics":["educational-software","factorization-machines","knowledge-tracing"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1811.03388","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jilljenn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-03-06T07:34:45.000Z","updated_at":"2025-01-25T15:38:55.000Z","dependencies_parsed_at":"2024-01-16T04:46:17.318Z","dependency_job_id":null,"html_url":"https://github.com/jilljenn/ktm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jilljenn/ktm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jilljenn%2Fktm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jilljenn%2Fktm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jilljenn%2Fktm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jilljenn%2Fktm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jilljenn","download_url":"https://codeload.github.com/jilljenn/ktm/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jilljenn%2Fktm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271415020,"owners_count":24755628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["educational-software","factorization-machines","knowledge-tracing"],"created_at":"2024-11-25T11:47:45.359Z","updated_at":"2025-08-21T01:32:08.919Z","avatar_url":"https://github.com/jilljenn.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Codecov](https://img.shields.io/codecov/c/github/jilljenn/ktm.svg)](https://codecov.io/gh/jilljenn/ktm/)\n\n# Knowledge Tracing Machines\n\n![](ktm.png)\n\n- Presented at the AAAI 2019 conference in Honolulu, Hawaii on January 27, 2019 [[arXiv]](https://arxiv.org/abs/1811.03388) [[slides]](https://jiji.cat/slides/aaai2019-ktm-slides.pdf).\n- Applied in the [Best Paper Award](https://arxiv.org/abs/1905.06873) of the EDM 2019 conference in Montreal, Canada on July 2, 2019.\n\n```bibtex\n@inproceedings{Vie2019,\n  Author = {{Vie}, Jill-J{\\^e}nn and {Kashima}, Hisashi},\n  Booktitle = {Proceedings of the 33th {AAAI} Conference on Artificial Intelligence},\n  Title = {{Knowledge Tracing Machines: Factorization Machines for Knowledge Tracing}},\n  Pages = {750--757},\n  Url = {https://arxiv.org/abs/1811.03388},\n  Year = 2019\n}\n```\n\nAuthors: [Jill-Jênn Vie](https://jjv.ie), [Hisashi Kashima](https://hkashima.github.io/index_e.html)\n\n## Follow our tutorial\n\nPresented at the [Optimizing Human Learning](https://jilljenn.github.io/humanlearn) workshop in Kingston, Jamaica on June 4, 2019.\n\nSlides from the tutorial are available [here](https://jjv.ie/slides/tuto.pdf). A Jupyter notebook will be available \"soon\" on Binder.\n\nThe tutorial makes you play with the models to assess **weak generalization**. To assess **strong generalization** and reproduce the experiments of the paper, you may want to use scikit-learn's [GroupShuffleSplit](https://scikit-learn.org/stable/modules/cross_validation.html#group-shuffle-split), cf. the [sktm.py](sktm.py) file.\n\n## Install\n\n    python3 -m venv venv\n    . venv/bin/activate\n    pip install -r requirements.txt  # Will install numpy, scipy, pandas, scikit-learn, pywFM\n\nIf you also want to get the factorization machines running (KTM for *d* \u003e 0), you should also do:\n\n    make libfm\n\n## Prepare data\n\nSelect a dataset and the features you want to include.\n\n### Case 1: There is only one skill per item.\n\n`data/\u003cdataset\u003e/data.csv` should contain the following columns:\n\n    user, item, skill, correct, wins, fails\n\nwhere wins and fails are the number of successful and unsuccessful\nattempts at the corresponding skill.\n\n### Case 2: There may be several skills associated to an item.\n\n`data/\u003cdataset\u003e/needed.csv` needs to contain:\n\n    user_id, item_id, correct\n\n(Note the difference.)\n\nAnd `data/\u003cdataset\u003e/q_mat.npz` should be a q-matrix under `scipy.sparse` format.\n\nIf you want to compute wins and fails like in PFA or DAS3H,\nyou should run `encode_tw.py` instead of this file, with the `--pfa` option for PFA or `--tw` for DAS3H.\n\n## Running\n\n### Available datasets\n\n- [Assistments 2009](https://sites.google.com/site/assistmentsdata/home/2009-2010-assistment-data)\n- Our [reformatted version of the Assistments 2009 dataset](https://jiji.cat/weasel2018/data.csv).\n- Berkeley and Castor datasets are private.\n- Datasets ECPE and TIMSS 2003 come from the [CDM package](https://alexanderrobitzsch.github.io/CDM/reference/):\n\n```R\n\u003e install.packages('CDM')\n\u003e library('CDM')\n\u003e dim(fraction.subtraction.data)\n[1] 536  20\n\u003e dim(data.ecpe$data)\n[1] 2922   29\n\u003e dim(data.timss03.G8.su$data)\n[1] 757  25\n```\n\n### Encoding data into sparse features (quick start)\n\n    python encode.py --users --items  # To get the encodings (npz)\n    python lr.py data/dummy/X-ui.npz  # To get results (txt)\n    python sktm.py data/dummy/data.csv --feat ui  # Will be faster\n\nYou can also download the [Assistments 2009 dataset](https://jiji.cat/weasel2018/data.csv) into `data/assistments09` and change the dataset:\n\n    python encode.py --dataset assistments09 --skills --wins --fails  # Will encode PFA sparse features into X-swf.npz\n\nIf you are lazy, you can also just do `make` and try to understand what is going on in the [Makefile](Makefile).\n\n### Encoding time windows\n\nChoffin et al. proposed the DAS3H model, and we implemented it using queues. This code is faster than the original KTM encoding.\n\nTo prepare a dataset like Assistments, see examples in the `data` folder.  \nSkill information should be available either as `skill_id`, or `skill_ids` separated with `~~`, or in a q-matrix `q_mat.npz`.\n\n    python encode_tw.py --dataset dummy_tw --tw  # Will encode DAS3H sparse features into X.npz\n\nThen you can run `lr.py` or `fm.py`, see below.\n\n### Running a ML model\n\nIf you want to encode PFA features:\n\n    python encode.py --skills --wins --fails  # Will create X-swf.npz\n\nFor logistic regression:\n\n    python lr.py data/dummy/X-swf.npz\n\t# Will save weights in coef0.npy\n\nFor factorization machines of size *d* = 5:\n\n    python fm.py --d 5 data/dummy/X-swf.npz\n\t# Will save weights in w.npy and V.npy\n\nThe following code does not work if you don't have user_id as column in CSV file.\n\n## Results\n\n### Strong generalization\n\nOn the Assistments 2009 dataset:\n\n| Model | Dimension | AUC | Note |\n|:-----:|:---------:|:---:|:-----------:|\n| KTM: items, skills, wins, fails, extra | 5 | **0.819** | |\n| KTM: items, skills, wins, fails, extra | 5 | 0.815 | +0.05 |\n| KTM: items, skills, wins, fails | 10 | 0.767 | |\n| KTM: items, skills, wins, fails | 0 | 0.759 | +0.02; 0.747 in 2025, random seed 42 |\n| (*DKT* (Wilson et al., 2016)) | 100 | 0.743 | +0.05 |\n| IRT: users, items | 0 | 0.691 | 0.678 in 2025, random seed 42 |\n| PFA: skills, wins, fails | 0 | 0.685 | +0.07; 0.703 in 2025, random 42 |\n| AFM: skills, attempts | 0 | 0.616 | |\n\nOn the [Duolingo](http://sharedtask.duolingo.com) French dataset:\n\n| Model | Dimension | AUC | Improvement |\n|:-----:|:---------:|:---:|:-----------:|\n| KTM   | 20        | **0.822** | +0.01 |\n| DeepFM | 20       | 0.814 | +0.04 |\n| Logistic regression + L2 reg | 0 | 0.771 |\n\nWe also showed that Knowledge Tracing Machines (Bayesian FMs) got better results than Deep Factorization Machines on the [Duolingo dataset](http://sharedtask.duolingo.com). See our article: [Deep Factorization Machines for Knowledge Tracing](https://arxiv.org/abs/1805.00356) and [poster](https://github.com/jilljenn/ktm/blob/master/poster/dfm-kt-poster.pdf) at the [BEA](https://www.cs.rochester.edu/~tetreaul/naacl-bea13.html) workshop at New Orleans, LA on June 5, 2018.\n\n    @inproceedings{Vie2018,\n      Author = {{Vie}, Jill-J{\\^e}nn},\n      Booktitle = {{Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications}},\n      Pages = {370--373},\n      Title = {{Deep Factorization Machines for Knowledge Tracing}},\n      Url = {http://arxiv.org/abs/1805.00356},\n      Year = 2018}\n\n### Weak generalization\n\nThose numbers may change according to your random state seed. For the numbers below we used random_state=42 and seed=42 for respectively (multicore) 5-fold cross validation and libfm.\n\nOn the Assistments 2009 dataset:\n\n| AUC time    | users + items  | skills + wins + fails | items + skills + wins + fails |\n|:------------|:---------------|:----------------------|:------------------------------|\n| LR          | **0.769** (IRT) 4.2s | 0.704 (PFA) 15s        | 0.747 48s                     |\n| (last) FM *d* = 5 | 0.742 1min14s   | 0.703 54s             | 0.730 1min48s                 |\n| (MCMC) FM *d* = 5 | 0.767 2min23s   | **0.705** 1min2s             | **0.749** 2min50s                 |\n\nComputation times are given for a i5 with 2.6 GHz in performance mode, with 200 epochs of FM training.\n\n## Improvement attempts\n\n### Efficient scikit-learn implementation for IRT\n\nCheck [sktm.py](sktm.py).\n\n```python\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import OneHotEncoder\nfrom sklearn.linear_model import LogisticRegression\n\npipe = Pipeline([\n    ('onehot', OneHotEncoder(handle_unknown='ignore')),\n    ('lr', LogisticRegression(solver='liblinear'))\n])\n\n# IRT\npipe.fit(df_train[['user', 'item']], df_train['correct'])\nprint(pipe.predict_proba(df_test[['user', 'item']]))\n```\n\nsktm contains efficient parallel cross validation over 5 folds, stratified by group (i.e. strong generalization).\n\nUsage:\n\n    mkdir data/assistments09\n    wget https://jiji.cat/weasel2018/data.csv -P data/assistments09  # Basically download it there\n    python sktm.py data/assistments09/data.csv --feat swf  # Choose which model, swf is PFA\n\nFor factorization machines, replace `LogisticRegression` with `from fm import FMClassifier`. There is a subtlety, please contact me to know more.\n\n---\n\nFor an online MIRT model:\n\n    python omirt.py --d 0 data/assist09/needed.csv  # Will load LR: coef0.npy\n    python omirt.py --d 5 data/assist09/needed.csv  # Will load FM: w.npy and V.npy\n\n    # Will train a IRT model on Fraction dataset with learning rate 0.01\n    python omirt.py --d 0 data/fraction/needed.csv --lr 0.01 --lr2 0.\n\nFor an IRT or deeper model with Keras, for batching and early stopping:\n\n    python dmirt.py data/assist09/needed.csv\n\nIt will also create a model.png file with the architecture (here just IRT with L2 regularization):\n\n![](model.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjilljenn%2Fktm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjilljenn%2Fktm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjilljenn%2Fktm/lists"}