{"id":20515426,"url":"https://github.com/snap-research/use","last_synced_at":"2025-07-07T06:35:53.946Z","repository":{"id":232872396,"uuid":"785356607","full_name":"snap-research/USE","owner":"snap-research","description":"USE: Dynamic User Modeling with Stateful Sequence Models","archived":false,"fork":false,"pushed_at":"2024-04-17T08:38:48.000Z","size":225,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-11-15T21:25:20.260Z","etag":null,"topics":["behavior-modeling","recommendation","sequence-models","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snap-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-11T18:15:29.000Z","updated_at":"2024-11-04T09:54:11.000Z","dependencies_parsed_at":"2024-04-17T09:43:03.308Z","dependency_job_id":"1851a40d-476a-48bb-8408-52f42c339ae8","html_url":"https://github.com/snap-research/USE","commit_stats":null,"previous_names":["snap-research/use"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FUSE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FUSE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FUSE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FUSE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snap-research","download_url":"https://codeload.github.com/snap-research/USE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234177738,"owners_count":18791620,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["behavior-modeling","recommendation","sequence-models","transformer"],"created_at":"2024-11-15T21:21:39.859Z","updated_at":"2025-01-16T09:31:45.272Z","avatar_url":"https://github.com/snap-research.png","language":"Python","readme":"# USE: Dynamic User Modeling with Stateful Sequence Models\n\nThe repo contains the official implementation of [USE: Dynamic User Modeling with Stateful Sequence Models](https://arxiv.org/abs/2403.13344)\n\n\n\n## Contents\n\n- [1. Introduction](#1-introduction)\n- [2. Data](#2-data)\n- [3. Setup Environment](#3-setup-environment)\n- [4. Pre-Training](#4-pre-training)\n- [5. Finetune](#5-finetune)\n- [6. Citation](#6-citation)\n\n\n\n## 1. Introduction\n\nIn this work, we propose User Stateful Embedding (USE), a behavior-based user model that seamlessly integrate historical and recent user behaviors in computing general-purpose user embedding that contains both long-term and short-term user bahevior pattern. We perform comprehensive empirical studies on Snapchat user behavior sequences to verify the effectiveness of USE.\n\n\n\n![figure1](./figures/figure1.png)\n\n\n\n![figure2](./figures/figure2.png)\n\n\n\n## 2. Data\n\nDue to privacy and legal reasons, we are not able to share the Snapchat user behavior sequence data we used in the paper. To illustrate USE, we use two open-sourced dataset: \n\n### 2.1 Media Interview\n\nThis dataset [1] consists of 463.6K transcripts with abstractive summaries collected from interview transcripts from NPR and CNN. On this dataset, we consider each interview as a user behavior sequence, while each character is a user behavior for simplicity. We train each model for 5 epochs. For evaluation, we keep 5000 sequences aside from model training and consider two dynamic user modeling tasks:\n\n[1] Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization. NAACL 2021\n\n\n\n### Dataset 2.2: Genome sequence\n\nThis dataset consists of 2 million DNA sequences from about 30k distinct species. It is not a user modeling dataset, yet we use it to further demonstrate the generalizability of USE. We consider each DNA sequence as a user sequence. We train each model for 1 epoch. In evaluation, we create a dataset with 5000 DNA sequence from 5000 distinct species that are not inlcuded in the training set. We perform the same two tasks as we did above.\n\n\n\nThe data can be downloaded [here](https://drive.google.com/drive/folders/19nHO3-ub79BHMromChMN8dD1D4Y8Dznp?usp=sharing).\n\n\n\n## 3. Setup environment\n\n    # create a virtual python environment from the template\n    conda env create -f environment.yml\n\n\n\n\n## 4. Pre-Training\n\n```\nexport DATA_DIR=/path/to/txt\nexport DATA_TYPE=dna # choose from [\"dialog\", \"dna\"]\nexport batch_size=42\nexport min_l=500\nexport max_l=512\nexport epoch=6\nexport gap=50\nexport learning_rate=4e-4\nexport gradient_accumulation_steps=2\nexport model=gpt2\nexport num_heads=12\nexport contrastive_embedding=token\nexport contrastive_type=Orig\nexport obj=FEP_SUP # select from CLM / FEP / SUP / FEP_SUP\nexport fep_window_size=5\nexport fep_context_length=0\nexport fep_loss_frequency=1\nexport run_name=retnet_${obj}\n\npython train_user_model.py \\\n    --run-name ${run_name} \\\n    --data-dir ${DATA_DIR} \\\n    --n-iter 1 \\\n    --num-train-epochs ${epoch} \\\n    --min-model-input-size ${min_l} \\\n    --max-model-input-size ${max_l} \\\n    --user-segments-gap ${gap} \\\n    --per-device-batch-size ${batch_size} \\\n    --test-size 0.01 \\\n    --warmup-ratio 0.06 \\\n    --learning-rate ${learning_rate} \\\n    --gradient-accumulation-steps ${gradient_accumulation_steps} \\\n    --tokenizer-path ./tokenizer/${DATA_TYPE} \\\n    --model-output-path ./artifacts_${DATA_TYPE}/${run_name}/clm \\\n    --fep-window-size ${fep_window_size} \\\n    --eval-steps 50000 \\\n    --save-steps 500 \\\n    --cache-dir cache_${DATA_TYPE} \\\n    --model ${model} \\\n    --num-heads ${num_heads} \\\n    --contrastive-embedding ${contrastive_embedding} \\\n    --contrastive-type ${contrastive_type} \\\n    --fep-context-length ${fep_context_length} \\\n    --fep-loss-frequency ${fep_loss_frequency} \\\n    --get-random-segment \\\n    --training-objective ${obj}\n```\n\n\n\n\n\n\n\n## 5. Evaluation\n\n### 5.1 Dynamic -\u003e User Re-ID\n\n\n```\nexport DATA_PATH=/path/to/tsv/file\nexport DATA_TYPE=dna # choose from [\"dialog\", \"dna\"]\n\npython eval_reid.py \\\n\t\t--model_path /path/to/trained/model \\\n\t\t--dataset_dir ${DATA_PATH} \\\n\t\t--tokenizer_path ./tokenizer/${DATA_TYPE}\n```\n\n\n\n### 5.2 Dynamic -\u003e User Classification\n\n```\nexport DATA_PATH=/path/to/tsv/file\nexport DATA_TYPE=dna # choose from [\"dialog\", \"dna\"]\n\npython eval_classification.py \\\n\t\t--model_path /path/to/trained/model \\\n\t\t--dataset_dir ${DATA_PATH} \\\n\t\t--tokenizer_path ./tokenizer/${DATA_TYPE}\n```\n\n\n\n\n\n## 6. Citation\n\nIf you have any question regarding our paper or codes, please feel free to start an issue or reach out the authors.\n\n\nIf you use USE in your work, please kindly cite our paper:\n\n```\n@misc{zhou2024use,\n      title={USE: Dynamic User Modeling with Stateful Sequence Models}, \n      author={Zhihan Zhou and Qixiang Fang and Leonardo Neves and Francesco Barbieri and Yozen Liu and Han Liu and Maarten W. Bos and Ron Dotsch},\n      year={2024},\n      eprint={2403.13344},\n      archivePrefix={arXiv},\n      primaryClass={cs.SI}\n}\n```\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-research%2Fuse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnap-research%2Fuse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-research%2Fuse/lists"}