{"id":26997179,"url":"https://github.com/adityalab/opentimer","last_synced_at":"2025-08-16T19:07:35.539Z","repository":{"id":279738874,"uuid":"938439275","full_name":"AdityaLab/OpenTimeR","owner":"AdityaLab","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-05T17:39:16.000Z","size":4121,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T02:17:02.000Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AdityaLab.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-25T00:42:43.000Z","updated_at":"2025-03-16T08:02:53.000Z","dependencies_parsed_at":"2025-02-27T06:33:19.257Z","dependency_job_id":"15495269-9f74-4d83-8ebd-6480fc6a8c88","html_url":"https://github.com/AdityaLab/OpenTimeR","commit_stats":null,"previous_names":["adityalab/opentimer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AdityaLab/OpenTimeR","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdityaLab%2FOpenTimeR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdityaLab%2FOpenTimeR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdityaLab%2FOpenTimeR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdityaLab%2FOpenTimeR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AdityaLab","download_url":"https://codeload.github.com/AdityaLab/OpenTimeR/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdityaLab%2FOpenTimeR/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266770140,"owners_count":23981512,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-04T02:17:06.082Z","updated_at":"2025-07-24T00:03:56.133Z","avatar_url":"https://github.com/AdityaLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/AdityaLab/OpenTimeR/blob/main/Picture/Logo-2-Rec4TS.jpg\" alt=\"Rec4TS-Logo\" style=\"width: 15%; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003e🔥 Rec4TS: Evaluating System 1 vs. 2 Reasoning Approaches for Zero-Shot Time-Series Forecasting 🔥\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2503.01895\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2503.01895-b31b1b.svg\" alt=\"arXiv\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\nREC4TS is the first benchmark that evaluates the effectiveness of reasoning strategies for zero-shot time series forecasting (TSF) tasks.\n\nSpecifically, REC4TS try to answer two research questions:\n\n-**RQ1: Can zero-shot TSF benefit from enhanced reasoning ability?**\n\n-**RQ2: What kind of reasoning strategies does zero-shot TSF need?**\n\nREC4TS covers three cognitive systems: Direct Sytem 1 (e.g. gpt-4o) ; the test-time enhanced System 1 (e.g., gpt-4o with Chain-of-Thought ) and System 2 (e.g. o1-mini).\n\n*Since reasoning strategies for foundational time-series models have not yet been studied and are difficult to implement directly, we have to reuse foundational language models to explore effective TSF\nreasoning strategies. We envision that our benchmark and insights offer promising potential for future research on understanding and designing effective reasoning strategies for zero-shot TSF.\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/AdityaLab/OpenTimeR/blob/main/Picture/reasoning_system_1.jpg\" width=\"800\"\u003e\n\u003c/div\u003e\n\n## Key  Insights\n\n0. Good News: reasoning is helpful for zero-shot TSF!\n1. Self-consistency is currently the most effective reasoning strategy\n2. Group-relative policy optimization enbaled DeepSeek-R1 is the only effective System 2 reasoning strategy\n3. Multimodal TSF benefits more from reasoning strategies than unimodal TSF\n4. The TimeT-hinking dataset:  containing reasoning trajectories of multiple advanced LLMs\n5. A new and simple test-time scaling on foundation time-series models：based on self-consistency reasoning strategies and inspired by our insights\n   \u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/AdityaLab/OpenTimeR/blob/main/Picture/Overall_Result_1.png\" width=\"500\"\u003e\n\u003c/div\u003e\n\n## Additional Toolkits\n\n- **TimeThinking**: Contains 1.5K filtered reasoning-annotated time series forecasting samples.\n- **Exp_Log**: Records all experimental results from benchmarking, including the complete outputs of various models and performance metrics.\n\n## Installation\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/yourusername/Rec4Time.git\ncd Rec4Time\n```\n\n2. Install dependencies:\n```bash\npip install -r requirements.txt\n```\n\n3. Set up your API keys:\n   - Open `A_Demo.py` or `A_Run_Exp.py` and add your OpenRouter API key:\n   ```python\n   openrouter_api_key = \"your_openrouter_api_key\"  # Replace with your API key\n   ```\n\n## Quick Start\n\n### Running a Demo\n\nTo run a simple demo with default parameters:\n\n```bash\npython A_Demo.py\n```\n\n### Running Experiments\n\nFor more control over experiment parameters, use the `A_Run_Exp.py` script:\n\n```bash\npython A_Run_Exp.py --future_months 6 --data_ids 0 1 2 --llm_ids 0 --multimodal 1\n```\n\n## Command-line Arguments\n\n`A_Run_Exp.py` supports the following command-line arguments:\n\n| Argument | Description | Default |\n|----------|-------------|---------|\n| `--future_months` | List of future months to predict | `[6]` |\n| `--data_ids` | List of dataset IDs to use | `[0]` |\n| `--llm_ids` | List of LLM IDs to use 0-OpenAI, 1-Gemini, 2-DeepSeek | `[0]` |\n| `--multimodal` | Whether to run multimodal experiments (0=No, 1=Yes) | `[0, 1]` |\n| `--repeat_times` | Number of times to repeat each experiment | `3` |\n| `--lookback_window_size` | Size of historical data window | `96` |\n| `--significant_figures` | Number of significant figures for data | `5` |\n| `--k_text` | Number of text samples to use | `10` |\n| `--unimodal_methods` | Unimodal reasoning strategies | `[\"naive\", \"cot\", \"self_consistency\", \"self_correction\", \"lrm\"]` |\n| `--multimodal_methods` | Multimodal reasoning strategies | `[\"naive\", \"cot\", \"self_consistency\", \"self_correction\", \"lrm\"]` |\n\n## Dataset IDs\n\nThe following datasets are available:\n\n| ID | Dataset Name |\n|----|--------------|\n| 0 | Retail Broiler Composit|\n| 1 | Drought Level |\n| 2 | International Trade Balance |\n| 3 | Gasoline Prices |\n| 4 | Influenza Patients Proportio  |\n| 5 | Disaster and Emergency Grant|\n| 6 | Unemployment Rate |\n| 7 | Travel Volume |\n\n## LLM IDs\n\nThe following sytem 1 and system 2 models are supported:\n\n| ID | Models |\n|----|--------------|\n| 0 | OpenAI (System 1: GPT-4o \u0026 Sytem 2: o1-mini) |\n| 1 | Google (System 1:  Gemini-2.0-flash \u0026 System 2:Gemini-2.0-flash-thinking) |\n| 2 | DeepSeek (System 1: DeepSeek-V3 \u0026 System 2: DeepSeek-R1) |\n\n## Reasoning Strategies\n\n- `naive`: Direct system 1 generation\n- `cot`: Chain-of-thought reasoning\n- `self_consistency`: Multiple predictions with averaging\n- `self_correction`: Self-correction of initial predictions\n- `lrm`: Using System 2 Models, also known as Large Reasoning Models\n\n## Experiment Results\n\nExperiment results are saved in the `experiment_results` directory in JSON format. \n\n## Collecting Results\n\nTo show experiment results, use the Jupyter notebook `A_DrawLatex.ipynb`. It will show the results in the form of LaTeX tables.\n\n\n\n### Custom Data\n\nTo use your own data, prepare CSV files with the following format:\n- Time series data: Date column and value column\n- Text data: Date column and fact/news column\n\nPlace your data files in the `data` directory.\n\n### Custom Models\n\nTo use different LLM providers, modify the `LLM_list` and `LRM_list` variables in `A_Demo.py` or `A_Run_Exp.py`:\n\n```python\nLRM_list = [\"openai/o1-mini-2024-09-12\", \"google/gemini-2.0-flash-thinking-exp-1219:free\", \"deepseek/deepseek-r1\"]\nLLM_list = ['openai/gpt-4o-2024-05-13', \"google/gemini-2.0-flash-001\", \"deepseek/deepseek-chat\"]\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## Citation\n\nIf you find this repo useful, please cite our paper.\n\n```\n@misc{liu2025evaluating1vs2,\n      title={Evaluating System 1 vs. 2 Reasoning Approaches for Zero-Shot Time-Series Forecasting: A Benchmark and Insights}, \n      author={Haoxin Liu and Zhiyuan Zhao and Shiduo Li and B. Aditya Prakash},\n      year={2025},\n      eprint={2503.01895},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2503.01895}, \n}\n```\n\n## Contact\nIf you have any questions or suggestions, feel free to contact:\nhliu763@gatech.edu\n\n## License\n\n\nThis project is licensed under the [CC-BY](https://creativecommons.org/licenses/by/4.0/) (Creative Commons Attribution 4.0 International) license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadityalab%2Fopentimer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadityalab%2Fopentimer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadityalab%2Fopentimer/lists"}