{"id":17596212,"url":"https://github.com/lamm-mit/preflexor","last_synced_at":"2025-04-06T23:16:11.119Z","repository":{"id":258486975,"uuid":"868038887","full_name":"lamm-mit/PRefLexOR","owner":"lamm-mit","description":"Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning","archived":false,"fork":false,"pushed_at":"2025-02-24T09:40:41.000Z","size":664,"stargazers_count":196,"open_issues_count":1,"forks_count":29,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-30T22:09:44.347Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lamm-mit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-05T10:10:47.000Z","updated_at":"2025-03-30T19:59:32.000Z","dependencies_parsed_at":"2024-10-19T08:24:09.348Z","dependency_job_id":"4c177162-9c88-40aa-afbb-642b4a7e6cf9","html_url":"https://github.com/lamm-mit/PRefLexOR","commit_stats":{"total_commits":52,"total_committers":1,"mean_commits":52.0,"dds":0.0,"last_synced_commit":"71654578f40ef18a7956c1af181ce516f4790143"},"previous_names":["lamm-mit/preflexor"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamm-mit%2FPRefLexOR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamm-mit%2FPRefLexOR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamm-mit%2FPRefLexOR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamm-mit%2FPRefLexOR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lamm-mit","download_url":"https://codeload.github.com/lamm-mit/PRefLexOR/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247563935,"owners_count":20958971,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-22T08:24:22.094Z","updated_at":"2025-04-06T23:16:11.087Z","avatar_url":"https://github.com/lamm-mit.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking\n\nWe introduce PRefLexOR (Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning), a framework that combines preference optimization with concepts from Reinforcement Learning (RL) to enable models to self-teach through iterative reasoning improvements. Central to PRefLexOR are thinking tokens, which explicitly mark reflective reasoning phases within model outputs, allowing the model to recursively engage in multi-step reasoning, revisiting, and refining intermediate steps before producing a final output. The foundation of PRefLexOR lies in Odds Ratio Preference Optimization (ORPO), where the model learns to align its reasoning with human-preferred decision paths by optimizing the log odds between preferred and non-preferred responses. The integration of Direct Preference Optimization (DPO) further enhances model performance by using rejection sampling to fine-tune reasoning quality, ensuring nuanced preference alignment. This hybrid approach between ORPO and DPO mirrors key aspects of RL, where the model is continuously guided by feedback to improve decision-making and reasoning. Active learning mechanisms allow PRefLexOR to dynamically generate new tasks, reasoning steps, and rejected answers on-the-fly during training. This adaptive process enables the model to self-teach as it continually improves through real-time feedback and recursive processing. \n\nOur method diverges from traditional approaches by not relying on pre-generated datasets; instead, it dynamically generates new tasks, reasoning steps, and feedback on the fly, allowing the model to continuously adapt and improve in real time. Recursive optimization within the thinking token framework introduces iterative feedback loops, where the model refines its reasoning, much like policy refinement in RL, achieving deeper coherence, consistency, and adaptability. By recursively optimizing reasoning through feedback-driven learning, PRefLexOR achieves significant flexibility in its ability to handle complex tasks, learning and evolving its cognitive abilities autonomously. This framework advances the field of cognitive alignment by demonstrating that models can iteratively teach themselves to reason with greater depth and reflectivity, akin to an RL-based self-improving system capable of solving open-domain problems with superior reasoning depth and logic. Our implementation is straightforward and can be Incorporated into any existing pretrained LLM. The approach is demonstrated in use cases of materials design applications, where a small language model is trained to develop sophisticated reasoning capabilities. Thereby, PRefLexOR builds a dynamic knowledge graph by generating questions from random text and using Retrieval-Augmented Generation (RAG) to retrieve contextually relevant data from the entire corpus, facilitating recursive reasoning through complex interactions between similar nodes in the embedding space.\n\n\n![Fig_100](https://github.com/user-attachments/assets/800de09d-64c4-4ead-903f-80525f8bf415)\n\nFigure 1: Illustration of the workflow and design principles behind generative materials informatics. Panel a: The process of transforming information into knowledge and actionable outcomes. Each individual piece of information (left) is synthesized into a network of interconnected knowledge, leading to informed decisions and innovative designs (right). Panel b: Conventional approaches in materials science rely on data-driven models, partial differential equations (PDEs), and experimental results, focusing on single-step predictions. Panel c: In contrast, generative materials informatics models built on the PRefLexOR framework proposed in this paper use 'thinking' and 'reflection' explicitly by incorporating iterative reasoning and contextual understanding, allowing for more complex, multi-step predictions. This approach expands from single inference steps, includes multiple modalities of data and responses, integrates real-world feedback and physics, and leverages self-assessment and self-learning. Using using reinforcement learning (RL) principles, the discovery of principles or the solution of specific tasks is further inspired by biological paradigms, using bio-inspired neural network designs. These advanced methods support continuous improvement in material predictions, enabling more adaptable and intelligent designs\n\n![image](https://github.com/user-attachments/assets/1119b9f7-5f45-4712-81a5-11699a02c571)\n\nFigure 2: PRefLexOR Recursive Reasoning Algorithm: An iterative approach leveraging a fine-tuned Reasoning Model and a general-purpose Critic Model to generate, refine, and optionally integrate responses. The process involves generating initial responses, extracting reflections, improving thinking processes, and creating new responses based on refined thinking, with an optional final integration step. The algorithm relies on extracting thinking processes (indicated via ```\u003c|thinking|\u003e...\u003c|/thinking|\u003e```) and reflection processes  (indicated via ```\u003c|reflect|\u003e...\u003c|/reflect|\u003e```). The use of special tokens allows us to easily construct such agentic modeling as it facilitates pausing inference, improving the strategy, and re-generating improved answers. The sampled responses can either be used in their final state or integrated into an amalgamated response that shows very rich facets in the scientific process.  \n\n![image](https://github.com/user-attachments/assets/b63a5a7c-465e-4117-aed7-10f74378eed7)\n\n## Installation\n\nInstall:\n```bash\npip install git+https://github.com/lamm-mit/PRefLexOR.git\n\n```\nIf you want to create an editable installation, clone the repository using `git`:\n```bash\ngit clone https://github.com/lamm-mit/PRefLexOR.git\ncd PRefLexOR\n```\nThen, install:\n```bash\npip install -r requirements.txt\npip install -e .\n```\n\n### Flash Attention\n\nInstall flash attention (https://github.com/Dao-AILab/flash-attention) if you want to use that option:\n```bash\nMAX_JOBS=4 pip install flash-attn --no-build-isolation\n```\n\n### Directory structure\n```\nPRefLexOR/\n│\n├── PRefLexOR/               # Source code directory\n│   ├── active_trainer.py\n│   ├── inference.py\n│   └── utils.py\n│\n├── setup.py                 # The setup file for packaging\n├── requirements.txt         # List of dependencies\n├── README.md                # Documentation\n└── ...\n```\n\n## Example codes\nMore will be added shortly, including full notebooks. Here are code snippets that show how the trainers are initialized and used. \n\n\u003cimg width=\"517\" alt=\"image\" src=\"https://github.com/user-attachments/assets/622de5bb-e446-4814-a356-7131ea13b184\"\u003e\n\nFigure 3: Overview of the PRefLexOR algorithm, consisting of Base Model Pre-training/Incipient Fine-tuning, Structured Thought Integration Training, Independent Reasoning Development, and the Recursive Reasoning Algorithm. Each phase can be scaled independently with additional compute to improve performance.\n\n## Model weights\n\nModel weightsare available via Hugging Face. For example:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nmodel_name='lamm-mit/PRefLexOR_ORPO_DPO_EXO_10242024'\nmodel = AutoModelForCausalLM.from_pretrained(model_name,     \n    torch_dtype =torch.bfloat16,\n    attn_implementation=\"flash_attention_2\",device_map=\"auto\",trust_remote_code=True,\n    )\ntokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True,\n                                          use_fast=False,\n                                         )\n```\n\n\n### PRefLexOR Structured Thought Integration Training via Odds Ratio Preference Optimization (ORPO) phase\n\n```python\nfrom trl import ORPOConfig\nfrom transformers import TrainingArguments\nfrom datasets import load_dataset, concatenate_datasets\n\n# Import PRefLexOR trainer classes and utils\nfrom PRefLexOR import *\nfrom utils import *\n\n# Configuration\nFT_model_name = 'PRefLexOR_ORPO_Model'\nrepo_ID='lamm-mit'\nmax_prompt_length = 512\nmax_length = 2048\nprivate = True #whether or not HF repos are private\n\nthink_start='\u003c|thinking|\u003e'\nthink_end='\u003c|/thinking|\u003e'\n\n# Adjust learning rate based on LoRA usage\nlearning_rate = 5e-5 if use_LoRA else 5e-6\n\n# ORPO Configuration\ncfg = ORPOConfig(\n    output_dir=FT_model_name,               # Output directory\n    num_train_epochs=1,                     # Number of training epochs\n    per_device_train_batch_size=1,          # Batch size per device during training\n    gradient_accumulation_steps=2,          # Steps before a backward/update pass\n    gradient_checkpointing=False,           # Use gradient checkpointing to save memory\n    optim=\"adamw_torch_fused\",              # Fused adamw optimizer\n    logging_steps=10,                       # Log every X steps\n    bf16=True,                              # Use bfloat16 precision\n    learning_rate=learning_rate,            # Learning rate\n    warmup_ratio=0,                         # Warmup ratio\n    warmup_steps=0,                         # Warmup steps\n    lr_scheduler_type=\"constant\",           # Learning rate scheduler type\n    max_prompt_length=max_prompt_length,    # Max length for prompts\n    remove_unused_columns=False,\n    max_length=max_length,                  # Max length for outputs\n    beta=0.1,                               # ORPO beta\n    save_total_limit=3,                     # Limit on total saved models\n    save_strategy=\"no\",                     # Save strategy\n    report_to=['none'],                     # Reporting\n    #hub_private_repo=private,                  # Use a private hub repo\n    #hub_model_id=f'{repo_ID}/{FT_model_name}' # Hub model ID\n)\n\n# Dataset and training parameters\ntopics = 50\nnum_questions_per_topic = 1\nnum_epochs_per_dataset_generation = 2\n\n# Calculate number of steps\nif isinstance(topics, list) and all(isinstance(t, str) for t in topics):\n    n_steps = len(topics) * num_questions_per_topic * num_epochs_per_dataset_generation\nelse:\n    n_steps = topics * num_questions_per_topic * num_epochs_per_dataset_generation\n\n# Trainer setup\ntrainer = PRefLexORORPOTrainer(\n    model=model,\n    args=cfg,\n    train_dataset=temp,\n    tokenizer=tokenizer,\n    n_steps=n_steps,                        # Train for n_steps before updating dataset\n    topics=topics,\n    number_nodes_to_get=3,\n    n_questions_for_each=num_questions_per_topic,\n    only_include_wrong_answers=False,\n    process=process,\n    generate_dataset=generate_dataset,\n    generate=generate_GPT_MistralRS,        # Function for generating datasets\n    index=index,\n    get_rejected_from_trained_model=True,\n)\n```\n\nTraining loop:\n\n```python\n# Configuration\nsystem_prompt = 'You are a materials scientist.'\nnum_iterations = 50  # Number of iterations for the training loop\n\n# Training Loop\nfor iteration in range(num_iterations):\n    print(f\"Starting iteration {iteration + 1}/{num_iterations}\")\n    \n    # Train for N steps (no specific steps defined here, but you can update it if needed to train for different steps in different iterations)\n    n_steps = None\n    trainer.train(n_steps=n_steps)\n    \n    print(\"#\" * 64)\n    \n    # Prompts and text generation examples\n    prompts = [\n        f'Tell me why hierarchical structures work so well. Use {think_start}.',\n        f'What is the relationship between materials and music? Use {think_start}.',\n    ]\n    \n    for txt in prompts:\n        output_text, _ = generate_local_model(\n            model=model,\n            tokenizer=tokenizer,\n            prompt=txt,\n            system_prompt=system_prompt,\n            prepend_response=f'{think_start}' if \"Use\" in txt else '',\n            num_return_sequences=1,\n            repetition_penalty=1.0,\n            temperature=0.1,\n            max_new_tokens=1024,\n            messages=[],\n            do_sample=True,\n        )\n        print(output_text)\n        print(\"-\" * 64)\n    \n    # Save the model\n    trainer.save_model(f\"./{FT_model_name}\")\n    model.push_to_hub(f\"lamm-mit/{FT_model_name}\", private=private)\n    tokenizer.push_to_hub(f\"lamm-mit/{FT_model_name}\", private=private)\n    \n    # Update the dataset\n    trainer.update_dataset()\n    \n    print(f\"Completed iteration {iteration + 1}/{num_iterations}\")\n    print(\"#\" * 64)\n```\n### PRefLexOR Independent Reasoning Development Phase via Efficient Exact Optimization (EXO)\n\n```python\nimport json\nfrom trl import DPOConfig, DPOTrainer\nfrom transformers import TrainingArguments\nfrom datasets import load_dataset, concatenate_datasets\nfrom PRefLexOR import *\n\n# Reward Logging Callback\nclass RewardLoggingCallback(TrainerCallback):\n    def on_log(self, args, state, control, logs=None, **kwargs):\n        # Safely access and print the last log entry\n        if state.log_history:\n            try:\n                print(f\"Step={state.log_history[-1]['step']}\",\n                      \"rewards/margins=\", state.log_history[-1]['rewards/margins'],\n                      \"loss=\", state.log_history[-1]['loss'],\n                      \"rewards/accuracy=\", state.log_history[-1]['rewards/accuracies'])\n            except KeyError:\n                print(end='')\n\n# Model and configuration settings\nFT_model_name = 'PRefLexOR_EXO_Model'\nrepo_id = 'lamm-mit'\n\nthink_start='\u003c|thinking|\u003e'\nthink_end='\u003c|/thinking|\u003e'\n\ncfg = DPOConfig(\n    output_dir=FT_model_name,              # Output directory\n    num_train_epochs=1,                    # Number of training epochs\n    per_device_train_batch_size=1,         # Batch size per device during training\n    gradient_accumulation_steps=2,         # Steps before a backward/update pass\n    gradient_checkpointing=False,          # Gradient checkpointing\n    optim=\"adamw_torch_fused\",             # Optimizer type\n    logging_steps=10,                      # Log every X steps\n    bf16=True,                             # Use bfloat16 precision\n    max_grad_norm=0.3,                     # Max gradient norm\n    learning_rate=5e-7,                    # Learning rate\n    warmup_ratio=0,\n    warmup_steps=0,\n    lr_scheduler_type=\"constant\",          # LR scheduler type\n    max_prompt_length=512,\n    max_length=2000,\n    remove_unused_columns=False,\n    beta=0.1,                              # DPO beta\n    save_total_limit=50,                   # Save limit\n    save_strategy=\"epoch\",\n    report_to=['none'],                    # Reporting\n    #hub_private_repo=True,                 # Private hub repo\n    #hub_model_id=f'lamm-mit/{FT_model_name}',\n    loss_type=\"exo_pair\",                  # Loss type for DPO\n    label_smoothing=5e-3,\n)\n\n# Dataset and training parameters\ntopics = 50\nnum_questions_per_topic = 1\nnum_epochs_per_dataset_generation = 2\nprivate = True #whether or not HF repos are private\n\n# Calculate number of steps\nif isinstance(topics, list) and all(isinstance(t, str) for t in topics):\n    n_steps = len(topics) * num_questions_per_topic * num_epochs_per_dataset_generation\nelse:\n    n_steps = topics * num_questions_per_topic * num_epochs_per_dataset_generation\n\n# Trainer setup\ntrainer =PRefLexORDPOTrainer(\n    model=model,\n    ref_model=ref_model,                      # Set to None if using PEFT\n    args=cfg,\n    train_dataset=temp,                        # Temporary training dataset\n    tokenizer=tokenizer,\n    n_steps=n_steps,                           # Train for n_steps before updating dataset\n    topics=topics,\n    number_nodes_to_get=3,\n    n_questions_for_each=num_questions_per_topic,\n    only_include_wrong_answers=False,\n    process=process,\n    generate_dataset=generate_dataset,\n    generate=generate_GPT_MistralRS,          # Function for generating datasets\n    get_rejected_from_trained_model=True,\n    index=index,\n    \n    # Dynamic Answer Comparison\n    dynamic_answer_comparison=True,           # Option for dynamic comparison\n    \n    # Mask Thinking Tokens Options\n    mask_thinking_tokens=False,               # Whether to mask thinking tokens\n    thinking_token_mask_percentage=0.2,       # Percentage of thinking tokens to mask in thinking sections\n\n    # Thinking Tokens\n    think_start_token=think_start,\n    think_end_token=think_end,\n    include_thinking_token_in_labels=True,\n\n    # Callbacks\n    callbacks=[RewardLoggingCallback()],\n)\n```\n\nTraining loop:\n```python\nimport json\n\n# Configuration\nnum_iterations = 50\n\n# Training Loop\nfor iteration in range(num_iterations):\n    print(f\"Starting iteration {iteration + 1}/{num_iterations}\")\n    \n    # Train for the current iteration\n    trainer.train()\n    \n    print(\"#\" * 64)\n    \n    # Prompts and text generation\n    prompts = [\n        f'Tell me why hierarchical structures work so well. Use {think_start}.',\n        f'Explain the relationship between materials and music. Use {think_start}.'\n    ]\n    \n    for txt in prompts:\n        output_text, _ = generate_local_model(\n            model=model,\n            tokenizer=tokenizer,\n            prompt=txt,\n            system_prompt=system_prompt,\n            prepend_response=f'{think_start}' if \"Use\" in txt else '',\n            num_return_sequences=1,\n            repetition_penalty=1.0,\n            temperature=0.1,\n            max_new_tokens=1024,\n            messages=[],\n            do_sample=True,\n        )\n        print(output_text)\n        print(\"-\" * 64)\n\n    # Save the model\n    trainer.save_model(f\"./{FT_model_name}\")\n    model.push_to_hub(f\"{repo_id}/{FT_model_name}\", private=private, commit_message=f'iteration_{iteration + 1}')\n    tokenizer.push_to_hub(f\"{repo_id}/{FT_model_name}\", private=private, commit_message=f'iteration_{iteration + 1}')\n\n    # Save training logs\n    try:\n        with open(\"trainer_log_history.txt\", \"w\") as f:\n            json.dump(trainer.log_history, f)\n        \n        # Path to the file and repository ID\n        file_path = \"trainer_log_history.txt\"\n        repo_id = f\"lamm-mit/{model_current}\"\n        \n        # Upload the file\n        api.upload_file(\n            path_or_fileobj=file_path,\n            path_in_repo=\"trainer_log_history.txt\",\n            repo_id=repo_id,\n            repo_type=\"model\",\n            commit_message=\"Upload trainer log history\"\n        )\n\n    except Exception as e:\n        print(\"Could not push training logs:\", e)\n\n    # Save dataset logs\n    try:\n        temp_data = trainer.concatenated_train_dataset\n        temp_data.push_to_hub(f\"lamm-mit/{model_current}_data\", private=True)\n    except Exception as e:\n        print(\"Could not push dataset logs:\", e)\n\n    # Update the dataset\n    trainer.update_dataset()\n\n    print(f\"Completed iteration {iteration + 1}/{num_iterations}\")\n    print(\"#\" * 64)\n```\n## Colab example: Inference notebooks\n\n#### PRefLexOR Inference: Thinking and Agentic Reflection \n[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lamm-mit/PRefLexOR/blob/main/PRefLexOR_inference_thinking.ipynb)\n\n#### PRefLexOR Inference: Thinking and Reflection and Agentic Reasoning \n[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lamm-mit/PRefLexOR/blob/main/PRefLexOR_inference_thinking-reflection.ipynb)\n\n## Graph-PRefLexOR\n\n![image](https://github.com/user-attachments/assets/e98d120a-067e-4825-a6c4-64c8f0cde8f8)\n\nSample graph generated based on autonomous extended reasoning (see: ```https://arxiv.org/abs/2502.13025```). \n\n### References\n\n```bibtex\n@article{buehler2024PRefLexOR,\n      title={PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking}, \n      author={Markus J. Buehler},\n      year={2024},\n      eprint={2410.12375},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2410.12375}, \n}\n\n@misc{buehler2025insitugraphreasoningknowledge,\n      title={In-situ graph reasoning and knowledge expansion using Graph-PReFLexOR}, \n      author={Markus J. Buehler},\n      year={2025},\n      eprint={2501.08120},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2501.08120}, \n}\n\n@misc{buehler2025agentic,\n    title={Agentic Deep Graph Reasoning Yields Self-Organizing Knowledge Networks},\n    author={Markus J. Buehler},\n    year={2025},\n    eprint={2502.13025},\n    archivePrefix={arXiv},\n    primaryClass={cs.AI}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flamm-mit%2Fpreflexor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flamm-mit%2Fpreflexor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flamm-mit%2Fpreflexor/lists"}