https://github.com/nvidia/logits-processor-zoo

A collection of LogitsProcessors to customize and enhance LLM behavior for specific tasks.
https://github.com/nvidia/logits-processor-zoo

llm logits postprocessing

Last synced: 10 months ago
JSON representation

A collection of LogitsProcessors to customize and enhance LLM behavior for specific tasks.

• Host: GitHub
• URL: https://github.com/nvidia/logits-processor-zoo
• Owner: NVIDIA
• License: apache-2.0
• Created: 2024-11-15T17:57:09.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2025-03-20T09:59:29.000Z (10 months ago)
• Last Synced: 2025-04-04T05:41:50.234Z (10 months ago)
• Topics: llm, logits, postprocessing
• Language: Python
• Homepage:
• Size: 460 KB
• Stars: 261
• Watchers: 3
• Forks: 11
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

README

          
[![PyPI version](https://badge.fury.io/py/logits-processor-zoo.svg)](https://badge.fury.io/py/logits-processor-zoo)

[![License: MIT](https://img.shields.io/badge/License-Apache2.0-yellow.svg)](https://opensource.org/licenses/Apache2.0)



    



# logits-processor-zoo

Struggling to get LLMs to follow your instructions? LogitsProcessorZoo offers a zoo of tools to use LLMs for specific tasks, beyond just grammar enforcement!

## Installation

```bash

pip install logits-processor-zoo

```

## Supported Frameworks

* transformers

* vLLM

* TensorRT-LLM

## Usage

```python

import vllm

from logits_processor_zoo.vllm import GenLengthLogitsProcessor, CiteFromPromptLogitsProcessor, ForceLastPhraseLogitsProcessor

model = vllm.LLM(

            model_name,

            trust_remote_code=True,

            dtype="half",

            enforce_eager=True

        )

tokenizer = model.get_tokenizer()

        

logits_processors = [

    CiteFromPromptLogitsProcessor(tokenizer, boost_factor=2.0),

    GenLengthLogitsProcessor(tokenizer, boost_factor=-0.2, p=1),

    ForceLastPhraseLogitsProcessor("\n\nReferences:\n", tokenizer)

]

gen_output = model.generate(

            prompts,

            vllm.SamplingParams(

                n=1,

                temperature=0,

                seed=0,

                skip_special_tokens=True,

                max_tokens=64,

                logits_processors=logits_processors

            ),

            use_tqdm=False

        )

```

For the detailed examples in each framework, please have a look at **example_notebook** directory.

## Available Logits Processors

### GenLengthLogitsProcessor

A logits processor that adjusts the likelihood of the end-of-sequence (EOS) token based on the length of the generated sequence, encouraging or discouraging shorter answers.

### CiteFromPromptLogitsProcessor

A logits processor which boosts or diminishes the likelihood of tokens present in the prompt (and optionally EOS token) to encourage the model to generate tokens similar to those seen in the prompt or vice versa.

### ForceLastPhraseLogitsProcessor

A logits processor which forces LLMs to use the given phrase before they finalize their answers. Most common use cases can be providing references, thanking user with context etc.

### MultipleChoiceLogitsProcessor

A logits processor to answer multiple choice questions with one of the choices. A multiple choice question is like:

```

I am getting a lot of calls during the day. What is more important for me to consider when I buy a new phone?

0. Camera

1. Screen resolution

2. Operating System

3. Battery

```

The goal is to make LLM generate "3" as an answer.

### TriggerPhraseLogitsProcessor

A logits processor which triggers phrases when it encounters a given token.

One common use case is to force writing python code just after thinking:

```python

trigger_python = TriggerPhraseLogitsProcessor(phrase="\n```python", trigger_token_phrase="", 

                                              tokenizer=tokenizer, trigger_count=1, trigger_after=True)

```