https://github.com/whitead/lm-seqopt

https://github.com/whitead/lm-seqopt

Last synced: about 1 month ago
JSON representation

• Host: GitHub
• URL: https://github.com/whitead/lm-seqopt
• Owner: whitead
• Created: 2023-12-12T06:41:36.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2023-12-12T07:57:49.000Z (about 1 year ago)
• Last Synced: 2024-10-16T02:59:08.179Z (3 months ago)
• Language: Python
• Size: 3.91 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Sequence Optimization

Messing around with sequence optimization in PyTorch

```python

# Example Usage

num_embeddings = 10

embedding_dim = 5

# Create a DifferentiableEmbedding instance

diff_embedding = DifferentiableEmbedding(num_embeddings, embedding_dim)

# Example input - using Gumbel Softmax to get indices

logits = torch.randn(3, num_embeddings, requires_grad=True)

# Forward pass

output = diff_embedding(logits)

# Compute gradients

loss = output.sum()

loss.backward()

print(logits.grad)

```

## Reference Implementations

Wrapper for PyTroch modules to output gradients of input tokens:

https://github.com/QData/TextAttack/blob/57bc36cc622e8c1a993d728066cb9f42cdec217d/textattack/models/wrappers/pytorch_model_wrapper.py#L50

Implementation for one-hot encoding multiplication by embedding matrix in jax:

https://github.com/ur-whitelab/wazy/blob/529eac8b473b9f17d6ff7824230b8fcf35fb99c3/wazy/utils.py#L89

Another one in pytorch (actually really good one)

https://github.com/llm-attacks/llm-attacks/blob/main/llm_attacks/minimal_gcg/opt_utils.py#L11