https://github.com/laion-ai/open-sci-hf

https://github.com/laion-ai/open-sci-hf

Last synced: about 1 month ago
JSON representation

• Host: GitHub
• URL: https://github.com/laion-ai/open-sci-hf
• Owner: LAION-AI
• Created: 2025-02-11T10:40:30.000Z (4 months ago)
• Default Branch: main
• Last Pushed: 2025-02-13T15:30:26.000Z (4 months ago)
• Last Synced: 2025-05-07T18:13:38.908Z (about 1 month ago)
• Language: Python
• Size: 962 KB
• Stars: 1
• Watchers: 5
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Opensci model hf

```bash

pip install -r requirements.txt

```

```bash

python example.py

```

```

OpensciForCausalLM(

  (model): OpensciModel(

    (embed_tokens): Embedding(50304, 2048)

    (layers): ModuleList(

      (0-23): 24 x OpensciDecoderLayer(

        (self_attn): OpensciAttention(

          (q_proj): Linear(in_features=2048, out_features=2048, bias=True)

          (k_proj): Linear(in_features=2048, out_features=2048, bias=True)

          (v_proj): Linear(in_features=2048, out_features=2048, bias=True)

          (o_proj): Linear(in_features=2048, out_features=2048, bias=True)

          (q_norm): OpensciRMSNorm((64,), eps=1e-05)

          (k_norm): OpensciRMSNorm((64,), eps=1e-05)

        )

        (mlp): OpensciMLP(

          (gate_proj): Linear(in_features=2048, out_features=5440, bias=True)

          (up_proj): Linear(in_features=2048, out_features=5440, bias=True)

          (down_proj): Linear(in_features=5440, out_features=2048, bias=True)

          (act_fn): SiLU()

        )

        (input_layernorm): OpensciRMSNorm((2048,), eps=1e-05)

        (post_attention_layernorm): OpensciRMSNorm((2048,), eps=1e-05)

      )

    )

    (norm): OpensciRMSNorm((2048,), eps=1e-05)

    (rotary_emb): OpensciRotaryEmbedding()

  )

  (lm_head): Linear(in_features=2048, out_features=50304, bias=False)

)

```