https://github.com/gip/yllama.oc

Fully On-chain Inference for LLMs
https://github.com/gip/yllama.oc

Last synced: 2 months ago
JSON representation

Fully On-chain Inference for LLMs

• Host: GitHub
• URL: https://github.com/gip/yllama.oc
• Owner: gip
• License: mit
• Created: 2024-06-19T18:32:55.000Z (5 months ago)
• Default Branch: main
• Last Pushed: 2024-07-11T22:10:02.000Z (4 months ago)
• Last Synced: 2024-07-12T06:50:27.729Z (4 months ago)
• Language: Rust
• Homepage:
• Size: 14.6 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-internet-computer - yllama - An on-chain Llama 3 inference engine optimized for real-world workloads using non-quantized 8b parameters Llama models. (Decentralized AI / Solana)

README

        
# yllama.oc

An on-chain implementation of inference for Llama 3 8b. 

## Overview

The project aims to create a generic yblock canister that's easily deployable and configurable on the Internet Computer Protocol (ICP). These yblock units serve as foundational components for uploading and executing AI algorithms, with the goal of distributing computation across a network of independent nodes. Consensus mechanisms ensure result accuracy without requiring knowledge of individual nodes.

In this implementation, the workload is distributed across 34 canisters. The code is currently unoptimized; future steps include reducing overhead and leveraging SIMD in WebAssembly.

The core algorithm is implemented in [here](https://github.com/gip/yllama.rs).

## Building

To build you will need:

* This repositiry [yllama.oc](https://github.com/gip/yllama.oc). It depends on `yllama.rs`.

* The [yllama.rs](https://github.com/gip/yllama.rs). It depends on `tokenizers`.

* A patched version of the Hugging Face [tokenizers](https://github.com/huggingface/tokenizers) repo. Crates built on getrandom need modification due to ICP's deterministic code execution and different randomness handling.

## Deploying and Running

[To be updated]

## Contact

[email protected]