https://github.com/metaskills/llamafile-on-lambda

Serverless AI Inference with Gemma 2 using Mozilla's llamafile on AWS Lambda
https://github.com/metaskills/llamafile-on-lambda

ai aws-lambda gemma gemma2 lambda llamafile

Last synced: 10 days ago
JSON representation

Serverless AI Inference with Gemma 2 using Mozilla's llamafile on AWS Lambda

• Host: GitHub
• URL: https://github.com/metaskills/llamafile-on-lambda
• Owner: metaskills
• License: mit
• Created: 2024-07-05T12:03:09.000Z (6 months ago)
• Default Branch: main
• Last Pushed: 2024-07-30T01:54:43.000Z (5 months ago)
• Last Synced: 2024-12-08T21:52:23.473Z (13 days ago)
• Topics: ai, aws-lambda, gemma, gemma2, lambda, llamafile
• Language: JavaScript
• Homepage: https://www.unremarkable.ai/serverless-ai-inference-with-gemma-2-using-mozillas-llamafile-on-aws-lambda
• Size: 864 KB
• Stars: 11
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
# Serverless AI Inference with Gemma 2 using Mozilla's llamafile on AWS Lambda

https://www.unremarkable.ai/serverless-ai-inference-with-gemma-2-using-mozillas-llamafile-on-aws-lambda

![Llamafile on AWS Lambda](public/llamafile-on-lambda.jpg)

## Setup

We assume you have the following configured or installed.

1. An AWS account with credentials configured.

2. The [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) installed for fast and easy serverless deployments.

3. Docker [installed](https://www.docker.com) for easy container builds and deployments.

After you clone the repo, setup your dependencies with the following command:

```shell

npm install

```

## Usage

Now you can run the following commands from the root directory.

* `./bin/build` - To download  and build a llamafile container for deployment.

* `./bin/server` - To run the download (above) llamafile server locally.

* `./bin/deploy` - Deploy to AWS Lambda. Also does a build if needed.

## Chat

This project uses [Inquirer.js](https://www.npmjs.com/package/inquirer) to chat with the model using OpenAI's API. The model can be running locally using `bin/server` or deployed to Lambda using `bin/deploy`. Inquirer will ask for your local or function URL at the beginning of the chat session.

![Llamafile on AWS Lambda](public/llamafile-on-lambda.gif)