Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/metaskills/llamafile-on-lambda
Serverless AI Inference with Gemma 2 using Mozilla's llamafile on AWS Lambda
https://github.com/metaskills/llamafile-on-lambda
ai aws-lambda gemma gemma2 lambda llamafile
Last synced: about 20 hours ago
JSON representation
Serverless AI Inference with Gemma 2 using Mozilla's llamafile on AWS Lambda
- Host: GitHub
- URL: https://github.com/metaskills/llamafile-on-lambda
- Owner: metaskills
- License: mit
- Created: 2024-07-05T12:03:09.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2024-07-30T01:54:43.000Z (2 months ago)
- Last Synced: 2024-07-30T06:17:47.837Z (2 months ago)
- Topics: ai, aws-lambda, gemma, gemma2, lambda, llamafile
- Language: JavaScript
- Homepage: https://www.unremarkable.ai/serverless-ai-inference-with-gemma-2-using-mozillas-llamafile-on-aws-lambda
- Size: 864 KB
- Stars: 10
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Serverless AI Inference with Gemma 2 using Mozilla's llamafile on AWS Lambda
https://www.unremarkable.ai/serverless-ai-inference-with-gemma-2-using-mozillas-llamafile-on-aws-lambda
## Setup
We assume you have the following configured or installed.
1. An AWS account with credentials configured.
2. The [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) installed for fast and easy serverless deployments.
3. Docker [installed](https://www.docker.com) for easy container builds and deployments.
After you clone the repo, setup your dependencies with the following command:
```shell
npm install
```
## Usage
Now you can run the following commands from the root directory.
* `./bin/build` - To download and build a llamafile container for deployment.
* `./bin/server` - To run the download (above) llamafile server locally.
* `./bin/deploy` - Deploy to AWS Lambda. Also does a build if needed.
## Chat
This project uses [Inquirer.js](https://www.npmjs.com/package/inquirer) to chat with the model using OpenAI's API. The model can be running locally using `bin/server` or deployed to Lambda using `bin/deploy`. Inquirer will ask for your local or function URL at the beginning of the chat session.
