https://github.com/nathankot/bashi

Extensible AI assistant platform that bridges LLMs to tasks and actions
https://github.com/nathankot/bashi

Last synced: 3 months ago
JSON representation

Extensible AI assistant platform that bridges LLMs to tasks and actions

• Host: GitHub
• URL: https://github.com/nathankot/bashi
• Owner: nathankot
• License: agpl-3.0
• Created: 2022-12-14T02:53:28.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2023-04-17T15:55:54.000Z (almost 2 years ago)
• Last Synced: 2024-10-26T22:48:00.831Z (3 months ago)
• Language: Swift
• Homepage:
• Size: 5.28 MB
• Stars: 37
• Watchers: 3
• Forks: 3
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# 🌁 Bashi

_Note: Bashi is still very early stage and under active development, if you are

looking for a polished product you will be disappointed. But if you are looking

to spend some time building your own personal assistant AI then you're at the

right place!_

Bashi is an extensible platform that bridges LLMs to tasks and actions.

It comes with an OSX personal assistant app, so you can try it out

quickly, and mould the app to your own needs.

This repo has two components:

- The Bashi API server.

- The OSX app 'assist', which serves as an example client implementation, as

  well as a usable product

Examples of the assist OSX app in action:



  

  



## LLM+REPL

Bashi uses a novel approach in that the LLM is asked to write Javascript and the

server effectively provides a REPL for the model. Below are some example

prompt+completions. All of these are real examples that you can try on the OSX

app :+1:

```markdown

Request: help me write a commit message please

Thought: I need to generate a commit message based on a diff

Action: returnText(writeCommitMessage(getInputText(\"diff\")));

```

```markdown

Request: there is a function I don't understand, can help me summarize it?

Thought: I need to extract the information from the given string

Action: returnText(extractInformation(\"summarize the function\", getInputText(\"what is the function?\")))

```

There seems to be some advantages to this:

- Reduced completion sizes since not only does code compact information, but by

  having multiple steps in a single action you save on model round trips.

- GPT3.5 was probably trained on lots of Javascript, so asking it to write Javascript

  may lead to tendencies towards emergent reasoning/logic characteristics.

## Extensible

### Client-defined commands

Clients are able to _extend_ the capabilties of the agent by providing their own

commands/functions. For example the OSX client provides the server with

information about the `createCalendarEvent` command:

```swift

AnonymousCommand(

    name: "createCalendarEvent",

    cost: .Low,

    description: "make calendar event for the given name, datetime and duration",

    args: [

            .init(type: .string, name: "name"),

            .init(type: .string, name: "iso8601Date"),

            .init(type: .number, name: "event duration in hours")

    ],

    returnType: .void,

    triggerTokens: ["calendar", "event", "appointment", "meeting"],

    runFn: { (api, ctx, args) async throws -> BashiValue in

        // ... redacted guard code

        let event = EKEvent.init(eventStore: self.eventStore)

        event.startDate = date

        event.title = name

        event.endDate = date.addingTimeInterval(60 * 60 * hours.doubleValue)

        event.calendar = defaultCalendar

        try self.eventStore.save(event, span: .thisEvent, commit: true)

        await api.indicateCommandResult(message: "Calendar event created")

        return .init(.void)

    }),

```

Commands can be resolved on either the client or the server. The example above

is a command resolved on the client. In contrast, there are commands like `math`

that are resolved on the server.

### Write your own client

Clients just need to interface with the API defined in

[`openapi.json`](server/static/openapi.json). There are plenty of OpenAPI

definition -> client library generation tools out there to help get things

started if you wish to write a client in a new language.

## Running it locally

### Server

After cloning the repo, set up your API keys. At minimum you'll need an OpenAI

API key.

```sh

cp server/.env.template server/.env

# edit server/.env

```

Run the entire server stack using docker:

```sh

make build

make up-all

open http://localhost:8003

```

The index page has some examples and you can play around with text or audio

prompts. Although any commands that must be resolved on the client-side are

fixtures/dummies. (The OSX app includes 'real' client-side commands).

If you are working on the server, you should set up the live-reloading

server. Not only does this pick up code changes, but it also generates the

OpenAPI spec when there are changes to API interface.

```sh

make dev

open http://localhost:8080

```

### OSX App

Currently pre-built binaries are not available so you'll need to build the OSX app yourself.

Open the `xcworkspace` in Xcode:

```

open assist/assist.xcworkspace

```

Build the 'assist' scheme. **Note** by default the app points to

`http://localhost:8003/api` which corresponds to the server running in docker

via `make up-all`. If you are running the server with `make dev` you'll want to

update the API base URL to `http://localhost:8080/api`by going to the app

settings.

Any changes to the API surface will require a new swift client to be generated

using `make clients`.

## Documentation

I still need to work on some more comprehensive documentation for the codebase 🙇

### API Spec

The API is described in [openapi.json](server/static/openapi.json) which can be

plugged into https://editor.swagger.io/ for viewing.

## Contributing

Let's build JARVIS together :)

There is no contribution guide for now, but you are welcome to make

contributions to the OSX client (or introduce new clients if you are okay with

an unstable API).

Issues and feature requests are accepted for the `server/`, but not code changes

at this moment.

### Testing

For the OSX client run tests via Xcode as per usual.

For the server use `make test` to run tests, and `make test-update` to update

test snapshots. [Snapshot

testing](https://deno.land/[email protected]/basics/testing/snapshot_testing) is

used liberally, for better or for worse 🙈

### Bug reports

Running the server in `make dev` results in more verbose logging, please copy

and paste the error output into any bug reports (after redacting sensitive

information, if any).

### Known issues

(Need to migrate these to GH issues)

- If the model does not have the commands to complete the request, it will tend

  to just repeat the question back to the user.

  - A general knowledge lookup command would be useful here, the existing search

    command is not sufficient as it does not provide a knowledge graph. Perhaps

    this command can have multiple layers - ask a LLM first, if no answer use

    some search API.

- The model gets confused when there are overlapping commands relevant to a

  request. For example if you ask it to `write example code to create a reminder

using swift`, it ends up calling the `createReminder()` command instead.

  - I'm exploring an approach to alleviate this by having a pre-process stage

    where the command set is filtered down, this could also result in token savings.

- The model will sometimes output an un-parseable expression leading to error. A

  quick fix is usually to rephrase your request, in the long term the following

  approaches are being considered:

  - Progressively loosen the definition of the action language to accomodate for common model errors.

  - Support configurable automatic retries.

  - Prompt engineering and fine tuning.

- Apple APIs for audio recording and transcription is hard :( Push-to-talk is

  very flaky with airpods on.