An open API service indexing awesome lists of open source software.

https://github.com/picteus/picteus

Extensible local-first framework for indexing and orchestrating AI image assets
https://github.com/picteus/picteus

ai automatic1111 cockpit comfyui dam electron freepik generation image localfirst metadata midjourney nestjs node openapi python recipe stable-diffusion swagger-ui typescript

Last synced: about 17 hours ago
JSON representation

Extensible local-first framework for indexing and orchestrating AI image assets

Awesome Lists containing this project

README

          

# Picteus

## Disclaimer

Please have a look at the [Disclaimer](DISCLAIMER.md) if you want to better understand what this project is about.

## Download

If you do not want to build the application, you may install it after having downloaded its installation package from the "Assets" section of the [Releases](../../releases). Currently, only Windows x64 and macOS ARM 64 (Apple Silicon) are supported.

## Architecture

The application is made of six components:
1. "shared": a Node.js module which is injected as a dependency to both the "server" and "electron" components ;
2. "server": the Node.js HTTP server back-end application, capable of running extensions ;
3. "extensions/sdk": the Python and TypeScript SDKs ;
4. "extensions/instances": the built-in extensions for the server ;
5. "web": the React.js web front-end application ;
6. "electron": the wrapping Electron application which embeds the "server" and "web" previous components.

## Prerequisites

- Node.js v22.7.1+ with npm (tested) or pnpm or yarn for building the application's components and the extensions.
- Python v3.11+ for building the Python extensions.
- The server module requires Java v17+, which should be accessible as a first Java runtime through the `PATH`, when generating the OpenAPI client code.
- When building the container image, a Docker-like application should be up and running.

## Compiling and building

The following commands will install dependencies and build the components. They should be run from the root `package.json` file.

- Run the traditional `npm run install` script for resolving the hereby package dependencies.
- Run the `npm run prerequisites` script for resolving all submodules' dependencies: do not run individually the `npm install` script on every module, because it will not properly install the "server" and "electron" components since the `prerequisites` npm script resorts to the `--install-link` option, which installed the "shared/back-end" module dependencies.
- Run the `npm run build` script for building all artifacts.

Most of the artifact files are located under the `build` directory, which is a symbolic link pointing to the `electron/build` directory.

## Cleaning

The following commands enable cleaning up the project. They should be run from the root `package.json` file.

- Run the `npm run clean` script for cleaning all artifacts coming from compilation and previous builds.
- Run the `npm run reset` script for resetting the state of the project's files to their git initial state — in particular, this deletes the `node_modules` directories.

## Versions

The versions of the various components are specified through the root `package.json` file via the `config.applicationVersion`, `config.apiVersion` and `config.sdkVersion` properties:
- `config.applicationVersion`: the version of the Electron application ;
- `config.serverVersion`: the version of the server ;
- `config.webVersion`: the version of the front-end application ;
- `config.apiVersion`: the version of the API and its OpenAPI web services contract ;
- `config.sdkVersion`: the version of the SDK.

Whenever changing any of those versions, think of running the `npm run updateVersion` script from the root directory to propagate the version changes to the relevant submodules, which updates the `src/constants.ts` file accordingly and the SDK version and the extensions' dependency version accordingly.

### Server (back-end)

Its source-code and scripts are located under the `server` directory.

All the commands specified in that section should be run from the `server` subdirectory and all resource locations are expressed from that directory, unless stated otherwise explicitly.

#### Build

- To build the server from scratch, run the following command from the root folder: `npm run server:prerequisites && npm run server:build`.
- To recompile it after having changed its code, run the `npm run build` script.

#### Run

To run the server, run the `npm run start` script. The environment variables which have an impact over the execution as well as the commands and parameters are described by running the `npm run start:help` script.

You may access to the SwaggerUI relative the OpenAPI specifications, captured through the JSON `openapi.json` file, open the browser to the [http://localhost:3001/swaggerui](http://localhost:3001/swaggerui) URL, 3001 being the server default port.

#### Development

- When changing the database Prisma schema:
- run the `npm run prisma:generate` script to regenerate the Prisma client code,
- then run the `npm run prisma:update` script to update the schema of the startup database file `database.db` and to add the new SQL migration script in the `prisma/migrations` directory (in case of a synchronization issue with the schema migration, read this article [Make Prisma Ignore a Migration Change](https://echobind.com/post/make-prisma-ignore-a-migration-change)),
- and then run the `npm run prisma:seed` script to update the `settings` table value startup database file `database.db` with its settings set to the latest version of the migration.

- When changing the OpenAPI contract, run the `openApi:generateOpenApi` script to regenerate the `openapi.json` file.
- Think of updating the `config.apiVersion` property.
- The `npm run openApi:typeScript:generateAndPackage` script generates the TypeScript OpenAPI client library into the directory `../generated/openapi/typescript-fetch`.
- The `npm run openApi:python:generateAndPackage` script generates the Python OpenAPI client library into the directory `../generated/openapi/python`.
- The `npm run openApi:generateAndPackage` script generates the 2 previous packages.

- Whenever the database schema changes, a file in the `secrets` or `assets` directory changes, run the `npm run build:copy` script to update the Electron application `dist` directory.

#### API web services client library publication

To publish a new version of the API web services client library, run the `npm run wsClient:publish` script, after having built the libraries via the previous `openApi:generateAndPackage` script:
- for publishing the Node.js package on npm, use the `npm login --scope=@koppasoft` command to log in first ;
- for publishing the Python package on PyPi, declare an API token at https://pypi.org/manage/account/token/ beforehand.

### Extensions

The SDKs and extensions and scripts are located in the `extensions` directory and the commands in the rest of this section should be run from that directory:
- the SDKs are in the `sdk` subdirectory ;
- the extensions are in the `instances` subdirectory.

For building the SDKs and extensions:
- run the `npm run sdk:build` script to build the SDKs ;
- run the `npm run instances:build` script to build the extensions ;
- run the `npm run build` script to build both.

### SDKs publication

To publish a new version of the SDKs, run the `npm run sdk:publish` script, after having built the SDKs via the `sdk:public:build` script:
- for publishing the Node.js SDK package on npm, use the `npm login --scope=@koppasoft` command to log in first ;
- for publishing the Python SDK package on PyPi, declare an API token at https://pypi.org/manage/account/token/ beforehand.

### Web (front-end)

Its source-code and scripts are located under the `web` directory.

All the commands specified in that section should be run from the `web` subdirectory and all resource locations are expressed from that directory, unless stated otherwise explicitly.

#### Build

- To build the web component from scratch, run the following command from the root folder: `npm run web:prerequisites && npm run web:build`.
- To recompile it after having changed its code, run the `npm run build` script.

#### Run

To run the web component, run the `npm run start` script.

### Shared

Its source-code and scripts are located under the `shared` directory. It only contains a `back-end` folder, which contains code common between the "server" and "electron" modules.

- Run the `npm run build` script to build it, which will compile the code.

### Electron

Its source-code and scripts are located under the `electron` directory.

- Run the `npm run start` script to start it on the local machine.
- Run the `npm run build` script to build it for the same target OS as the hosting machine, which outputs a runnable artifact inside the `dist` directory.
- Run the `npm run package` script to package it for the same target OS as the hosting machine.

### Container image

The container image specifications are classically defined through the `Dockerfile` file and the ignored files through the `.dockerignore` file.

#### Build

To build the container image of the server application via Docker, which also embeds the web application, run the `npm run docker:build` script from the root directory, which creates an image with the `koppasoft/picteus:latest` tag.

#### Run

When running the container image through Docker, you should beforehand declare a volume so that the application data are persisted beyond the container execution, use the `docker volume create picteus` command to create a Docker volume named `picteus`.

Then use the following command to run the container:
```
docker run -p 3001:3001 -p 2999:2999 -p 3002:3002 -v picteus:/app/internal -v :/app/external -v :/app/files -e filesMountPath= --name picteus koppasoft/picteus:latest
```
where:
- `` is the host path to the directory where you want the application to scan files ;
- `` is the host path to the directory where you want the application to store database files.

This will build an image and create a container with the name `picteus`, using the previously created `picteus` volume:
- its API server Swagger UI URL is http://localhost:3001/swaggerui# (adapt the port number according to your configuration, see below);
- its front-end application URL is http://localhost:2999/?webServicesBaseUrl=http://localhost:3001#/ (adapt the port numbers according to your configuration, see below) ; notice the `webServicesBaseUrl` URI query parameter, which indicates the URL of the API web services, which is http://localhost:3001.

You may fine-tune the container with the following additional environment variables:
- `apiServerPort`: the port number of the API server, which defaults to `3001` ; if you change it, think of changing the port mapping accordingly ;
- `webServerPort`: the port number of the web server exposing the UI, which defaults to `2999` ; if you change it, think of changing the port mapping accordingly ;
- `vectorDatabasePort`: the port number of the vector database server, which defaults to `3002` ; if you change it, think of changing the port mapping accordingly ;
- `useSsl`: a boolean indicating whether the API server should use SSL / TLS, which defaults to `true` ;
- `requiresApiKeys`: a boolean indicating whether the API server only works with API keys, which defaults to `false`.

## Packaging and distributing

The Electron application:
1. is packaged via the `npm run package` script: this invokes the previously mentioned Electron `package` npm script ;
2. is distributed via the `npm run distribute` script — which also invokes the previous one —, which signs, zips and notarize the application distribution package on macOS, which should be executed with the following environment variables set, when run on macOS:
- `MACOS_APPLICATION_CERTIFICATE_BASE64_CONTENT`: the base64 encoded content of the application certificate. This content is obtained via the `base64 -i ` command, where `` is the path of the "Developer ID Application" certificate file in P12 format ;
- `MACOS_APPLICATION_CERTIFICATE_PASSWORD`: the password of the previous certificate ;
- `MACOS_NOTARIZE_APPLICATION_PASSWORD`: the [Apple application-specific](https://discussions.apple.com/thread/254805086?sortBy=rank) password related to the `Picteus` entry, used to notarize the application package ;
3. is deployed via the `npm run deploy` script, which uploads the previously generated application distribution package: the `gcloud login` command should have been run beforehand, with GCP credentials having permissions over the destination GCS bucket, or with the `CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE` environment variable set with the contents of a GCP account service key with writing permissions over the GCS bucket.

## Continuous Integration (CI)

The project is configured with GitHub Actions for CI. For simulating locally what the CI does, you can install [act](https://github.com/nektos/act) and resort to the following command lines, on macOS:
- for the "Server CI" chain: `act --workflows .github/workflows/server.yml --container-architecture linux/amd64 -P macos-latest=catthehacker/ubuntu:act-latest` ;
- for the "Electron CI" chain: `act --workflows .github/workflows/electron.yml --container-architecture linux/amd64 -P macos-latest=catthehacker/ubuntu:act-latest`.

- Use the `--bind` option if you wish to prevent "act" from copying the files to the container, which takes time because of the large number of files.
- Only for the `.github/workflows/server.yml` workflow, use the `--env skip="true` to skip the installation steps.