https://github.com/sillsdev/serval
A REST API for natural language processing services
https://github.com/sillsdev/serval
Last synced: 3 days ago
JSON representation
A REST API for natural language processing services
- Host: GitHub
- URL: https://github.com/sillsdev/serval
- Owner: sillsdev
- License: mit
- Created: 2023-02-27T06:42:14.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2026-02-19T14:37:01.000Z (2 months ago)
- Last Synced: 2026-02-19T18:13:37.121Z (2 months ago)
- Language: C#
- Homepage:
- Size: 22.7 MB
- Stars: 4
- Watchers: 5
- Forks: 2
- Open Issues: 44
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/sillsdev/serval/actions/workflows/ci.yml)
[](https://github.com/sillsdev/serval/actions/workflows/ci-e2e.yml)
[](https://codecov.io/gh/sillsdev/serval)
# Serval - an API supporting Computer Aided Translation for all the languages of the world!
Serval is a REST API for natural language processing services for **all languages**.
For the REST documentation use the Swagger site [here](https://prod.serval-api.org/swagger/index.html).
# Serval Architecture
Serval is designed as a modular monolith with a single deployable unit:
- Serval.ApiServer
- The sole deployment — all REST calls are made through this layer.
- Hosts all domain modules in-process; modules communicate via interfaces rather than over the network.
- Domain modules under `./src/Serval/src`:
- `Serval.Translation` — translation engine management and pretranslation assembly
- `Serval.WordAlignment` — word alignment engine management
- `Serval.DataFiles` — file and corpus management
- `Serval.Webhooks` — webhook delivery
- `Serval.Shared` — common models, configuration, and services shared across modules
- Engine implementations (also hosted in-process):
- `Serval.Machine.Shared` — NMT, SMT Transfer, and Statistical engine implementations; also runs Hangfire background build jobs and queues ClearML GPU training jobs
- `EchoEngine` — echo engine for testing (translation and word alignment stubs)
- External runtime dependencies: MongoDB (persistence), Hangfire (job scheduling), ClearML (GPU training jobs), S3 (shared file storage for training data and models)
- SIL.DataAccess
- Abstracts all MongoDB operations
- Enables in-memory database for testing purposes
- Replicates the functionality of [EF Core](https://learn.microsoft.com/en-us/ef/core/)
# Development
## Setting up Your Environment
- Use VS Code with all the recommended extensions
- Development is supported in Ubuntu and Windows WSL2
- For Ubuntu, use [microsoft's distribution of .net](https://learn.microsoft.com/en-us/dotnet/core/install/linux-ubuntu)
- Ubuntu 22.04 and 24.04 are currently supported
- Install the repositories:
- To develop Serval, you will also likely need to make changes to the [Machine repo](https://github.com/sillsdev/machine) as well - they are intricately tied.
- To enable Serval to use your current edits in Machine (rather than the nuget package) you need to install Machine in an adjacent folder to Serval
- i.e., if your serval repo is cloned into /home/username/repos/serval, machine should be in /home/username/repos/machine
- Make sure that you build Machine using before you build Serval
## Option 1: Docker Compose local testing deployment
These instructions are for developing/testing using docker-compose (rather than locally/bare metal)
With both the serval and machine repos installed, in the serval root folder, run `./docker_deploy.sh`
To debug in VSCode, launch "ServalApi Docker" after to containers come up (about 5 -10 seconds). This will allow you to debug all 4 serval containers.
## Option 2: Bare metal local testing deployment
Alternatively, you can develop without containerizing Serval.
Install MongoDB 8.0 as a replica set run it on localhost:27017. (You can run `docker compose -f docker-compose.mongo.yml up` from the root of the serval repo to do so).
Make sure that the environment variable ASPNETCORE_ENVIRONMENT is set to "Development" by running `export ASPNETCORE_ENVIRONMENT=Development` or adding it to your `.bashrc`.
Open "Serval.sln" and debug the ApiServer.
## Coding guidelines
Coding guidelines are documented [on the wiki](https://github.com/sillsdev/serval/wiki/Development-Guide)
## (Optional) Get your machine.py images setup
When jobs are run, they are queued up on ClearML. If you want to have your own agents for integration testing (and you have a GPU with 24GB RAM), you can do the following:
- clone the [machine.py repo](https://github.com/sillsdev/machine.py)
- Build the docker image with `docker build . -t local.mpy` for a GPU image or `docker build . -f dockerfile.cpu_only -t local.mpy.cpu_only` for a CPU only image.
- Register your machine as a ClearML agent (see dev team for details)
- Make sure you do NOT "always pull the image"! The images you are building are stored locally.
- Set the following environment variables:
```
export MACHINE_PY_IMAGE=local.mpy
export MACHINE_PY_CPU_IMAGE=local.mpy.cpu_only
```
### Running the API E2E Tests
In order to run the E2E tests, you will need to have the appropriate credentials
- Get Client ID and Client Secret from auth0.com
- Login, go to Applications-> Applications -> "Machine API (Test Application)" or similar
- Copy `Client ID` into Environment variable `SERVAL_CLIENT_ID`
- Copy `Client Secret` into Environment variable `SERVAL_CLIENT_SECRET`
- Copy the auth0 url into Environment variable `SERVAL_AUTH_URL` (e.g. `SERVAL_AUTH_URL=https://sil-appbuilder.auth0.com`)
- Set `SERVAL_HOST_URL` to the api's URL (e.g. `SERVAL_HOST_URL=http://localhost`)
Now, when you run the tests from `Serval.E2ETests`, the token will automatically be retrieved from Auth0.
## Special thanks to
BugSnag for error reporting:
