Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/fabianlindfors/restate
Build reliable, understandable and debuggable backends with state machines
https://github.com/fabianlindfors/restate
api backend typescript
Last synced: about 2 months ago
JSON representation
Build reliable, understandable and debuggable backends with state machines
- Host: GitHub
- URL: https://github.com/fabianlindfors/restate
- Owner: fabianlindfors
- License: mit
- Created: 2023-04-05T21:37:20.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-04-26T22:15:51.000Z (over 1 year ago)
- Last Synced: 2024-10-31T21:35:12.452Z (about 2 months ago)
- Topics: api, backend, typescript
- Language: TypeScript
- Homepage:
- Size: 445 KB
- Stars: 37
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Restate
Restate is an experimental Typescript framework for building backends using state machines. With Restate, you define all database models as state machines which can only be modified through state transitions. The logic for the transitions are defined in code, and it's also possible to run code asynchronously in response to state transitions, for example to trigger new transitions, send emails or make API requests. This enables more complex business logic to be expressed by connecting together simpler state machines.
The point of Restate is to help build systems which are:
- **Debuggable:** All state transitions are tracked, making it easy to trace how a database object ended up in its current state and what triggered its transitions (an admin interface is in the works).
- **Understandable:** All business logic is encoded in state transitions and consumers, making it easy to understand the full behavior of your system. Writing decoupled code also becomes easier with consumers.
- **Reliable:** Consumers are automatically retried on failure and change data capture is used to ensure no transitions are missed.
Does that sound interesting? Then keep reading for a walkthrough of a sample project!
## Getting started
### Installation
To get started with Restate, we are going to create a standard Node project and install Restate:
```text
$ mkdir my-first-restate-project && cd my-first-restate-project
$ npm init
$ npm install --save restate-ts
```
For this example, we are going to be using Express to build our API, so we need to install that as well:
```text
$ npm install --save express
```
Restate has a built in development tool with auto-reloading, start it and keep it running in the background as you code:
```text
$ npx restate
```
You'll see a warning message saying that no project definition was found, but don't worry about that, we'll create one soon!
### Defining models and transitions
Database models in Reshape are defined in a custom file type, `.rst`, and stored in the `restate/` folder of your project. Every database model is a state machine and hence we need to define the possible states and transitions between those states.
For this project, we are going to model a very simple application that tracks orders. Orders start out as created and are then paid by the customer. Once the order has been paid, we want to book a delivery with our carrier. To model this in Restate, let's create a new file called `restate/Order.rst` with the following contents:
```
model Order {
// All models have an autogenerated `id` field with a prefix to make them easily identifiable
// In this case, they will look something like: "order_01gqjyp438r30j3g28jt78cx23"
prefix "order"
// The common fields defined here will be available across all states
field amount: Int
state Created {}
state Paid {
field paymentReference: String
}
// States can inherit other state's fields, so in this case `DeliveryBooked` will have `amount` and `paymentReference` fields as well
state DeliveryBooked: Paid {
field trackingNumber: String
}
// `Create` doesn't have any starting states and is hence an initializing transition.
// It will be used to create new orders.
transition Create: Created {
field amount: Int
}
// `Pay` is triggered when payment is received for the order
transition Pay: Created -> Paid {
field paymentReference: String
}
// `BookDelivery` is triggered when an order has been sent and we are ready to book delivery
transition BookDelivery: Paid -> DeliveryBooked {}
}
```
### Generating the Restate client
Once we have defined our models, the dev session you have running will automatically generate types for your models as well as a client to interact with them. All of this can be imported directly from the `restate-ts` module.
The starting point of any Restate project is the project definition, which lives in `src/restate.ts`. The definition we export from that file defines how our models' transitions are handled. Let's start with some placeholder values, create a `src/restate.ts` file with the following code:
```typescript
import { RestateProject, RestateClient, Order } from "restate-ts";
const project: RestateProject = {
async main(restate: RestateClient) {
// main is the entrypoint for your project. Here you could for example start a web server and use
// `restate` to create and transition objects.
},
transitions: {
// We need to provide implementations for all transitions
order: {
async create(restate: RestateClient, transition: Order.Create) {
throw new Error("Create transition not implemented");
},
async pay(
restate: RestateClient,
order: Order.Created,
transition: Order.Pay
) {
throw new Error("Pay transition not implemented");
},
async bookDelivery(
restate: RestateClient,
order: Order.Paid,
transition: Order.BookDelivery
) {
throw new Error("BookDelivery transition not implemented");
},
},
},
};
// The definition should be the default export
export default project;
```
### Creating orders
Before we can create orders, we need to actually implement the `Create` transition in `src/restate.ts`:
```typescript
const project: RestateProject = {
// ...
transitions: {
order: {
async create(restate: RestateClient, transition: Order.Create) {
// We should return the shape of the object after the transition has been applied
// As this is an initializing transition, it will result in a new object being created
return {
state: Order.State.Created,
// amount is passed through the transition and saved to the object
amount: transition.data.amount,
};
},
},
},
};
```
To interact with our backend, we are going to create a simple HTTP API using `express`. Restate is fully API agnostic though so you can interact with it however you want; REST, GraphQL, SOAP, anything goes! Let's start a simple web server from the `main` function in `src/restate.ts` with a single endpoint to create a new order:
```typescript
import express from "express";
import { RestateProject, RestateClient, Order } from "restate-ts";
const project: RestateProject = {
async main(restate: RestateClient) {
const app = express();
app.post("/orders", async (req, res) => {
// Get amount from query parameter
const amount = parseInt(req.query.amount);
// Trigger `Create` transition to create a new order object
const [order] = await restate.order.transition.create({
data: {
amount,
},
});
// Respond with our new order object in JSON format
res.json(order);
});
app.listen(3000, () => {
console.log("API server started!");
});
},
};
```
The dev session should automatically reload and you should see "API server started!" in the output. Let's test it!
```shell
$ curl -X POST "localhost:3000/orders?amount=100"
{
"id": "order_01gqjyp438r30j3g28jt78cx23",
"state": "created",
"amount": 100
}
```
It works and we get a nice order back! Here you see both the `amount` field, which we specified in `Order.rst`, but also `id` and `state`. These are fields which are automatically added for all models.
### Querying orders
Being able to create data wouldn't do much good if we can't get it back, which Restate handles using queries. We'll add the following code to our main function to introduce a new endpoint for getting all orders:
```typescript
app.get("/orders", async (req, res) => {
// Get all orders from the database
const orders = await restate.order.findAll();
res.json(orders);
});
```
And if we try that, we unsurprisingly get back:
```shell
$ curl localhost:3000/orders
[
{
"id": "order_01gqjyp438r30j3g28jt78cx23",
"state": "created",
"amount": 100
}
]
```
### Transitioning orders when paid
Now we are getting to the nice parts. We've created our order and the next step is to update it to the paid state once we receive a payment. The first step is to add a very simple implementation for the `Pay` transition:
```typescript
const project: RestateProject = {
// ...
transitions: {
order: {
async pay(
restate: RestateClient,
order: Order.Created,
transition: Order.Pay
) {
return {
// The spread operator is a convenient way of avoiding having to specify all fields again
...order,
state: Order.State.Paid,
paymentReference: transition.data.paymentReference,
};
},
},
},
};
```
For this example, let's say our payment provider will send us a webhook when an order is paid for. To handle that, we'll need another endpoint which should trigger the `Pay` transition for an order:
```typescript
app.post("/webhook/order_paid/:orderId", async (req, res) => {
// Get payment reference from query parameters
const reference = req.query.reference;
// Trigger the `Pay` transition for the order, which returns the updated object
const [order] = await restate.order.transition.pay({
object: req.params.orderId,
data: {
// The `Pay` transition requires us to pass the payment reference
paymentReference: req.query.reference,
},
});
// Respond with the updated object
res.json(order);
});
```
If we were to simulate a webhook request from our payment provider, we get back an order in the expected state and with the passed reference saved to a new field (remember to replace the order ID with the one you got in the last request):
```shell
$ curl -X POST "localhost:3000/webhook/order_paid/order_01gqjyp438r30j3g28jt78cx23?reference=abc123"
{
"id": "order_01gqjyp438r30j3g28jt78cx23",
"state": "paid",
"amount": 100,
"paymentReference": "abc123"
}
```
### Asynchronously booking deliveries
For the final part of this example, we want to book a delivery when an order is paid, with an imagined API call to our shipping carrier. Let's start by implementing the final `bookDelivery` transition for this:
```typescript
const project: RestateProject = {
// ...
transitions: {
order: {
async bookDelivery(
restate: RestateClient,
order: Order.Paid,
transition: Order.BookDelivery
) {
// This is where we'd call the shipping carriers API and get a tracking number back, but for the sake
// of the example, we'll use a static value
const trackingNumber = "123456789";
return {
...order,
state: Order.State.DeliveryBooked,
trackingNumber,
};
},
},
},
};
```
What we could do is simply trigger this transition right in our payment webhook, but our shipping carrier's API is really slow and unreliable, so we don't want to bog down the webhook handler with that. Preferably we want to perform the delivery booking asynchronously! This is where one of Restate's central features come in: consumers.
Consumers let's us write code that runs asynchronously in response to transitions. This lets us improve reliability, performance and code quality through decoupling. Like most everything in Restate, consumers are defined in `src/restate.ts`. In our case, we want to trigger the `BookDelivery` transition when the `Pay` transition has completed, so let's add a consumer for that:
```typescript
const project: RestateProject = {
// ...
consumers: [
Order.createConsumer({
// Every consumer should have a unique name
name: "BookDeliveryAfterOrderPaid",
// We can tell our consumer to only trigger on specific transitions
transition: Order.Transition.Pay,
async handler(
restate: RestateClient,
order: Order.Any,
transition: Order.Pay
) {
// You might notice that `order` has type `Order.Any` rather than `Order.Paid`.
// It's possible that the object changed since the consumer was queued but we'll always
// get the latest version in here. Because consumers are asynchronous, this is something
// we must take into consideration.
if (order.state != Order.State.Paid) {
return;
}
// Trigger `BookDelivery` transition, which will take a little while but that is completely fine!
await restate.order.transition.bookDelivery({
object: order,
});
},
}),
],
// ...
};
```
If you now mark a payment as paid using the webhook endpoint, you should soon after see that the order has been updated again:
```shell
$ curl localhost:3000/orders
[
{
"id": "order_01gqjyp438r30j3g28jt78cx23",
"state": "deliveryBooked",
"amount": 100,
"paymentReference": "abc123",
"trackingNumber": "123456789"
}
]
```
That's it for the introduction! Keep reading to learn more about the different features of Restate.
## Model definitions
### IDs and prefixes
Every Restate model has an implicit field, `id`, which stores an autogenerated identifier. All IDs are prefixed with a string unique to the model, which makes it easier to identify what an ID is for. Here's an example of defining an `Order` model with prefix `order`. Objects of this model will automatically get IDs that look like: `order_01gqjyp438r30j3g28jt78cx23`.
```
model Order {
prefix "order"
}
```
### Fields
All Restate models have two implicit fields: `id` and `state`, which store an autogenerated ID and the current state respectively. When defining a model, it's also possible to add custom fields. Fields can be defined top-level, in which case they will be part of all states, or only on specific states.
```
model User {
field name: String
state Verified {
field age: Int
}
}
```
Every field has a data type and is by default non-nullable. If you want to make a field nullable, wrap the type in an `Optional`:
```
model User {
field name: Optional[String]
}
```
Restate supports the following data types:
| **Data type** | **Description** | **Typescript equivalent** |
| ---------------- | -------------------------------- | ------------------------- |
| `String` | Variable-length string | `string` |
| `Int` | Integer which may be negative | `number` |
| `Decimal` | Decimal number | `number` |
| `Bool` | Boolean, either true or false | `boolean` |
| `Optional[Type]` | Nullable version of another type | `Type \| null` |
## Client
The Restate client is used to create, transition and query objects. In the following examples, we'll be working with a model definition that looks like this:
```
model Order {
prefix "order"
field amount: Int
state Created {}
state Paid {}
transition Create: Created {
field amount: Int
}
transition Pay: Created -> Paid {}
}
```
### Transitions
The client can be used to trigger initializing transitions, which create new objects. The transition call will return the new object after the transition has been applied.
```typescript
const [order] = await restate.order.transition.create({
data: {
amount: 100,
},
});
```
For regular transitions, one must also specify which object to apply the transition to by passing an object ID or a full object.
```typescript
const [paidOrder] = await restate.order.transition.pay({
object: "order_01gqjyp438r30j3g28jt78cx23",
});
```
If passing a full object, the types will ensure it's in the correct state for the transition to apply:
```typescript
const order: Order.Paid = {
// ...
};
const [paidOrder] = await restate.order.transition.pay({
// This will trigger a type error because the `Pay` transition can
// only be applied to orders in state `Created`
object: order,
});
```
Transition calls will also return the full transition object if needed:
```typescript
const [paidOrder, transition] = await restate.order.transition.pay({
object: "order_01gqjyp438r30j3g28jt78cx23",
});
console.log(transition.id);
// tsn_01gqjyp438r30j3g28jt78cx23
// Transition IDs have prefix "tsn"
```
It's of course also possible to get a transition by ID or all transitions for an object:
```typescript
const [paidOrder, transition] = await restate.order.transition.pay({
object: "order_01gqjyp438r30j3g28jt78cx23",
});
// Find a single transition by ID
const transitionById = await restate.order.getTransition(transition.id);
// Find all transitions for an object (starting with the latest one)
const allTransitions = await restate.order.getObjectTransitions(paidOrder);
```
For debugging purposes, it's possible to add a free text note to a transition. This field is designed to be human readable and should not be relied upon by your code:
```typescript
const [paidOrder] = await restate.order.transition.pay({
order: "order_01gqjyp438r30j3g28jt78cx23",
note: "Payment manually verified",
});
```
### Queries
There are different kinds of queries depending on how many results you expect back. To find a single object by ID, you can do:
```typescript
const order: Order.Any | null = await restate.order.findOne({
where: {
id: "order_01gqjyp438r30j3g28jt78cx23",
},
});
```
Similarly, it's possible to filter by all fields on a model and to find many objects:
```typescript
const orders: Order.Any[] = await restate.order.findAll({
where: {
amount: 100,
},
});
```
When querying by state, the resulting object will have the expected type:
```typescript
const orders: Order.Created[] = await restate.order.findAll({
where: {
state: Order.State.Created,
},
});
```
If you want an error to be thrown if no object could be found, use `findOneOrThrow`:
```typescript
const order: Order.Any = await restate.order.findOneOrThrow({
where: {
id: "order_01gqjyp438r30j3g28jt78cx23",
},
});
```
You can also limit the number of objects you want to fetch:
```typescript
const orders: Order.Created[] = await restate.order.findAll({
where: {
state: Order.State.Created,
},
limit: 10,
});
```
## Testing
Restate has built-in support for testing with a real database. In your test cases, import the project definition from `src/restate.ts` and pass it to `setupTestClient` to create a new Restate client for testing. This client will automatically configure an in-memory SQLite database and will run any consumers synchronously when transitions are triggered.
Here's an example in [Jest](https://jestjs.io), but any test framework will work:
```typescript
import { test, expect, beforeEach } from "@jest/globals";
import { Order, RestateClient, setupTestClient } from "restate-ts";
// Import project definition from "restate.ts"
import project from "./restate";
let restate: RestateClient;
beforeEach(async () => {
// Create a new test client for each test run
restate = await setupTestClient(project);
});
test("delivery is booked when order is paid", async () => {
// Create order
const order = await restate.order.transition.create({
data: {
amount: 100,
},
});
// Trigger `Pay` transition on order
await restate.order.transition.pay({
object: order,
data: {
paymentReference: "abc123",
},
});
// The `BookDeliveryAfterOrderPaid` consumer should have been triggered when the order was paid
// and transitioned it into `DeliveryBooked`. With the test client, consumers are run synchronously.
const updatedOrder = await restate.order.findOneOrThrow({
where: {
id: order.id,
},
});
expect(user.state).toBe(Order.State.DeliveryBooked);
expect(user.trackingNumber).toBe("123456789");
});
```
## Config
If you want to configure Restate, create a `restate.config.json` file in the root of your project. In your config file, you can specify settings based on environment and the environment will be based on the `NODE_ENV` environemnt variable. When running `restate dev`, the default environment will be `development`. For all other commands, it will default to `production`.
In your config file, you can configure what database to use. Restate supports both Postgres and SQLite, where we recommend using Postgres in production and SQLite during development and testing. Below is an annotated example of a config file, showing what settings exist and the defaults:
```jsonc
{
"database": {
"type": "postgres",
"connection_string": "postgres://postgres:@localhost:5432/postgres"
},
// The settings in here will only be used in the development environment
"development": {
"database": {
"type": "sqlite",
"connection_string": "restate.sqlite"
}
}
}
```
## Commands
### `restate dev`
Starts an auto-reloading dev server for your project. It will automatically generate a client and run both your main function and a worker to handle consumers.
### `restate main`
Starts the main function as defined in your project definition.
### `restate worker`
Starts a worker which handles running consumers in response to transitions.
### `restate generate`
Regenerates the Restate client and types based on your `*.rst` files.
### `restate migrate`
Automatically sets up tables for all your models. Runs automatically as part of `restate dev`.
## License
Restate is [MIT licensed](LICENSE.md)