Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/technologiestiftung/stadtpuls-api

API of stadtpuls.com, an open IoT platform for storing and visualizing sensor data and making it accessible via a REST API
https://github.com/technologiestiftung/stadtpuls-api

citylab-berlin iot sensors stadtpuls

Last synced: about 1 month ago
JSON representation

API of stadtpuls.com, an open IoT platform for storing and visualizing sensor data and making it accessible via a REST API

Host: GitHub
URL: https://github.com/technologiestiftung/stadtpuls-api
Owner: technologiestiftung
License: mit
Archived: true
Created: 2021-04-08T08:08:20.000Z (over 3 years ago)
Default Branch: main
Last Pushed: 2023-02-17T03:31:46.000Z (over 1 year ago)
Last Synced: 2024-09-24T03:33:40.116Z (about 1 month ago)
Topics: citylab-berlin, iot, sensors, stadtpuls
Language: TypeScript
Homepage: https://stadtpuls.com/
Size: 2.34 MB
Stars: 6
Watchers: 6
Forks: 0
Open Issues: 11
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE

Awesome Lists containing this project

README

![](https://img.shields.io/badge/Built%20with%20%E2%9D%A4%EF%B8%8F-at%20Technologiestiftung%20Berlin-blue)

# Stadtpuls.com API

This is an fastify based API layer that is used by [technologiestiftung/stadtpuls-frontend](https://github.com/technologiestiftung/stadtpuls-frontend). It does:

- Issuing and maintaining authtokens for verified users
- Recieving POST requests from external sources via The Things Network (TTN) and HTTP for posting them on the users behalf
- Recieving GET requests from users to provide access to sensors and their records
- Wrap supabases signup and login functionality to allow users to provide a username on signup

Within this repo you also can find:

- Code for running a local version of supabase which is also used in integration tests
- Code for provisioning the database. We are SQL scripts only. The workflow is not yet finally defined. There is no fixed way of doing schema migrations yet. Possible tools could be [dbmate](https://github.com/amacneil/dbmate) or once it is stable [the new supabase cli](https://github.com/supabase/cli/tree/new)
- Code for having a small React client to test interaction with this API
- Code for running a Node.js MQTT client for further explorations

The API is deployed using docker on render.com

## Setting Up…

To get the project ready you need to do some tasks.

- Create a supabase project
- Get your service key and `postgresql://…` connection string
- Add your service role key to `.env`
- Provision the dev database
- use the scripts `stadtpuls-supabase/supabase-docker-compose/dockerfiles/postgres/docker-entrypoint-initdb.d/` to give your DB the final touches. Watch out: 00-initial-schema.sql, 01-auth-schema.sql and 02-storage-schema.sql are covered by supabase. You don't need these when working with the cloud. The other SQL scripts
- create replication of users into the public users table (like mentioned in their [docs](https://supabase.io/docs/guides/auth#create-a-publicusers-table))
- disable realtime for all non public tables (see also the link above on the why to do this)
- enable row level security on all tables
- create delete cascades
- create remote procedure calls that allow a user to delete a his account

## Crypto On Tokens

The user can request a JWT (authtokens) and gets a token based on the jwt secret from supabase. This token gets hashed and is used as primary key for the table authtokens. WARNING: This token can also be used to access the supabase API. If you don't want that you need to use a different secret for the signing of the authtokens in `src/lib/authtokens.ts`. This also needs some refactoring of the usage of jwt.sign which currently uses the fastify-jwt plugin.

When a request over TTN, HTTP, or any other integration, comes in we take the token and verify it. Then we look up if there is a token that is aimed at the project and user id encoded in the token. If we find it we can compare the hash (primary key id) against the incoming token. If they match the request is verified and we insert records on the users behalf. If does not match it was deleted or never create in the first place and is not allowed to add records.

## Development

You need Docker and Node.js.

To start your local redis database run the following commands:

```bash
git clone https://github.com/technologiestiftung/stadtpuls-redis
cd stadtpuls-redis/
docker composse up --detach
```

To start you local copy of supabase run the following steps:

```bash
git clone https://github.com/technologiestiftung/stadtpuls-supabase
cd stadtpuls-supabase/supabase-docker-compose
cp .env.example .env
mkdir dockerfiles/postgres/pg-data
docker compose up --detach
```

When your supabase instance is running you can proceed. Test if the supabase is by running the following command. Make sure to replace `` with the anon key you can find in `stadtpuls-supabase/supabase-docker-compose/dockerfiles/kong/kong.yml` at the bottom. The port may change based on `KONG_PORT` in `stadtpuls-supabase/supabase-docker-compose/.env`.

```bash
curl http://localhost:8000/rest/v1/ \
-H "apikey: "
```

To start your local copy of the API create your `.env` file in the root of the repository `cp .env.example .env` and update the values. You can find them in `stadtpuls-supabase/supabase-docker-compose/.env` and `stadtpuls-supabase/supabase-docker-compose/dockerfiles/kong/kong.yml`. Use the `KONG_PORT` for your `SUPABASE_URL` (`http://localhost:`)

```bash
cp .env.example .env
nvm install
npm ci
npm run dev
```

## Making Requests

When running the API you will see all possible routes in the output of your terminal. Test if it is running by making a call to unprotected routes.

```bash
curl http://localhost:4000/
curl http://localhost:4000/api
curl http://localhost:4000/api/v3
curl http://localhost:4000/api/v3/sensors
curl http://localhost:4000/api/v3/sensors/:sensorId/records
curl http://localhost:4000/api/v3/sensors/:sensorId/records/:recordId
```

To make calls to protected routes you will need a supabase user token (created by supabase when you signup or login). The following routes can be called with supabase user tokens:

```plain
GET /api/v/authtokens
POST /api/v/authtokens
DELETE /api/v/authtokens
```

The following routes need an auth token created by this API.

```plain
POST /api/v/integrations/ttn/v3
POST /api/v/sensors/:sensorId/records
```

### Create an Auth Token

First you need to signup or login.

```bash
# signup
curl --location --request POST 'http://localhost:8000/auth/v1/signup' \
--header 'apikey: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJzdXBhYmFzZSIsImlhdCI6MTYyNzIwODU0MCwiZXhwIjoxOTc0MzYzNzQwLCJhdWQiOiIiLCJzdWIiOiIiLCJyb2xlIjoiYW5vbiJ9.sUHErUOiKZ3nHQIxy-7jND6B80Uzf9G4NtMLmL6HXPQ' \
--header 'Content-Type: application/json' \
--data-raw '{
"email": "[email protected]",
"password": "1234password"
}'
# or login
curl --location --request POST 'http://localhost:8000/auth/v1/token?grant_type=password' \
--header 'apikey: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJzdXBhYmFzZSIsImlhdCI6MTYyNzIwODU0MCwiZXhwIjoxOTc0MzYzNzQwLCJhdWQiOiIiLCJzdWIiOiIiLCJyb2xlIjoiYW5vbiJ9.sUHErUOiKZ3nHQIxy-7jND6B80Uzf9G4NtMLmL6HXPQ' \
--header 'Content-Type: application/json' \
--data-raw '{
"email": "[email protected]",
"password": "1234password"
}'
```

You will get an response that contains an `access_token` property. That can be used to create (POST), rotate (PUT), get (GET) and delete (DELETE) our auth tokens.

```bash
# get all existing tokens
curl --location --request GET 'http://localhost:4000/api/v3/authtokens?projectId=61' \
--header 'Authorization: Bearer '
# create a new token
curl --location --request POST 'http://localhost:4000/api/v3/authtokens' \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json' \
--data-raw '{
"description": "my fancy token"
}'
# delete a token
curl --location --request DELETE 'http://localhost:4000/api/v3/authtokens' \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json' \
--data-raw '{
"tokenId": 28,
}'
```

Once you created a new token via POST you can move on to posting records.

### POST Records via HTTP

To post data via HTTP you need to optain an auth token like described above. Then you can POST data.

```bash
curl --location --request POST 'http://localhost:4000/api/v3/sensors/14/records' \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json' \
--data-raw '{
"latitude": 52.483107,
"longitude": 13.390679,
"altitude": 30,
"measurements": [
1,
2,
3
]
}'
```

### POST Records via TTN

You will need an auth token like described above. Then you can hook up your TTN device to the webhooks in https://eu1.cloud.thethings.network/console/. See our extended documentation on https://stadtpuls.com for further infos.

## Testing

To test the API you need to run the integration tests. You can do this by running `npm test`. Make sure your local supabase is running. Currently the tests use the environment variables from `.env.test`

## Running with Docker

You can run the stadtpuls-api with docker in several ways.

1. Attaching to an already existing local subase instance.
2. Running within your supabase setup.
3. Running with a remote supabase project.

Take a look at [the hub.docker.com page](https://hub.docker.com/repository/docker/technologiestiftung/stadtpuls-api) of the image to see which tag to use. Don't use the latest tag for production.

### Attaching to an already existing local supabase instance

For attaching to the already existing instance use the following `docker-compose.yml`. You should adjust the environment variables to your needs and then run

```bash
# MacOS & Windows
docker compose up
# Linux
docker-compose up
```

```yml
version: "3"
services:
stadtpuls-api:
environment:
PORT: 4000
SUPABASE_URL: http://kong:8000
SUPABASE_SERVICE_ROLE_KEY: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJzdXBhYmFzZSIsImlhdCI6MTYyNzIwNzUzMiwiZXhwIjoxNjkwMjc5NTMyLCJhdWQiOiIiLCJzdWIiOiIiLCJyb2xlIjoic2VydmljZV9yb2xlIn0.hfdXFZV5PdvUdo2xK0vStb1i97GJukSkRqfwd4YIh2M
SUPABASE_ANON_KEY: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJzdXBhYmFzZSIsImlhdCI6MTYyNzIwODU0MCwiZXhwIjoxOTc0MzYzNzQwLCJhdWQiOiIiLCJzdWIiOiIiLCJyb2xlIjoiYW5vbiJ9.sUHErUOiKZ3nHQIxy-7jND6B80Uzf9G4NtMLmL6HXPQ
JWT_SECRET: your-super-secret-jwt-token-with-at-least-32-characters-long
ISSUER: stadtpuls.com
LOG_LEVEL: info
SUPABASE_MAX_ROWS: 1000
DATABASE_URL: postgres://postgres:your-super-secret-and-long-postgres-password@localhost:5432/postgres
image: "technologiestiftung/stadtpuls-api:latest"
ports:
- "4000:4000"
networks:
default:
external: true
name: supabase_default
```

### Running within your supabase setup

- Copy the whole service `stadtpuls-api` to the file `stadtpuls-supabase/supabase-docker-compose/docker-compose.yml`.
- Dont copy the network part.
- Adjust the `SUPABASE_URL` to (TBD)
- Adjust the `DATABASE_URL` to (TBD)

```bash
cd stadtpuls-supabase/supabase-docker-compose/
# if you had the whole setup already running
docker compose down && rm -rf dockerfiles/postgres/pg-data/ && mkdir dockerfiles/postgres/pg-data
# MacOS & Windows
docker compose up --build --force-recreate
# Linux
docker-compose up --build --force-recreate
```

### Running with a remote supabase project

- Adjust the `SUPABASE_URL`
- Adjust the `DATABASE_URL`
- Adjust the `SUPABASE_SERVICE_ROLE_KEY`
- Adjust the `SUPABSE_ANON_KEY`
- Adjust the `JWT_SECRET`

You can find these values under `https://app.supabase.io/project//settings/api`

```bash
# MacOS & Windows
docker compose up
# Linux
docker-compose up
```

## Contributors

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

_{Fabian Morón Zirfas}
💻 📖

_{Dennis Ostendorf}
👀 💻 📖

_{Lucas Vogel}
👀 💻 📖

_lucasoeth
💻

_{Julia Zet}
💻

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!

## Credits

A project by:

Supported by: