{"id":15043082,"url":"https://github.com/jay-johnson/restapi","last_synced_at":"2025-04-14T20:46:35.966Z","repository":{"id":57660641,"uuid":"465084332","full_name":"jay-johnson/restapi","owner":"jay-johnson","description":"A secure-by-default, async Rest API with hyper, tokio, bb8, kafka-threadpool, postgres and prometheus for monitoring. Includes: a working user management and authentication backend written for postgres, async s3 uploading/downloading, async publishing to kafka with mTLS for encryption in transit","archived":false,"fork":false,"pushed_at":"2022-09-25T04:22:21.000Z","size":309,"stargazers_count":18,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-28T09:04:00.175Z","etag":null,"topics":["bb8","docker-compose","ecdsa","encryption","helm","hyper","jwt","kafka","pgadmin4","postgres","prometheus","rest-api","rest-framework","rust","s3-uploader","s3-uploads","tls","tokio","user-auth","user-authenication"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jay-johnson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-01T23:01:40.000Z","updated_at":"2024-06-08T17:25:44.000Z","dependencies_parsed_at":"2022-09-06T07:22:55.712Z","dependency_job_id":null,"html_url":"https://github.com/jay-johnson/restapi","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Frestapi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Frestapi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Frestapi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Frestapi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jay-johnson","download_url":"https://codeload.github.com/jay-johnson/restapi/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248960752,"owners_count":21189988,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bb8","docker-compose","ecdsa","encryption","helm","hyper","jwt","kafka","pgadmin4","postgres","prometheus","rest-api","rest-framework","rust","s3-uploader","s3-uploads","tls","tokio","user-auth","user-authenication"],"created_at":"2024-09-24T20:48:32.803Z","updated_at":"2025-04-14T20:46:35.933Z","avatar_url":"https://github.com/jay-johnson.png","language":"Rust","readme":"# Rust Rest API Stack with User Management, Kafka Message Publishing, S3 uploads/downloads, and Prometheus for Monitoring\n\nA secure-by-default Rest API using [hyper](https://crates.io/crates/hyper), [tokio](https://crates.io/crates/tokio), [bb8](https://crates.io/crates/bb8), [kafka-threadpool](https://crates.io/crates/kafka-threadpool), postgres, and [prometheus](https://crates.io/crates/prometheus) for monitoring.\n\n## Features\n\n1.  User management and authentication stored in postgres\n1.  Async s3 uploading and downloading (to/from local files or to/from memory)\n1.  Decoupled, async kafka threadpool that uses environment variables to connect to a kafka cluster with client mtls for authentication and encryption in transit\n1.  Async publishing for all successful user events to a kafka topic (topic default: ``user.events``) and partition key (key default: ``user-{user.id}``)\n1.  Async kafka messaging for one-off messages using custom kafka topic(s), partition key(s) and custom header(s).\n\n## Overview\n\n### User\n\n- User password reset and user email change support using one-time-use tokens that are stored in postgres.\n- Users can upload and manage files stored on AWS S3 (assuming valid credentials are loaded outside this rust project).\n- User passwords are hashed using [argon2](https://docs.rs/argon2/latest/argon2/).\n\n### Auth\n\n- User authentication enabled by default\n- Default JWT signing keys included with [documentation for building new keys as needed](https://github.com/jay-johnson/restapi/tree/main/jwt).\n\n### Database\n\n- The rest api server utilizes postgres with a [bb8 client threadpool](https://github.com/djc/bb8).\n- The postgres database requires each client connection include the postgres tls certificate authority file for encrypting data in-transit.\n- Includes [pg4admin](https://www.pgadmin.org/docs/pgadmin4/latest/index.html) for database management in a browser (deployed with podman compose).\n\n### TLS Encryption\n\n- Includes a tls asset generator tool ([./tls/create-tls-assets.sh](https://github.com/jay-johnson/restapi/blob/main/tls/create-tls-assets.sh)) for building self-signed tls assets including your own private Certificate Authority (CA).\n\n#### Ingress\n\nComponent        | Status\n---------------- | ------\nRest API Server  | Listening for encrypted client connections on tcp port **3000**\nPostgres         | Listening for encrypted client connections on tcp port **5432** (tls Certificate Authority required)\npgAdmin          | Listening for encrypted HTTP client connections on tcp port **5433**\n\n## Getting Started\n\n### Clone the repo\n\n```bash\ngit clone https://github.com/jay-johnson/restapi\ncd restapi\n```\n\n### Generate TLS Assets and a Private Certificate Authority (CA) using CFSSL\n\nGenerate new tls assets under the ``./tls`` directory with these commands:\n\n```bash\ncd tls\n./create-tls-assets.sh\ncd ..\n```\n\nPlease refer to the [Generating TLS Assets with CFSSL](./tls/README.md) for more information.\n\n### Generate JWT Private and Public Signing Keys\n\nGenerate new signing JWT keys under the ``./jwt`` directory with these commands:\n\n```bash\ncd jwt\n./recreate-jwt.sh\ncd ..\n```\n\nPlease refer to the [How to build JWT private and public keys for the jsonwebtokens crate doc](./jwt/README.md) for more information.\n\n### Deploy Postgres and pgAdmin using Podman\n\nPlease refer to the [Build and Deploy a Secured Postgres backend doc](./docker/db/README.md) for more information.\n\n### Build API Server\n\n```bash\ncargo build --example server\n```\n\n### Run API Server\n\n```bash\nexport RUST_BACKTRACE=1 \u0026\u0026 export RUST_LOG=info,kafka_threadpool=info \u0026\u0026 ./target/debug/examples/server\n```\n\n## Environment Variables\n\n### Rest API\n\nEnvironment Variable  | Default\n--------------------- | -------\nSERVER_NAME_API       | api\nSERVER_NAME_LABEL     | rust-restapi\nAPI_ENDPOINT          | 0.0.0.0:3000\nAPI_TLS_DIR           | ./tls/api\nAPI_TLS_CA            | ./tls/ca/ca.pem\nAPI_TLS_CERT          | ./tls/api/server.pem\nAPI_TLS_KEY           | ./tls/api/server-key.pem\n\n### User Email Verification\n\nEnvironment Variable                   | Default\n-------------------------------------- | -------\nUSER_EMAIL_VERIFICATION_REQUIRED       | \"0\"\nUSER_EMAIL_VERIFICATION_ENABLED        | \"1\"\nUSER_EMAIL_VERIFICATION_EXP_IN_SECONDS | \"2592000\"\n\n### User One-Time-Use Token Expiration for Password Recovery\n\nEnvironment Variable    | Default\n----------------------- | -------\nUSER_OTP_EXP_IN_SECONDS | \"2592000\"\n\n### Postgres Database\n\nEnvironment Variable  | Default\n--------------------- | -------\nPOSTGRES_USERNAME     | datawriter\nPOSTGRES_PASSWORD     | \"123321\"\nPOSTGRES_ENDPOINT     | 0.0.0.0:5432\nPOSTGRES_TLS_DIR      | ./tls/postgres\nPOSTGRES_TLS_CA       | ./tls/ca/ca.pem\nPOSTGRES_TLS_CERT     | ./tls/postgres/client.pem\nPOSTGRES_TLS_KEY      | ./tls/postgres/client-key.pem\nPOSTGRES_DB_CONN_TYPE | postgresql\n\n### Kafka Cluster\n\nPlease refer to the [kafka_threadpool docs](https://crates.io/crates/kafka-threadpool) for more information.\n\nEnvironment Variable             | Purpose / Value\n-------------------------------- | ---------------\nKAFKA_PUBLISH_EVENTS             | if set to ``true`` or ``1`` publish all user events to kafka\nKAFKA_ENABLED                    | toggle the kafka_threadpool on with: ``true`` or ``1`` anything else disables the threadpool\nKAFKA_LOG_LABEL                  | tracking label that shows up in all crate logs\nKAFKA_BROKERS                    | comma-delimited list of brokers (``host1:port,host2:port,host3:port``)\nKAFKA_TOPICS                     | comma-delimited list of supported topics\nKAFKA_PUBLISH_RETRY_INTERVAL_SEC | number of seconds to sleep before each publish retry\nKAFKA_PUBLISH_IDLE_INTERVAL_SEC  | number of seconds to sleep if there are no message to process\nKAFKA_NUM_THREADS                | number of threads for the threadpool\nKAFKA_TLS_CLIENT_KEY             | optional - path to the kafka mTLS key (./tls/kafka-cluster-0/client-key.pem)\nKAFKA_TLS_CLIENT_CERT            | optional - path to the kafka mTLS certificate (./tls/kafka-cluster-0/client.pem)\nKAFKA_TLS_CLIENT_CA              | optional - path to the kafka mTLS certificate authority (CA) (./tls/ca/ca.pem)\nKAFKA_METADATA_COUNT_MSG_OFFSETS | optional - set to anything but ``true`` to bypass counting the offsets\n\n#### Sample kafka.env file\n\n```bash\n# enable the cluster\nexport KAFKA_ENABLED=1\nexport KAFKA_LOG_LABEL=\"ktp\"\nexport KAFKA_BROKERS=\"host1:port,host2:port,host3:port\"\nexport KAFKA_TOPICS=\"testing\"\nexport KAFKA_PUBLISH_RETRY_INTERVAL_SEC=\"1.0\"\nexport KAFKA_NUM_THREADS=\"5\"\nexport KAFKA_TLS_CLIENT_CA=\"./tls/ca/ca.pem\"\nexport KAFKA_TLS_CLIENT_CERT=\"./tls/kafka-cluster-0/client.pem\"\nexport KAFKA_TLS_CLIENT_KEY=\"./tls/kafka-cluster-0/client-key.pem\"\n# the KafkaPublisher can count the offsets for each topic with \"true\" or \"1\"\nexport KAFKA_METADATA_COUNT_MSG_OFFSETS=\"true\"\n```\n\n### S3\n\nEnvironment Variable | Default\n-------------------- | -------\nS3_DATA_BUCKET       | YOUR_BUCKET\nS3_DATA_PREFIX       | /rust-restapi/tests\nS3_STORAGE_CLASS     | STANDARD\nS3_DATA_UPLOAD_TO_S3 | \"0\"\n\n### JWT\n\nEnvironment Variable                 | Default\n------------------------------------ | -------\nTOKEN_EXPIRATION_SECONDS_INTO_FUTURE | \"2592000\"\nTOKEN_ORG                            | example.org\nTOKEN_HEADER                         | Bearer\nTOKEN_ALGO_PRIVATE_KEY               | ./jwt/private-key-pkcs8.pem\nTOKEN_ALGO_PUBLIC_KEY                | ./jwt/public-key.pem\nSERVER_PKI_DIR_JWT                   | ./jwt\nSERVER_PASSWORD_SALT                 | 78197b60-c950-4339-a52c-053165a04764\n\n### Rust\n\nEnvironment Variable | Default\n-------------------- | -------\nRUST_BACKTRACE       | \"1\"\nRUST_LOG             | info\n\n### Debug\n\nEnvironment Variable | Default\n-------------------- | -------\nDEBUG                | \"1\"\n\n## Docker Builds\n\n### Build Base Image\n\nThis will build an initial base image using podman. Note: this base image will **not** work on a different cpu chipset because the openssl libraries are compiled within the image for this base image.\n\n```bash\n./build-base.sh\n```\n\n### Build Derived Image\n\nBy reusing the base image, this derived image only needs to recompile the server. With minimal code changes, this is a much faster build than the base image build.\n\n```bash\n./build-derived.sh\n```\n\n## Kubernetes\n\n### Start Kafka\n\nIf you do not have a running Kafka cluster, you can deploy your own with:\n\nhttps://github.com/jay-johnson/rust-with-strimzi-kafka-and-tls\n\n### Helm Chart\n\n#### Deploy TLS Assets into Kubernetes\n\nThis command will deploy all jwt keys, tls assets and credentials into the ``dev`` namespace:\n\n```bash\n./deploy-kubernetes-assets.sh -e dev\n```\n\n#### Deploy the Rust Rest API into Kubernetes\n\nPlease refer to the [Deploying the Rust Rest API helm chart into kubernetes guide](https://github.com/jay-johnson/restapi/blob/main/charts/rust-restapi/README.md) for deploying the example helm chart into a kubernetes cluster.\n\nBy default this uses the ``jayjohnson/rust-restapi`` container image\n\n```bash\nhelm upgrade --install -n dev dev-api ./charts/rust-restapi -f ./charts/rust-restapi/values.yaml\n```\n\n## Monitoring\n\n### Prometheus\n\nThis section assumes you have a working prometheus instance already running inside kubernetes. Below is the Prometheus ``scrape_config`` to monitor the rest api deployment replica(s) within kubernetes. Note this config also assumes the api chart is running in the ``dev`` namespace:\n\n```yaml\nscrape_configs:\n- job_name: rust-restapi\n  scrape_interval: 10s\n  scrape_timeout: 5s\n  metrics_path: /metrics\n  scheme: https\n  tls_config:\n    insecure_skip_verify: true\n  static_configs:\n  - targets:\n    - dev-api.dev.svc.cluster.local:3000\n```\n\n## Supported APIs\n\nHere are the supported json contracts for each ``Request`` and ``Response`` based off the url. Each client request is handled by the [./src/handle_requests.rs module](./src/handle_request.rs) and returned as a response back to the client (serialization using ``serde_json``)\n\n### User APIs\n\n#### Create User\n\nCreate a single ``users`` record for the new user\n\n- URL path: ``/user``\n- Method: ``POST``\n- Handler: [create_user](https://docs.rs/restapi/latest/restapi/requests/user/create_user/fn.create_user.html)\n- Request: [ApiReqUserCreate](https://docs.rs/restapi/latest/restapi/requests/user/create_user/struct.ApiReqUserCreate.html)\n- Response: [ApiResUserCreate](https://docs.rs/restapi/latest/restapi/requests/user/create_user/struct.ApiResqUserCreate.html)\n\n#### Update User\n\nUpdate supported ``users`` fields (including change user email and password)\n\n- URL path: ``/user``\n- Method: ``PUT``\n- Handler: [update_user](https://docs.rs/restapi/latest/restapi/requests/user/update_user/fn.update_user.html)\n- Request: [ApiReqUserUpdate](https://docs.rs/restapi/latest/restapi/requests/user/update_user/struct.ApiReqUserUpdate.html)\n- Response: [ApiResUserUpdate](https://docs.rs/restapi/latest/restapi/requests/user/update_user/struct.ApiResqUserUpdate.html)\n\n#### Get User\n\nGet a single user by ``users.id`` - by default, a user can only get their own account details\n\n- URL path: ``/user/USERID``\n- Method: ``GET``\n- Handler: [get_user](https://docs.rs/restapi/latest/restapi/requests/user/get_user/fn.get_user.html)\n- Request: [ApiReqUserGet](https://docs.rs/restapi/latest/restapi/requests/user/get_user/struct.ApiReqUserGet.html)\n- Response: [ApiResUserGet](https://docs.rs/restapi/latest/restapi/requests/user/get_user/struct.ApiResqUserGet.html)\n\n#### Delete User\n\nDelete a single ``users`` record (note: this does not delete the db record, just sets the ``users.state`` to inactive ``1``)\n\n- URL path: ``/user``\n- Method: ``DELETE``\n- Handler: [delete_user](https://docs.rs/restapi/latest/restapi/requests/user/delete_user/fn.delete_user.html)\n- Request: [ApiReqUserDelete](https://docs.rs/restapi/latest/restapi/requests/user/delete_user/struct.ApiReqUserDelete.html)\n- Response: [ApiResUserDelete](https://docs.rs/restapi/latest/restapi/requests/user/delete_user/struct.ApiResqUserDelete.html)\n\n#### Search Users in the db\n\nSearch for matching ``users`` records in the db\n\n- URL path: ``/user/search``\n- Method: ``POST``\n- Handler: [search_users](https://docs.rs/restapi/latest/restapi/requests/user/search_users/fn.search_users.html)\n- Request: [ApiReqUserSearch](https://docs.rs/restapi/latest/restapi/requests/user/search_users/struct.ApiReqUserSearch.html)\n- Response: [ApiResUserSearch](https://docs.rs/restapi/latest/restapi/requests/user/search_users/struct.ApiResqUserSearch.html)\n\n#### Create One-Time-Use Password Reset Token (OTP)\n\nCreate a one-time-use password reset token that allows a user to change their ``users.password`` value by presenting the token\n\n- URL path: ``/user/password/reset``\n- Method: ``POST``\n- Handler: [create_otp](https://docs.rs/restapi/latest/restapi/requests/user/create_otp/fn.create_otp.html)\n- Request: [ApiReqUserCreateOtp](https://docs.rs/restapi/latest/restapi/requests/user/create_otp/struct.ApiReqUserCreateOtp.html)\n- Response: [ApiResUserCreateOtp](https://docs.rs/restapi/latest/restapi/requests/user/create_otp/struct.ApiResUserCreateOtp.html)\n\n#### Consume a One-Time-Use Password Reset Token (OTP)\n\nConsume a one-time-use password and change the user's ``users.password`` value to the new argon2-hashed password\n\n- URL path: ``/user/password/change``\n- Method: ``POST``\n- Handler: [consume_user_otp](https://docs.rs/restapi/latest/restapi/requests/user/consume_user_otp/fn.consume_user_otp.html)\n- Request: [ApiReqUserConsumeOtp](https://docs.rs/restapi/latest/restapi/requests/user/consume_user_otp/struct.ApiReqUserConsumeOtp.html)\n- Response: [ApiResUserConsumeOtp](https://docs.rs/restapi/latest/restapi/requests/user/consume_user_otp/struct.ApiResUserConsumeOtp.html)\n\n#### Verify a User's email\n\nConsume a one-time-use verification token and change the user's ``users.verified`` value verified (``1``)\n\n- URL path: ``/user/verify``\n- Method: ``GET``\n- Handler: [verify_user](https://docs.rs/restapi/latest/restapi/requests/user/verify_user/fn.verify_user.html)\n- Request: [ApiReqUserVerify](https://docs.rs/restapi/latest/restapi/requests/user/verify_user/struct.ApiReqUserVerify.html)\n- Response: [ApiResUserVerify](https://docs.rs/restapi/latest/restapi/requests/user/verify_user/struct.ApiResUserVerify.html)\n\n### User S3 APIs\n\n#### Upload a file asynchronously to AWS S3 and store a tracking record in the db\n\nUpload a local file on disk to AWS S3 asynchronously and store a tracking record in the ``users_data`` table. The documentation refers to this as a ``user data`` or ``user data file`` record.\n\n- URL path: ``/user/data``\n- Method: ``POST``\n- Handler: [upload_user_data](https://docs.rs/restapi/latest/restapi/requests/user/upload_user_data/fn.upload_user_data.html)\n- Request: [ApiReqUserUploadData](https://docs.rs/restapi/latest/restapi/requests/user/upload_user_data/struct.ApiReqUserUploadData.html)\n- Response: [ApiResUserUploadData](https://docs.rs/restapi/latest/restapi/requests/user/upload_user_data/struct.ApiResUserUploadData.html)\n\n#### Update an existing user data file record for a file stored in AWS S3\n\nUpdate the ``users_data`` tracking record for a file that exists in AWS S3\n\n- URL path: ``/user/data``\n- Method: ``PUT``\n- Handler: [update_user_data](https://docs.rs/restapi/latest/restapi/requests/user/update_user_data/fn.update_user_data.html)\n- Request: [ApiReqUserUpdateData](https://docs.rs/restapi/latest/restapi/requests/user/update_user_data/struct.ApiReqUserUpdateData.html)\n- Response: [ApiResUserUpdateData](https://docs.rs/restapi/latest/restapi/requests/user/update_user_data/struct.ApiResUserUpdateData.html)\n\n#### Search for existing user data files from the db\n\nSearch for matching records in the ``users_data`` db based off the request's values\n\n- URL path: ``/user/data/search``\n- Method: ``POST``\n- Handler: [search_user_data](https://docs.rs/restapi/latest/restapi/requests/user/search_user_data/fn.search_user_data.html)\n- Request: [ApiReqUserSearchData](https://docs.rs/restapi/latest/restapi/requests/user/search_user_data/struct.ApiReqUserSearchData.html)\n- Response: [ApiResUserSearchData](https://docs.rs/restapi/latest/restapi/requests/user/search_user_data/struct.ApiResUserSearchData.html)\n\n### User Authentication APIs\n\n#### User Login\n\nLog the user in and get a json web token (jwt) back for authentication on subsequent client requests\n\n- URL path: ``/login``\n- Method: ``POST``\n- Handler: [login](https://docs.rs/restapi/latest/restapi/requests/auth/login_user/fn.login_user.html)\n- Request: [ApiReqUserLogin](https://docs.rs/restapi/latest/restapi/requests/auth/login_user/struct.ApiReqUserLogin.html)\n- Response: [ApiResUserLogin](https://docs.rs/restapi/latest/restapi/requests/auth/login_user/struct.ApiResUserLogin.html)\n\n## Integration Tests\n\nThis project focused on integration tests for v1 instead of only rust tests (specifically everything has been tested with **curl**):\n\nPlease refer to the [Integration Tests Using curl Guide](./tests/integration-using-curl.md)\n\n## Podman Image Push\n\n```bash\ncur_tag=$(cat Cargo.toml | grep version | head -1 | sed -e 's/\"//g' | awk '{print $NF}')\npodman tag IMAGE_ID \"docker://docker.io/jayjohnson/rust-restapi:${cur_tag}\"\npodman tag IMAGE_ID \"docker://docker.io/jayjohnson/rust-restapi:latest\"\npodman push \"docker.io/jayjohnson/rust-restapi:${cur_tag}\"\npodman push \"docker.io/jayjohnson/rust-restapi:latest\"\n```\n\n## Build Docs\n\n```bash\ncargo doc --example server\n```\n\n## Start Development Server\n\n```bash\nsource ./env/api.env \u0026\u0026 source ./env/kafka.env \u0026\u0026 source ./env/postgres.env \u0026\u0026 cargo build --example server \u0026\u0026 export RUST_BACKTRACE=1 \u0026\u0026 export RUST_LOG=info,kafka_threadpool=info,rdkafka=error \u0026\u0026 ./target/debug/examples/server\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Frestapi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjay-johnson%2Frestapi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Frestapi/lists"}