{"id":38081940,"url":"https://github.com/bryanhughes/proto-crudl","last_synced_at":"2026-01-16T20:49:46.640Z","repository":{"id":160213715,"uuid":"342939844","full_name":"bryanhughes/proto-crudl","owner":"bryanhughes","description":"A simple Erlang database mapping tool that maps protobuffers to a relational schema, generating the .proto as well as all the CRUDL (Create, Read, Update, Delete, and List/Lookup) code. You can now serialize protobuffers from a web client straight to the database using protobuffers. Supports integration with gpb. Currently only support Postgres.","archived":false,"fork":false,"pushed_at":"2023-11-14T17:18:54.000Z","size":211,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2023-11-14T18:46:31.490Z","etag":null,"topics":["erlang","postgres","protobuf","protobuffer"],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bryanhughes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-02-27T19:24:00.000Z","updated_at":"2022-03-05T22:27:17.000Z","dependencies_parsed_at":"2023-11-14T18:41:30.941Z","dependency_job_id":"cdb9d2f0-9a59-437d-bc5d-485a8db6e685","html_url":"https://github.com/bryanhughes/proto-crudl","commit_stats":null,"previous_names":[],"tags_count":3,"template":null,"template_full_name":null,"purl":"pkg:github/bryanhughes/proto-crudl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanhughes%2Fproto-crudl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanhughes%2Fproto-crudl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanhughes%2Fproto-crudl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanhughes%2Fproto-crudl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bryanhughes","download_url":"https://codeload.github.com/bryanhughes/proto-crudl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanhughes%2Fproto-crudl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28482328,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["erlang","postgres","protobuf","protobuffer"],"created_at":"2026-01-16T20:49:46.574Z","updated_at":"2026-01-16T20:49:46.625Z","avatar_url":"https://github.com/bryanhughes.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"# proto-crudl\n------\nAn escript that generates protobuffers and Erlang CRUDL (Create, Read, Update, Delete, List/Lookup) based on your\nrelational data model. The tool will generate code that supports either records or maps. The tool supports the \nErlang [gpb](https://github.com/tomas-abrahamsson/gpb) library, though not required. Currently, it only supports Postgres. \n\nAside from simple CRUDL, this tool allows you to generate functions based on any standard SQL query, lookups, and\ntransformations. The custom mappings and transformations are very powerful. \n\nPlease look at the `example` directory to an example schema and configuration file to generate not just simple\nCRUD, but search/lookup operations based on indexed fields, as well as custom mapping queries including how to use\ntransformations to work with PostGis where you want to tranform lat/lon to geography columns. \n\n# Installing Erlang\nMake sure you have a minimum of Erlang/OTP 23 installed. Currently, this has only been tested on MacOS.\n\n## Mac OS X\nUsing Homebrew:\n\n    brew install erlang\n\nUsing MacPorts:\n\n    sudo port install erlang\n\n## Linux\nMost operating systems have pre-built Erlang distributions in their package management systems.\nFor Ubuntu/Debian:\n\n    sudo apt-get update\n    sudo apt-get install erlang\n\nFor Fedora:\n\n    sudo yum install erlang\n\nFor FreeBSD\n\n    sudo pkg update\n    sudo pkg install erlang\n\n##  Windows\n\n[Download the Window installer](https://www.erlang.org/downloads)\n\n\u003chr\u003e\n\n# Download and install Rebar3\nPlease follow the [getting started instruction](https://rebar3.readme.io/docs/getting-started)\n\n\u003chr\u003e\n\n# Install Docker\nIf you are setting up docker for the first time on Ubuntu using snap, the follow these instructions. Due to the \nconfinement issues on snappy, it requires some manual setup to make docker-snap works on your machine.\n\n## Mac OS X\nUsing Homebrew:\n\n    brew install --cask docker\n    open /Applications/Docker.app\n\n## Ubuntu\nOn Ubuntu classic, before installing the docker snap,\nplease run the following command to add the login user into docker group.\n\n    sudo addgroup --system docker\n    sudo adduser $USER docker\n    newgrp docker\n\nOnce installed, you can then manage the services\n\n    sudo snap services\n    \nTo start the docker service\n\n    sudo snap start docker\n    \n## Setting Up Database For Building The Examples\nFirst, start the docker image in the examples subdirectory and then create the `proto_crudl` Role and Database with \ncreate database and superuser privileges and password `proto_crudl`.\n\n    cd example\n    docker-compose up\n \nNow log in as the user locally to postgres running in the container\n\n    psql -h localhost -p 5432 -U proto_crudl\n\nor you can try...\n\n    docker run -it --link some-postgis:postgres --rm postgres sh -c 'exec psql -h \"$POSTGRES_PORT_5432_TCP_ADDR\" -p \"$POSTGRES_PORT_5432_TCP_PORT\" -U proto_crudl'\n\n\n## Resetting the Example Database\nThis helper script will allow you to rapidly drop and recreate your database. This expects the role and database `proto_crudl`\n\n    bin/reset_db.sh\n\n\u003chr\u003e\n\nBuild\n-----\nFrom the top level directory\n\n    rebar3 escriptize\n\nRun\n---\n\n    cd example\n    bin/generate.sh\n\nor directly\n\n    _build/default/bin/proto_crudl \u003cconfig\u003e\n\nTest\n----\nSeveral of the eunit tests rely on the presence of the example database. Currently unit and functional tests are commingled.\n\n    rebar3 eunit\n\n**Note**, when rerunning eunit tests after generating the example code, you must `reset` the database otherwise the `version` \ncolumns may still be present and cause the tests to faile.\n\n\n## Building the Example\nThe project includes an example schema and scripts to build located in the `example` directory. You\nwill find the build scripts in `example/bin` to create or reset the example database as well as to generate\nthe code in the example schema, which is located in `example/database`. The schema was generated using [DbSchema](https://www.dbschema.com/).\n\n**PLEASE NOTE:** You will see errors and warnings in the output. This is intentional as the example schema includes a\nlot of corner cases, like a table without a primary key and unsupported postgres types.\n\n### Testing the Generated Code    \nThe example code also contains some tests that tests the results of the generated code. To run these tests They \nare located in `example/apps/test/example_test.erl` and do full CRUDL test.\n\n    cd example\n    rebar3 eunit\n\n\nPlease refer to these test to better understand how to use the generated code.\n    \n# Using proto_crudl \nCurrently, I have not had time to write a github pull script for the `escript` executable \nartifact: `_build/default/bin/proto_crudl`. At this time I manually copy the file to my project. So not build friendly\njust yet.\n\n## Mapping to Protobuffers\nThis is important to read. One of the most significant disjointed aspects of using Erlang records that are mapped to \nProtobuffers is the how `timestamps` are supported. For protobuffers, there is only a `timestamp` data type while in \nErlang, and relational databases, there is support for multiple datetime data types including `date` and `timestamp`.\n\n`proto_crudl` is a framework that generates Erlang code the maps between a relational database, Erlang\nmaps or records, and protobuffers, we make a conscious choice to err on the side of native support. If you are \nusing the default `gpb` generated records, it presumes that all datetime fields are timestamps. This means that \n`proto_crudl` will treat datetimes as native Erlang data types (i.e. `{{Year, Month, Day}, {Hour, Minute, Seconds}}` and \n`{Year, Month, Day}`). When serializing a protobuffer across the wire with `gpb` you must call `to_proto/1` and \n`from_proto/1` to encode and decode to a `google.protobuf.Timestamp`.\n\nThere are built-in helper functions called `ts_encode/1`, `ts_decode/1`, `date_encode/1`, and `date_decode/1` when \noperating on the record values natively in Erlang.\n\n## proto_crudl.config\nYou will want to look at [example/config/proto_crudl.config](example/config/proto_crudl.config) as a guide\nfor your own config. It gives a complete example with inline documentation of the current functionality of the tool.\n\nAlmost all the configs follow the pattern of a list of tuples where the first element in the tuple is the table, and\nthe second element is then a list. \n\nThe more complex feature of `proto_crudl` is the ability to apply transformations or sql functions. If you are using PostGIS,\nor need to apply other functions, you will need to use this feature.\n\n```\n{transforms, [\n    {\"test_schema.user\", [\n        {insert, [{\"geog\", \"ST_POINT($lon, $lat)::geography\"}]},\n        {update, [{\"geog\", \"ST_POINT($lon, $lat)::geography\"}]},\n        {select, [{\"lat\", \"ST_Y(geog::geometry)\"},\n                  {\"lon\", \"ST_X(geog::geometry)\"}]}]},\n    {\"public.foo\", [\n        {select, [{\"foobar\", \"1\"}]}]}\n]}.\n```\nFollowing the same pattern of a list of tables with a list of tuples. In the case of converting a `lat` and `lon` to a \n`geography`, you must define each of the operations insert/create, update, and select/read on how the column values will\nbe handled to and from the database. The result is that the columns `lat` and `lon` will be generated as `virtual` \ncolumns in the mapping. Note that when referencing them in the function body (the second element of the tuple), you will\nneed to prepend them with the `$` so that `proto_crudl` knows they are the virtual columns being operated on. \nFor the `insert` operation, a single tuple is defined which will\nresult in the extension function `ST_POINT($lat, $lon)::geography` to be applied to the bind values of the `INSERT` \nstatement. Resulting in the following code:\n\n```erlang\n-define(INSERT, \"INSERT INTO test_schema.user (first_name, last_name, email, user_token, enabled, aka_id, my_array, user_type, number_value, created_on, updated_on, due_date, version, geog) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, 0, ST_POINT($14, $13)::geography) RETURNING user_id, first_name, last_name, email, user_token, enabled, aka_id, my_array, user_type, number_value, created_on, updated_on, due_date, version, ST_Y(geog::geometry) AS lat, ST_X(geog::geometry) AS lon\").\n```\nand\n```\ncreate(M = #{lon := Lon, lat := Lat, due_date := DueDate, updated_on := UpdatedOn, created_on := CreatedOn, number_value := NumberValue, user_type := UserType, my_array := MyArray, aka_id := AkaId, enabled := Enabled, user_token := UserToken, email := Email, last_name := LastName, first_name := FirstName}) when is_map(M) -\u003e\n    Params = [FirstName, LastName, Email, UserToken, Enabled, AkaId, MyArray, user_type_value(UserType), NumberValue, ts_decode_map(CreatedOn), ts_decode_map(UpdatedOn), DueDate, Lat, Lon],\n    case pgo:query(?INSERT, Params, #{decode_opts =\u003e [{return_rows_as_maps, true}, {column_name_as_atom, true}, {decode_fun, fun decode_row/2}]}) of\n        #{command := insert, num_rows := 0} -\u003e\n            {error, failed_to_insert};\n        #{command := insert, num_rows := 1, rows := [Row]} -\u003e\n            {ok, Row};\n        {error, {pgsql_error, #{code := \u003c\u003c\"23505\"\u003e\u003e}}} -\u003e\n             {error, exists};\n        {error, Reason} -\u003e\n            {error, Reason}\n    end;\ncreate(M = #{lon := Lon, lat := Lat, due_date := DueDate, updated_on := UpdatedOn, created_on := CreatedOn, number_value := NumberValue, user_type := UserType, my_array := MyArray, aka_id := AkaId, email := Email, last_name := LastName, first_name := FirstName}) when is_map(M) -\u003e\n    Params = [FirstName, LastName, Email, AkaId, MyArray, user_type_value(UserType), NumberValue, ts_decode_map(CreatedOn), ts_decode_map(UpdatedOn), DueDate, Lat, Lon],\n    case pgo:query(?INSERT_DEFAULTS, Params, #{decode_opts =\u003e [{return_rows_as_maps, true}, {column_name_as_atom, true}, {decode_fun, fun decode_row/2}]}) of\n        #{command := insert, num_rows := 0} -\u003e\n            {error, failed_to_insert};\n        #{command := insert, num_rows := 1, rows := [Row]} -\u003e\n            {ok, Row};\n        {error, {pgsql_error, #{code := \u003c\u003c\"23505\"\u003e\u003e}}} -\u003e\n             {error, exists};\n        {error, Reason} -\u003e\n            {error, Reason}\n    end;\ncreate(_M) -\u003e\n    {error, invalid_map}.\n```\n\nI would recommend that you build the example project and then review the generated code for `test_schema_user_db.erl` to get a better\nunderstanding.\n\n## Updating foreign key values\n`proto-crudl` generates all the Create, Read, Update, Delete, and List/Lookup functions based on the table schema while\nsupporting several key features. It is important to note that guards are needed to keep from accidentally updating foreign\nkey relationships, to this end, the `update` function does not include any foreign key fields, rather, it generates\nspecific update functions for the foreign key.\n\n## The special version column\nThe proto_crudl framework implements all the necessary code to support handling stale changes to a record with a version column. \n\n    {options, [{version_column, \"version\"}, indexed_lookups, check_constraints_as_enums]},\n\nIn this example, the column is called `version`. When a table has this column, proto_crudl will generate code that will \nautomatically handle updating the column value on a good update, as well as guard against updating the record from a \nstale update. If the current value of `version` is `100` and you attempt to update using a map or record that has the version value \nof 90, the update will return with `notfound`, otherwise update returns `{ok, Map}` or `{ok, Record}` (depending on how you generated your code).\n\nIf the column is not present, the tool will automatically inject the column into the table by performing an `ALERT TABLE ...`\n\nIn our example database, the `user` table has a column named `version`. \n```\n-define(UPDATE, \"UPDATE test_schema.user SET first_name = $2, last_name = $3, email = $4, user_token = $5, enabled = $6, aka_id = $7, my_array = $8, user_type = $9, number_value = $10, created_on = $11, updated_on = $12, due_date = $13, version = version + 1, geog = ST_POINT($16, $15)::geography WHERE user_id = $1 AND version = $14 RETURNING user_id, first_name, last_name, email, user_token, enabled, aka_id, my_array, user_type, number_value, created_on, updated_on, due_date, version, ST_Y(geog::geometry) AS lat, ST_X(geog::geometry) AS lon\").\n```\n\nThis results in the following code generation and logic for UPDATES. Please note that the INSERT sql and code is also different.\n```\nupdate(M = #{lon := Lon, lat := Lat, version := Version, due_date := DueDate, updated_on := UpdatedOn, created_on := CreatedOn, number_value := NumberValue, user_type := UserType, my_array := MyArray, aka_id := AkaId, enabled := Enabled, user_token := UserToken, email := Email, last_name := LastName, first_name := FirstName, user_id := UserId}) when is_map(M) -\u003e\n    Params = [UserId, FirstName, LastName, Email, UserToken, Enabled, AkaId, MyArray, user_type_value(UserType), NumberValue, ts_decode_map(CreatedOn), ts_decode_map(UpdatedOn), DueDate, Version, Lat, Lon],\n    case pgo:query(?UPDATE, Params, #{decode_opts =\u003e [{return_rows_as_maps, true}, {column_name_as_atom, true}, {decode_fun, fun decode_row/2}]}) of\n        #{command := update, num_rows := 0} -\u003e\n            notfound;\n        #{command := update, num_rows := 1, rows := [Row]} -\u003e\n            {ok, Row};\n        {error, Reason} -\u003e\n            {error, Reason}\n    end;\nupdate(_M) -\u003e\n    {error, invalid_map}.\n```\n\n## erleans/pgo\n\nFinally, `proto_crudl` uses [erleans/pgo](https://github.com/erleans/pgo) for its Postgres connectivity (which is currently)\nthe only supported database. If you want the pgo client to return UUID as binary strings, set an application environment\nvariable or in your sys.config:\n\n    {pg_types, [{uuid_format, string}]},\n\n    {pgo, [{pools, [{default, #{pool_size =\u003e 10,\n                                host =\u003e \"127.0.0.1\",\n                                database =\u003e \"proto_crudl\",\n                                user =\u003e \"proto_crudl\",\n                                password =\u003e \"proto_crudl\"}}]}]}\n\nSize the pool according to your requirements.    \n    \n## Generating Protobuffers\n\nproto_crudl will generate a `.proto` that maps to each table in the schema (unless explicitly excluded). \n\nThe package for each proto will be the schema that the table is located in. The `.proto` files will be generated in \nthe `output` directory specified in the config file with those proto to table mappings being written to a subdirectory \nthat corresponds to the schema the table is in.  \n\nRelational databases are inherently namespaced by schema. This means that when supporting multiple schemas, and the fact that Erlang has\nno concept of namespaces, the module name will be the name of the table prepended by the schema name (if set in the config).\nNote that protobuffers are correctly generated where the schema maps to a package.\n\nIt is recommended that you download and install the protocol buffer compiler, which is necessary to compile to other\nlanguages such as Java, Objective-C, or even Go. If you are new to protocol buffers, start\n[by reading the developer docs](https://developers.google.com/protocol-buffers/).\n\nIt is important to note that the default configuration is to use `gpb` to compile and support protobuffers in Erlang.\n\nPLEASE NOTE: There is a bug that has been filed in `gpb` where protos of the same name in different packages will get \noverwritten since there is no namespacing.\n\n# Using In Your Project\nYou will need to copy the `proto_crudl` escript to your project.\n\nNext, create your `proto_crudl.config`. You can simply copy and modify the one in the repo. Currently, I have a manual\ngenerate script that I locate in the top level `bin` directory:\n\n```\n#!/usr/bin/env bash\n\nDIR=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" \u003e/dev/null 2\u003e\u00261 \u0026\u0026 pwd )\"\n\n\"$DIR\"/proto_crudl \"$DIR\"/../config/proto_crudl.config\n```\n\n## Using the Maps or Records\n\n`proto_crutl` supports code generation using both maps and records. It is important to note that there are several issues\nwith using maps with `gpb` and `pgo` that do align very well when using together. \n\nFirst `pgo` favors maps and uses the atom (as expected) `null` for NULL values. Unfortunately, here `gpb` seems to \nfavor records when serializing and deserializing, this is likely a bug. For now, `proto_crudl` generates code based on\neach library implementations support for maps and records. When using the `maps` config option with `gpb`, fields that are\nexplicitly `undefined` are not handled properly and cause an exception as the library attempts to serialize them. The\n`gpb` generated code for records does not have this issue. As expected, `undefined` is treated as missing.\n\nNext, `pgo` does not handle query parameters that have `undefined` present. While this is strictly correct, it breaks\nwhen trying to map between the code generated by `gpo` and query parameters expected by `pgo`. This project includes a\nforked version of `pgo` that is more lenient and allows `undefined` to also mean `null`. This has not been committed \nback to the original project yet.\n\nBecause of the subtle misalignment between libraries, using maps requires the use of `to_proto/1` and `from_proto/1` \nto correctly convert the serialized protobuffers as maps as input values to queries. It also means that the developer\nhas to handle `null` values from the `pgo` and ultimately `proto_crutl` and remember to convert to `undefined` when\nserializing. Maps are very flexible, but also very problematic.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanhughes%2Fproto-crudl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbryanhughes%2Fproto-crudl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanhughes%2Fproto-crudl/lists"}