{"id":13741858,"url":"https://github.com/jackskj/protoc-gen-map","last_synced_at":"2025-05-08T22:32:34.787Z","repository":{"id":57501487,"uuid":"216451528","full_name":"jackskj/protoc-gen-map","owner":"jackskj","description":"SQL Data mapper framework for grpc/protobuf","archived":true,"fork":false,"pushed_at":"2020-08-31T03:31:18.000Z","size":290,"stargazers_count":129,"open_issues_count":4,"forks_count":11,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-11-27T15:39:44.862Z","etag":null,"topics":["go","grpc","protobuf","sql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jackskj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-21T01:15:53.000Z","updated_at":"2024-10-10T18:08:01.000Z","dependencies_parsed_at":"2022-09-14T19:41:01.244Z","dependency_job_id":null,"html_url":"https://github.com/jackskj/protoc-gen-map","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackskj%2Fprotoc-gen-map","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackskj%2Fprotoc-gen-map/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackskj%2Fprotoc-gen-map/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackskj%2Fprotoc-gen-map/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jackskj","download_url":"https://codeload.github.com/jackskj/protoc-gen-map/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253158420,"owners_count":21863290,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","grpc","protobuf","sql"],"created_at":"2024-08-03T04:01:03.419Z","updated_at":"2025-05-08T22:32:34.466Z","avatar_url":"https://github.com/jackskj.png","language":"Go","funding_links":[],"categories":["Protocol Buffers","Go"],"sub_categories":["Tools"],"readme":"# protoc-gen-map\nWARNING: This project is no longer maintained. For SQL data mapping, use [Carta](https://github.com/jackskj/carta). \n\nprotoc-gen-map simplifies the management of complex datasets by mapping SQL data to protocol buffers. \nAside from defining proto messages and SQL statements, the developer does not need to write any data retrieval or mapping code. \n\n## Approach\nprotoc-gen-map adopts the \"database mapping\" approach (described in Martin Fowler's [book](https://books.google.com/books?id=FyWZt5DdvFkC\u0026lpg=PA1\u0026dq=Patterns%20of%20Enterprise%20Application%20Architecture%20by%20Martin%20Fowler\u0026pg=PT187#v=onepage\u0026q=active%20record\u0026f=false)) which is useful among organizations with strict code review processes and dedicated database modeler teams.\n\nprotoc-gen-map is language agnostic. Any language with protocol buffer support can request and retrieve data over gRPC using defined messages.\n\nThis framework is not an object-relational mapper(ORM). With large and complex datasets, using ORMs becomes restrictive and reduces performance when working with complex queries. \n\n## SQL Templating\n\nprotoc-gen-map uses golang's template engine (text/template). This allows developers to dynamically modify sql parameters based on the gRPC request message, use if statements or for loops, as well as split large SQL statements into multiple logical blocks. More in the examples below.  \n\n## Examples and Guides\n### Simple example\nLet's use a very simple schema\n\n[![SimpleSchema](https://i.ibb.co/DQsBgBv/Simple-Schema.png \"SimpleSchema\")](https://i.ibb.co/DQsBgBv/Simple-Schema.png \"SimpleSchema\")\n\nSay we want to retrieve blog information based on some request.\nTo do so we can create a gRPC service and SQL template as follows\n\nproto file:\n```\nservice BlogService {\n    rpc SelectBlog (BlogRequest) returns (BlogResponse) {}\n    rpc SelectBlogs (BlogRequest) returns (stream BlogResponse) {}\n}\n\nmessage BlogRequest {\n    uint32 id = 1;\n    string author_id  = 2;\n}\n\nmessage BlogResponse {\n    uint32 id = 1;\n    string title  = 2;\n    string author_id  = 3;\n}\n```\nSQL statement using go's template:\n```\n{{ define \"SelectBlog\" }}\n    select id, title, author_id from blog where id = {{ .Id }}  limit 1\n{{ end }}\n\n{{ define \"SelectBlogs\" }}\n    select id, title, author_id from blog where author_id = {{ .AuthorId }}\n{{ end }}\n```\nNow we would need to template the SQL statements based on an incoming request, map the retrieved SQL data to the response, and return the response. protoc-gen-map generates code that accomplishes all this.\n\nIn the above example, the client sending a request to SelectBlog receives a BlogResponse with sql response properly mapped to the BlogResponse proto message.\nStreaming services indicate that we are requesting multiple responses. In this case. SelectBlogs would return all BlogResponses for a given author. \n\n### When things get complex\nThings usually aren't as simple. SQL statements often tend to be complex and lengthy, especially when projects grow in size and feature. \nprotoc-gen-map helps manage those complexities. All that is required are the SQL statement and the corresponding proto messages\n\nNow Lets use a more complex schema, one with has-one and has-many relationships.\n\n[![ComplexSchema](https://i.ibb.co/QQRpTCV/Complex-Schema.png)](https://i.ibb.co/QQRpTCV/Complex-Schema.png)\n\nNow, say we need to retrieve a much more detailed information about a blog, as with the following query.\n```\n{{ define \"SelectDetailedBlog\" }}\nselect\n       id                  as  blog_id,\n       title               as  blog_title,\n       A.id                as  author_id,\n       A.username          as  author_username,\n       A.password          as  author_password,\n       A.email             as  author_email,\n       A.bio               as  author_bio,\n       A.favourite_section as  author_favourite_section,\n       P.id                as  post_id,\n       P.blog_id           as  post_blog_id,\n       P.author_id         as  post_author_id,\n       P.created_on        as  post_created_on,\n       P.section           as  post_section,\n       P.subject           as  post_subject,\n       P.draft             as  draft,\n       P.body              as  post_body,\n       C.id                as  comment_id,\n       C.post_id           as  comment_post_id,\n       C.comment           as  comment_text,\n       T.id                as  tag_id,\n       T.name              as  tag_name\nfrom blog\n       left outer join author A    on  blog.author_id = A.id\n       left outer join post P      on  blog.id = P.blog_id\n       left outer join comment C   on  P.id = C.post_id\n       left outer join post_tag PT on  PT.post_id = P.id\n       left outer join tag T       on  PT.tag_id = T.id\nwhere blog.id =  {{ .Id }}\n{{ end }}\n```\nNote that this query involves a number of has-one and has-many relationships. With protoc-gen-map, all the retrieved rows are mapped into structured proto messages, defined below. There is no need to write any data retrieval or mapping code.\n\n``` \nservice BlogQueryService {\n  rpc SelectDetailedBlog (BlogRequest) returns (DetailedBlogResponse) {}\n}\n\nmessage BlogRequest {\n  uint32 id = 1;\n}\n\nmessage DetailedBlogResponse {\n  uint32 blog_id = 1;\n  string blog_title = 2;\n  Author author = 3;\n  repeated Post posts = 4;\n}\n\nmessage Author {\n  uint32 author_id = 1;\n  string author_username = 2;\n  string author_password = 3;\n  string author_email = 4;\n  string author_bio = 5;\n  Section author_favourite_section = 6;\n}\n\nmessage Post {\n  uint32 post_id = 1;\n  uint32 post_blog_id = 2;\n  uint32 post_author_id = 3;\n  google.protobuf.Timestamp post_created_on = 4;\n  Section post_section = 5;\n  string post_subject = 6;\n  string draft = 7;\n  string post_body = 8;\n  repeated Comment comments = 9;\n  repeated Tag tags = 10;\n}\n\nmessage Comment {\n  uint32 comment_id = 1;\n  uint32 comment_post_id = 2;\n  string comment_name = 3;\n  string comment_text = 4;\n}\n\nmessage Tag {\n  uint32 tag_id = 1;\n  string tag_name = 2;\n}\n\nenum Section {\n  cooking = 0;\n  painting = 1;\n  woodworking = 2;\n  snowboarding = 3;\n}\n```\nClient requesting a detailed blog information will receive the DetailedBlogResponse with properly mapped data from the rows retrieved by the query\n\n## Installation \n```\ngo get -u github.com/jackskj/protoc-gen-map\n```\n## Workflow\nprotoc-sql-map is a code-generating proto plugin and a data mapping framework. This section explains the 3 steps to get started.\nFor an example case, head over to [examples](https://github.com/jackskj/protoc-gen-map/tree/master/examples) \n### 1. Define SQL and Proto\nDefine your SQL statements with go's templating syntax in a file directory. protoc-gen-map will recursively read all sql files in the directory.\nNext, define the corresponding gRPC services and protobuf messages. \nFor instructions on defining SQL/proto head over to [SQL/Proto Definition](https://github.com/jackskj/protoc-gen-map#sqlproto-definition).\n\n### 2. Generate Code\nOnce proto message and SQL templates are created, protoc-gen-map will generate \".pb.map.go\" files containing service servers for each defined server in your proto. The name of the generated servers is your name of the server followed by \"MapServer\".\n\nTo generate the protoc-gem-map, make sure that the protoc-gem-map binary (located in $GOPATH/bin) is in your in your PATH.\nThe following will generate \".pb.go\" (protoc-gen-go with grpc) and \".pb.map.go\" (sql data mapper).\n```\nprotoc --map_out=\"sql=/my/SQL/directory:/my/out/directory\"   \\\n       --go_out=\"plugins=grpc:/my/out/directory\"   \\\n       -I=. \\\n       ./my_proto_file.proto\n```\nMake sure to change the SQL directory, proto files, and out location. \n\n### 3. Create the Server\n\nTo create a server, register an instance of \"MapServer\" struct created by protoc-gen-map. Make sure to provide a database connection object and dialect name.\n\n```\nlis, err := net.Listen(\"tcp\", fmt.Sprintf(\":%d\", myPort))\nif err != nil {\n\t// error when listening on the port fails\n}\ngrpcServer := grpc.NewServer()\ndb, err := sql.Open(dialect, connectionString)\nif err != nil {\n\t// error when connection to DB fails\n}\nmapServer := BlogQueryServiceMapServer{DB: db, Dialect: dialect}\nRegisterBlogQueryServiceServer(grpcServer, \u0026mapServer)\n... \ngrpcServer.Serve(lis)\n```\n\nprotoc-gen-map intends to be a bridge between complex sql statements and defined proto messages. Any languages with protobuf support can be a client. \n\n## Templating Guide\nprotoc-gen-map uses go's \"text/template\" therefore, any valid template function will work. This includes if statements, for loops, and the template command. \nIn addition, any helper functions from [Masterminds/sprig](https://github.com/Masterminds/sprig/) are supported.\n\n### If Statements / For Loops\nWe are able to modify out sql statements based on, say, the request message. In the following example, we will receive blog information for the provided ID. If we do not provide the ID, we get the latest Blog.\n```\n{{ define \"SelectBlog\" }}\n    select id, title, author_id from blog  \n    {{ if .Id }}\n       where id = {{ .Id }}  \n    {{ else }}\n       order by created_at limit 1\n    {{ end }}\n{{ end }}\n```\nWe are also able to use for loops, for more information head over to the [template](https://golang.org/pkg/text/template/) package.\n\n### Sprig functions.\nprotoc-gen-map uses [sprig](https://github.com/Masterminds/sprig/) helper functions, which can come in handy. For example, when client request message contains a list or when we want to filter based on the request, we can use sprig functions like so.\n\nProto:\n```\nmessage BlogRequest {\n  repeated uint32 ids = 1;\n}\n```\nSQL:\n```\n{{ define \"SelectBlog\" }}\n    select id, title, author_id from blog  \n    where id in (\n    {{ .Ids | join \" , \" }} \n    )\n{{ end }}\n```\nNote that we are  joining each title with a comma. In some cases, we will need to quote our input. \n\n### Quoting functions.\nTo  generate correct SQL syntax, we often need to quote our values. To do that, you can use built in function \"quote\" and \"squote\" for  double or single quotations. \nIn addition, you can use \"qouteall\" or \"squoteall\" to quote all repeated items. For example, lets say we want to retrieve blob based on ids or titles. \n```\nmessage BlogRequest {\n  repeated uint32 ids = 1;\n  repeated string titles = 2;\n}\n```\nSQL:\n```\n{{ define \"SelectBlog\" }}\n    select id, title, author_id from blog  \n    where id in (\n    {{ .Ids | join \" , \" }} \n    ) or \n    title in (\n    {{ .Titles | squoteall | join \" , \" }} --strings must be quoted\n    )\n{{ end }}\n```\n\n### Splitting SQL Statements \nAt some point, SQL queries can get big, very big. To help manage lengthy statements, you can divide your sql statement into logical components. Say that we may or may not want to receive blog responses in order. Our request would look like this.\n```\nmessage BlogRequest {\n  repeated uint32 ids = 1;\n  bool order = 2;\n}\n```\n\nWe could template our statement in multiple parts like this.  \n```\n{{ define \"myOrderStatement\" }}\n    order by title\n{{ end }}\n\n{{ define \"SelectBlog\" }}\n    select id, title, author_id from blog  \n    where id in (\n    {{ .Ids | join \" , \" }} \n    )\n    {{ if .Order }}\n        {{ template \"myOrderStatement\" }}\n    {{end}}\n{{ end }}\n```\n\n## Parameterized Queries\n\nTo prevent potential SQL injection when exposing your service, you can use the built in \"param\" function to pass arguments as sql parameters.\n\n\nThis is expecially usefull if your request messages contain sensitive fields of string type.\n\n\nFor example, assumme the following request and sql pairs.\n\n```\nmessage AddrRequest {\n  string username = 1;\n}\n```\n\n```\n{{ define \"ParameterizedQuerie\" }}\n    select addr from user_addresses where username = {{ param .Username }} \n{{ end }}\n```\nThe above query will translate to \"select addr from user_addresses where username = $1\" for postgres (? for mysql).\n\nNote: You must specify the database dialect name in the mapper object to use this feature. Supported dialects include: mysql, postgres, mssql, and sqlite3.\n\n## Callbacks\n\nTo customise protoc-gen-map to your logic needs, the developer is able to specify callback functions which will be run before or after query execution. \n\nIf your application requires query caching, custom monitoring, sending custom API request, or etc, callbacks are they way to go.\n\n### Defining Callbacks\n\nTo register a callback function for a particular RPC. protoc-gen-map creates a register methods in the format: \n1. Before query execution: \n```\nRegister{{ RPC Name}}BeforeQueryCallback(myFunc func(queryString string, req {{ Proto Request Type }}) error)\n```\n\n2. After query execution:\n```\nRegister{{ RPC Name}}AfterQueryCallback(myFunc func(queryString string, req {{ Request Type }}, resp {{ Response Type }}) error)\n```\n\n3. For Caching (described below):\n```\nRegister{{ RPC Name}}Cache(myFunc func(queryString string, req {{ Request Type }}) ({{ Response Type }}, error))\n```\nWhere the Response Type is:\n\n   a. Pointer to a proto response for unary services.\n\n   b. Slice of pointers to a proto responses for streaming services.\n\nFor example, If we would like to run custom monitoring before the query is run, We can create the callbacks like so:\n```\n// Instantiate MapServer if not done yet\nmapServer := BlogQueryServiceMapServer{DB: db, Dialect: \"postgres\"}\n\n// Define custom function\nfunc MyFunction(queryString string, req *BlogRequest) error {\n\t// Do some monitoritg\n\treturn nil\n}\n// Register the Callback \nmapServer.RegisterSelectBlogBeforeQueryCallback(MyFunction)\n\n// Move on to register the gRPC and run the server\n\n```\nSimilarly, if we wish to run custom logic after the query has been executed, we can do it like so:\n```\n// For Unary RPC\nfunc MyFunctionU(queryString string, req *BlogRequest, resp *BlogResponse) error {\n\t // run custom logic\n\treturn nil\n}\n// For Streaming RPC\nfunc MyFunctionS(queryString string, req *BlogRequest, resp []*BlogResponse) error {\n\t// run custom logic\n\treturn nil\n}\n// Register the Callbacks\nmapServer.RegisterSelectBlogAfterQueryCallback(MyFunctionU)\nmapServer.RegisterSelectBlogsAfterQueryCallback(MyFunctionS)\n```\nAnd that's it, your registered functions will run every time the RPC is run. \nYou can register multiple callbacks, if you wish.\n\n### Caching\nWith large and complex queries, it's a good idea to implement some caching layer. This can lead to major improvements in performance of your service, especially if your database grows in size and/or your app becomes more complex. \n\nTo populate your cache, use the AfterQueryCallback described above. It provides proto response for specific proto request and query string. For example,\n```\nfunc UpdateCache(queryString string, req *BlogRequest, resp []*BlogResponse) error {\n\t// populate your cache with query or request as keys \n\t// and response as values\n\treturn nil\n}\n```\nTo implement your cache, simply create a function which returns a proto response.\n```\nfunc MyCache(queryString string, req *BlogRequest) ([]*BlogResponse, error) {\n\t// retrieve response from my cache\n\treturn response, nil\n}\n// Register the Cache\nmapServer.RegisterSelectBlogCache(MyCache)\n```\nAnd that's it. Your custom caching function will be execute before querying the DB. \nIf the caching function returns nil response, query will be executed and client will receive a response based on the result.\nOnly one cache function can be registered.\n\n## SQL/Proto Definition\nHere is a list of things to keep in mind when writing your SQL statements and proto files\n1. The name of the defined SQL template must match your proto rpc name.\nFor example, in the following SQL snippet\n```\n{{ define \"SelectBlog\" }}\n```\ncorresponds to the following rpc \n```\nrpc SelectBlog (BlogRequest) returns (BlogResponse) {}\n```\n2. The SQL Template can be populated with request message.\nFor example, for the following rpc\n```\nBlogQueryService {\n  rpc SelectBlog (BlogRequest) returns (BlogResponse) {}\n}\nmessage BlogRequest {\n  uint32 id = 1;\n  uint32 author_id  = 2;\n}\n```\nthe developer can populate the SQL template with \".AuthorId\" and \".Id\" as follows\n```\n{{ .AuthorId }}\n{{ .Id }}\n```\n3. protoc-gen-map expects the returned column names to match either\n - The response message field name or\n - Field name of the response struct generated by go proto compiler or \n - Lower case of either of the above \n\nIf there exists a discrepancy, you can use the standard SQL \"as\" statement to name an alias. For example, the following SQL/proto pairs would match correctly.\n```\nselect\n       id as  blog_id,\n       title as  BlogTitle,\nfrom blog\n```\n```\nmessage BlogResponse {\n  uint32 blog_id = 1;\n  string blog_title  = 2;\n}\n```\n4. Has-one relationships (associations) are identified by nested fields. For example, a Blog has one Author\n```\nmessage Blog {\n  uint32 blog_id = 1;\n  Author author = 2;\n}\n```\n5. Has-many relationships (collections) are identified by a repeated and nested fields. For example, a Blog has many Posts\n```\nmessage Blog {\n  uint32 blog_id = 1;\n  repeated Post posts = 2;\n}\n```\n6. If we expect the SQL query to map to multiple responses, the rpc must be server-streaming. For example, if we query for many Blogs\n```\n  rpc SelectBlogs (BlogRequest) returns (stream BlogResponse) {}\n```\n7. At least one primitive, timestamp.Timestamp or enum field must be present in a message. \n8. For time definitions, use Timestamp from \"google/protobuf/timestamp.proto\"\n9. When using Timestamp from google/protobuf/timestamp.proto as input, you must use the helper functions \"time\", \"date\", and \"timestamp\" to correctly generate input for your column type. For example, \n```\nINSERT INTO logins\nVALUES (\n {{ .Id }},\n {{ .CreatedOn | date | squote }}, --column type: date\n {{ .LastLogin | timestamp | squote }}, --column type: timestamp\n {{ .LoginTime | time | squote }} --column type: time\n);\n{{ end }}\n```\n```\nmessage InsertLoginRequest {\n  uint32 blog_id = 1;\n  google.protobuf.Timestamp created_on = 2;\n  google.protobuf.Timestamp last_login = 3;\n  google.protobuf.Timestamp login_time = 4;\n}\n```\n\n## Important Notes \n1. protoc-gen-map automatically removes any duplicate rows returned by your query. If this is not a desired outcome, you should include a uniquely identifiable columns in your query and the corresponding fields in your message.\n\n2. Data mapping blueprint is generated with the first query request. Successive requests should be consistent. For example, column names should not change depending on the request message.\n\n3. Queries that do not expect any returned records (insert, update, create, delete operations) must satisfy one of the following criteria. Note that queries may fail if at least one is not satisfied. \n - RPC mame name must begin with keywords \"insert\", \"update\", \"create\", or \"delete\"(case not sensitive)\n - Response name must begin with \"empty\",\"nil\" or \"null\"(case not sensitive)\n \n Example\n ```\n  rpc InsertBlog (InsertBlogRequest) returns (EmptyResponse) {}\n ```\n### Roadmap\n\n| Goal | Status | Label |\n| :--- | --- | --- | \n| Proto Enum Support | `ready` | `enhancement` |\n| Allow developer to specify callback methods | `ready` | `enhancement` |\n| Implement Caching  | `ready` | `enhancement` |\n| Add parameterized query support | `ready` | `enhancement` |\n| Performance improvements around go reflection | `in progress` | `enhancement` |\n| Reduce ammount of generated code | `in progress` | `enhancement` |\n\n### License\nApache License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjackskj%2Fprotoc-gen-map","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjackskj%2Fprotoc-gen-map","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjackskj%2Fprotoc-gen-map/lists"}