{"id":13413718,"url":"https://github.com/a8m/rql","last_synced_at":"2025-04-04T14:07:06.030Z","repository":{"id":33862551,"uuid":"136219460","full_name":"a8m/rql","owner":"a8m","description":"Resource Query Language for REST","archived":false,"fork":false,"pushed_at":"2024-07-25T08:15:57.000Z","size":225,"stargazers_count":351,"open_issues_count":16,"forks_count":42,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-28T13:06:45.456Z","etag":null,"topics":["go","orm","rest-api","rql","sql","web-development"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/a8m.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-05T18:37:29.000Z","updated_at":"2025-03-25T19:45:03.000Z","dependencies_parsed_at":"2024-12-14T05:04:40.449Z","dependency_job_id":"8289c348-3cf3-4dd6-9e98-3eb9cdf9c387","html_url":"https://github.com/a8m/rql","commit_stats":{"total_commits":54,"total_committers":6,"mean_commits":9.0,"dds":0.09259259259259256,"last_synced_commit":"cd8b893ef75d6de59da274fbaa2f0fca5a4ffd8f"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/a8m%2Frql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/a8m%2Frql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/a8m%2Frql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/a8m%2Frql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/a8m","download_url":"https://codeload.github.com/a8m/rql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247190218,"owners_count":20898699,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","orm","rest-api","rql","sql","web-development"],"created_at":"2024-07-30T20:01:47.320Z","updated_at":"2025-04-04T14:07:06.010Z","avatar_url":"https://github.com/a8m.png","language":"Go","readme":"\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"assets/logo.png\" height=\"100\" border=\"0\" alt=\"RQL\"\u003e\n\t\u003cbr/\u003e\n\t\u003ca href=\"https://godoc.org/github.com/a8m/rql\"\u003e\n\t\t\u003cimg src=\"https://img.shields.io/badge/api-reference-blue.svg?style=flat-square\" alt=\"GoDoc\"\u003e\n\t\u003c/a\u003e\n\t\u003ca href=\"LICENSE\"\u003e\n\t\t\u003cimg src=\"https://img.shields.io/badge/license-MIT-blue.svg?style=flat-square\" alt=\"LICENSE\"\u003e\n\t\u003c/a\u003e\n\t\u003ca href=\"https://app.circleci.com/pipelines/github/a8m/rql?branch=master\"\u003e\n\t\t\u003cimg src=\"https://img.shields.io/circleci/build/github/a8m/rql?style=flat-square\" alt=\"Build Status\"\u003e\n\t\u003c/a\u003e\n\u003c/p\u003e\n\nRQL is a resource query language for REST. It provides a simple and light-weight API for adding dynamic querying capabilities to web-applications that use SQL-based database. It functions as the connector between the HTTP handler and the DB engine, and manages all validations and translations for user inputs.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/diagram.png\" alt=\"rql diagram\"\u003e\n\u003c/p\u003e\n\n## Motivation\nIn the last several years I have found myself working on different web applications in Go, some of them were small and some of them were big with a lot of entities and relations. In all cases I never found a simple and standard API to query my resources.\n\nWhat do I mean by query? Let's say our application has a table of `orders`, and we want our users to be able to search and __filter by dynamic parameters__. For example: _select all orders from today with price greater than 100_.  \nIn order to achieve that I used to pass these parameters in the query string like this: `created_at_gt=X\u0026price_gt=100`.  \nBut sometimes it became complicated when I needed to apply a disjunction between two conditions. For example, when I wanted to _select all orders that canceled or created last week and still didn't ship_. And in SQL syntax:\n```sql\nSELECT * FROM ORDER WHERE canceled = 1 OR (created_at \u003c X AND created_at \u003e Y AND shipped = 0)\n```  \nI was familiar with the MongoDB syntax and I felt that it was simple and robust enough to achieve my goals, and decided to use it as the query language for this project. I wanted it to be project agnostic in the sense of not relying on anything that related to some specific application or resource. Therefore, in order to embed rql in a new project, a user just needs to import the package and add the desired tags to his struct definition. Follow the [Getting Started](#Getting_Started) section to learn more.\n\n## Getting Started\nrql uses a subset of MongoDB query syntax. If you are familiar with the MongoDB syntax, it will be easy for you to start. Although, it's pretty simple and easy to learn.    \nIn order to embed rql you simply need to add the tags you want (`filter` or `sort`) to your struct definition, and rql will manage all validations for you and return an informative error for the end user if the query doesn't follow the schema.\nHere's a short example of how to start using rql quickly, or you can go to [API](#API) for more expanded documentation.\n```go\n// An example of an HTTP handler that uses gorm, and accepts user query in either the body\n// or the URL query string.\npackage main\n\nvar (\n\tdb *gorm.DB\n\t// QueryParam is the name of the query string key.\n\tQueryParam = \"query\"\n\t// MustNewParser panics if the configuration is invalid.\n\tQueryParser = rql.MustNewParser(rql.Config{\n\t\tModel:    User{},\n\t\tFieldSep: \".\",\n\t})\n)\n\n// User is the model in gorm's terminology.\ntype User struct {\n\tID          uint      `gorm:\"primary_key\" rql:\"filter,sort\"`\n\tAdmin       bool      `rql:\"filter\"`\n\tName        string    `rql:\"filter\"`\n\tAddressName string    `rql:\"filter\"`\n\tCreatedAt   time.Time `rql:\"filter,sort\"`\n}\n\n\n// GetUsers is an http.Handler that accepts a db query in either the body or the query string.\nfunc GetUsers(w http.ResponseWriter, r *http.Request) {\n\tvar users []User\n\tp, err := getDBQuery(r)\n\tif err != nil {\n\t\tio.WriteString(w, err.Error())\n\t\tw.WriteHeader(http.StatusBadRequest)\n\t\treturn\n\t}\n\terr = db.Where(p.FilterExp, p.FilterArgs).\n\t\tOffset(p.Offset).\n\t\tLimit(p.Limit).\n\t\tOrder(p.Sort).\n\t\tFind(\u0026users).Error\n\tif err != nil {\n\t\tw.WriteHeader(http.StatusInternalServerError)\n\t\treturn\n\t}\n\tif err := json.NewEncoder(w).Encode(users); err != nil {\n\t\tw.WriteHeader(http.StatusInternalServerError)\n\t\treturn\n\t}\n\tw.Header().Set(\"Content-Type\", \"application/json\")\n}\n\n// getDBQuery extract the query blob from either the body or the query string\n// and execute the parser.\nfunc getDBQuery(r *http.Request) (*rql.Params, error) {\n\tvar (\n\t\tb   []byte\n\t\terr error\n\t)\n\tif v := r.URL.Query().Get(QueryParam); v != \"\" {\n\t\tb, err = base64.StdEncoding.DecodeString(v)\n\t} else {\n\t\tb, err = ioutil.ReadAll(io.LimitReader(r.Body, 1\u003c\u003c12))\n\t}\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn QueryParser.Parse(b)\n}\n```\nGo to [examples/simple](examples/simple.go) to see the full working example.\n\n\n## API\nIn order to start using rql, you need to configure your parser. Let's go over a basic example of how to do this. For more details and updated documentation, please checkout the [godoc](https://godoc.org/github.com/a8m/rql/#Config).  \nThere are two options to build a parser, `rql.New(rql.Config)`, and `rql.MustNew(rql.Config)`. The only difference between the two is that `rql.New` returns an error if the configuration is invalid, and `rql.MustNew` panics.\n```go\n// we use rql.MustPanic because we don't want to deal with error handling in top level declarations.\nvar Parser = rql.MustNew(rql.Config{\n\t// User if the resource we want to query.\n\tModel: User{},\n\t// Since we work with gorm, we want to use its column-function, and not rql default.\n\t// although, they are pretty the same.\n\tColumnFn: gorm.ToDBName,\n\t// Use your own custom logger. This logger is used only in the building stage.\n\tLog: logrus.Printf,\n\t// Default limit returned by the `Parse` function if no limit provided by the user.\n\tDefaultLimit: 100,\n\t// Accept only requests that pass limit value that is greater than or equal to 200.\n\tLimitMaxValue: 200,\n})\n```\nrql uses reflection in the build process to detect the type of each field, and create a set of validation rules for each one. If one of the validation rules fails or rql encounters an unknown field, it returns an informative error to the user. Don't worry about the usage of reflection, it happens only once when you build the parser.\nLet's go over the validation rules:\n1. `int` (8,16,32,64), `sql.NullInt6` - Round number\n2. `uint` (8,16,32,64), `uintptr` - Round number and greater than or equal to 0\n3. `float` (32,64), sql.NullFloat64: - Number\n4. `bool`, `sql.NullBool` - Boolean\n5. `string`, `sql.NullString` - String\n6. `time.Time`, and other types that convertible to `time.Time` - The default layout is time.RFC3339 format (JS format), and parsable to `time.Time`.\n   It's possible to override the `time.Time` layout format with custom one. You can either use one of the standard layouts in the `time` package, or use a custom one. For example:\n   ```go\n   type User struct {\n\t\tT1 time.Time `rql:\"filter\"`                         // time.RFC3339\n\t\tT2 time.Time `rql:\"filter,layout=UnixDate\"`         // time.UnixDate\n\t\tT3 time.Time `rql:\"filter,layout=2006-01-02 15:04\"` // 2006-01-02 15:04 (custom)\n   }\n   ```  \n\nNote that all rules are applied to pointers as well. It means, if you have a field `Name *string` in your struct, we still use the string validation rule for it.\n\n### User API\nWe consider developers as the users of this API (usually FE developers). Let's go over the JSON API we export for resources.  \nThe top-level query accepts JSON with 4 fields: `offset`, `limit`, `filter` and `sort`. All of them are optional.\n\n#### `offset` and `limit`\nThese two fields are useful for paging and they are equivalent to `OFFSET` and `LIMIT` in a standard SQL syntax.\n- `offset` must be greater than or equal to 0 and its default value is 0\n- `limit` must be greater than 0 and less than or equal to the configured `LimitMaxValue`.\n   The default value for `LimitMaxValue` is 100\n\n#### `sort`\nSort accepts a slice of strings (`[]string`) that is translated to the SQL `ORDER BY` clause. The given slice must contain only columns that are sortable (have tag `rql:\"sort\"`). The default order for column is ascending order in SQL, but you can control it with an optional prefix: `+` or `-`. `+` means ascending order, and `-` means descending order. Let's see a short example:\n```\nFor input - [\"address.name\", \"-address.zip.code\", \"+age\"]\nResult is - address_name, address_zip_code DESC, age ASC\n```\n\n#### `select`\nSelect accepts a slice of strings (`[]string`) that is joined with comma (\",\") to the SQL `SELECT` clause.\n```\nFor input - [\"name\", \"age\"]\nResult is - \"name, age\"\n```\n\n#### `filter`\nFilter is the one who is translated to the SQL `WHERE` clause. This object that contains `filterable` fields or the disjunction (`$or`) operator. Each field in the object represents a condition in the `WHERE` clause. It contains a specific value that matched the type of the field or an object of predicates. Let's go over them:\n- Field follows the format: `field: \u003cvalue\u003e`, means the predicate that will be used is `=`. For example:\n  ```\n  For input:\n  {\n    \"admin\": true\n  }\n  \n  Result is: admin = ?\n  ```\n  You can see that RQL uses placeholders in the generated `WHERE` statement. Follow the [examples](#examples) section\n  to see how to use it properly. \n- If the field follows the format: `field: { \u003cpredicate\u003e: \u003cvalue\u003e, ...}`, For example:\n  ```\n  For input:\n  {\n    \"age\": {\n      \"$gt\": 20,\n      \"$lt\": 30\n    }\n  }\n  \n  Result is: age \u003e ? AND age \u003c ?\n  ```\n  It means that the logical `AND` operator used between the two predicates.\n  Scroll below to see the full list of the supported predicates.\n- `$or` is a field that represents the logical `OR` operator, and can be in any level of the query. Its type need to be an\n  array of condition objects and the result of it is the disjunction between them. For example:\n  ```\n  For input:\n  {\n    \"$or\": [\n      { \"city\": \"TLV\" },\n      { \"zip\": { \"$gte\": 49800, \"$lte\": 57080 } }\n    ]\n  }\n  \n  Result is: city = ? OR (zip \u003e= ? AND zip \u003c= ?)\n  ```\nTo simplify that, the rule is `AND` for objects and `OR` for arrays. Let's go over the list of supported predicates and then we'll show a few examples.\n\n##### Predicates\n- `$eq` and `$neq` - can be used on all types\n- `$gt`, `$lt`, `$gte` and `$lte` - can be used on numbers, strings, and timestamp\n- `$like` - can be used only on type string\n\nIf a user tries to apply an unsupported predicate on a field it will get an informative error. For example:\n```\nFor input:\n{\n  \"age\": {\n    \"$like\": \"%0\"\n  }\n}\n\nResult is: can not apply op \"$like\" on field \"age\"\n```\n\n## Examples\nAssume this is the parser for all examples.\n```go\n\nvar QueryParser = rql.MustNewParser(rql.Config{\n\tModel:    \tUser{},\n\tFieldSep: \t\".\",\n\tLimitMaxValue:\t25,\n})\n\t\ntype User struct {\n\tID          uint      `gorm:\"primary_key\" rql:\"filter,sort\"`\n\tAdmin       bool      `rql:\"filter\"`\n\tName        string    `rql:\"filter\"`\n\tAddress     string    `rql:\"filter\"`\n\tCreatedAt   time.Time `rql:\"filter,sort\"`\n}\n```\n#### Simple Example\n```go\nparams, err := QueryParser.Parse([]byte(`{\n  \"limit\": 25,\n  \"offset\": 0,\n  \"filter\": {\n    \"admin\": false\n  }\n  \"sort\": [\"+name\"]\n}`))\nmust(err, \"parse should pass\")\nfmt.Println(params.Limit)\t// 25\nfmt.Println(params.Offset)\t// 0\nfmt.Println(params.Sort)\t// \"name ASC\"\nfmt.Println(params.FilterExp)\t// \"name = ?\"\nfmt.Println(params.FilterArgs)\t// [true]\n```\n\nIn this case you've a valid generated `rql.Param` object and you can pass its to your favorite package connector.\n\n```go\nvar users []*User\n\n// entgo.io (A type-safe entity framework)\nusers, err = client.User.Query().\n    Where(func(s *sql.Selector) {\n        s.Where(sql.ExprP(p.FilterExp, p.FilterArgs...))\n    }).\n    Limit(p.Limit).\n    Offset(p.Offset).\n    All(ctx)\nmust(err, \"failed to query ent\")\n\n// gorm\nerr = db.Where(p.FilterExp, p.FilterArgs).\n\tOffset(p.Offset).\n\tLimit(p.Limit).\n\tOrder(p.Sort).\n\tFind(\u0026users).Error\nmust(err, \"failed to query gorm\")\n\n// xorm\nerr = engine.Where(p.FilterExp, p.FilterArgs...).\n\tLimit(p.Limit, p.Offset).\n\tOrderBy(p.Sort).\n\tFind(\u0026users)\nmust(err, \"failed to query xorm\")\n\n// go-pg/pg\nerr = db.Model(\u0026users).\n\tWhere(p.FilterExp, p.FilterArgs).\n\tOffset(p.Offest).\n\tLimit(p.Limit).\n\tOrder(p.Sort).\n\tSelect()\nmust(err, \"failed to query pg/orm\")\n\n// Have more example? feel free to add.\n```\n\n#### Medium Example\n```go\nparams, err := QueryParser.Parse([]byte(`{\n  \"limit\": 25,\n  \"filter\": {\n    \"admin\": false,\n    \"created_at\": {\n      \"$gt\": \"2018-01-01T16:00:00.000Z\",\n      \"$lt\": \"2018-04-01T16:00:00.000Z\"\n    }\n    \"$or\": [\n      { \"address\": \"TLV\" },\n      { \"address\": \"NYC\" }\n    ]\n  }\n  \"sort\": [\"-created_at\"]\n}`))\nmust(err, \"parse should pass\")\nfmt.Println(params.Limit)\t// 25\nfmt.Println(params.Offset)\t// 0\nfmt.Println(params.Sort)\t// \"created_at DESC\"\nfmt.Println(params.FilterExp)\t// \"admin = ? AND created_at \u003e ? AND created_at \u003c ? AND (address = ? OR address = ?)\"\nfmt.Println(params.FilterArgs)\t// [true, Time(2018-01-01T16:00:00.000Z), Time(2018-04-01T16:00:00.000Z), \"TLV\", \"NYC\"]\n```\n\n\n## Future Plans and Contributions\nIf you want to help with the development of this package, here is a list of options things I want to add\n- [ ] JS library for query building\n- [ ] Option to ignore validation with specific tag\n- [ ] Add `$not` and `$nor` operators\n- [ ] Automatically (or by config) filter and sort `gorm.Model` fields\n- [ ] benchcmp for PRs\n- [ ] Support MongoDB. Output need to be a bison object. here's a [usage example](https://gist.github.com/congjf/8035830)\n- [ ] Right now rql assume all fields are flatted in the db, even for nested fields.\n  For example, if you have a struct like this:\n  ```go\n  type User struct {\n      Address struct {\n          Name string `rql:\"filter\"`\n      }\n  }\n  ```\n  rql assumes that the `address_name` field exists in the table. Sometimes it's not the case and `address` exists in\n  a different table. Therefore, I want to add the `table=` option for fields, and support nested queries.\n- [ ] Code generation version - low priority\n\n## Performance and Reliability\nThe performance of RQL looks pretty good, but there is always a room for improvement. Here's the current bench result:\n\n|      __Test__       | __Time/op__    | __B/op__   | __allocs/op__  |\n|---------------------|----------------|------------|----------------|\n| Small               |    1809        |   960      |   19           |\n| Medium              |    6030        |   3100     |   64           |\n| Large               |    14726       |   7625     |   148          |\n\nI ran fuzzy testing using `go-fuzz` and I didn't see any crashes. You are welcome to run by yourself and find potential failures. \n\n## LICENSE\nI am providing code in the repository to you under MIT license. Because this is my personal repository, the license you receive to my code is from me and not my employer (Facebook)\n","funding_links":[],"categories":["Misc","开源类库","查询语言","Query Language","Open source library","Go语言包管理","Go","Relational Databases"],"sub_categories":["查询语言","HTTP客户端","HTTP Clients","Query Language","查询语","Advanced Console UIs","交流"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fa8m%2Frql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fa8m%2Frql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fa8m%2Frql/lists"}