{"id":13562474,"url":"https://github.com/araddon/qlbridge","last_synced_at":"2025-05-16T05:05:54.309Z","repository":{"id":22416166,"uuid":"25753725","full_name":"araddon/qlbridge","owner":"araddon","description":"A golang expression evaluator \u0026 Library to build SQL query engine based functionality. ","archived":false,"fork":false,"pushed_at":"2023-08-29T10:59:30.000Z","size":8020,"stargazers_count":867,"open_issues_count":36,"forks_count":79,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-04-08T15:07:01.984Z","etag":null,"topics":["expressionengine","go","golang","query-engine","sql","sql-runtime"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/araddon.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-10-26T01:58:25.000Z","updated_at":"2025-04-06T08:57:57.000Z","dependencies_parsed_at":"2024-06-18T13:40:59.942Z","dependency_job_id":"20a1a5d5-4483-4509-b6fd-d3333462316a","html_url":"https://github.com/araddon/qlbridge","commit_stats":{"total_commits":874,"total_committers":19,"mean_commits":46.0,"dds":0.419908466819222,"last_synced_commit":"dbc85609ca044c8f4b7144503bf31ac79b087ed2"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/araddon%2Fqlbridge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/araddon%2Fqlbridge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/araddon%2Fqlbridge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/araddon%2Fqlbridge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/araddon","download_url":"https://codeload.github.com/araddon/qlbridge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254471061,"owners_count":22076585,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["expressionengine","go","golang","query-engine","sql","sql-runtime"],"created_at":"2024-08-01T13:01:09.052Z","updated_at":"2025-05-16T05:05:49.299Z","avatar_url":"https://github.com/araddon.png","language":"Go","readme":"QLBridge - Go SQL Runtime Engine\n=====================================================\n\nA SQL execution engine for embedded use as a library for SQL or SQL-Like functionality.\nHackable, add datasources (\"Storage\" can be rest apis, or anything), and add functions.  See usage in https://github.com/dataux/dataux\na federated Sql Engine mysql-compatible with backends (Elasticsearch, Google-Datastore, Mongo, Cassandra, Files).\n\n[![Code Coverage](https://codecov.io/gh/araddon/qlbridge/branch/master/graph/badge.svg)](https://codecov.io/gh/araddon/qlbridge)\n[![GoDoc](https://godoc.org/github.com/araddon/qlbridge?status.svg)](http://godoc.org/github.com/araddon/qlbridge)\n[![Build Status](https://travis-ci.org/araddon/qlbridge.svg?branch=master)](https://travis-ci.org/araddon/qlbridge)\n[![Go ReportCard](https://goreportcard.com/badge/araddon/qlbridge)](https://goreportcard.com/report/araddon/qlbridge)\n\n\n### QLBridge Features and Goals\n* expression engine for evaluation of single expressions\n* execution of sql queries against your data, embedable, not coupled to storage layer\n* extend VM with custom go functions, provide rich basic library of functions\n* provide example backends (csv, elasticsearch, etc)\n\n### Dialects\n* SQL [see examples](https://github.com/araddon/qlbridge/blob/master/exec/exec_test.go)\n* FilterQL (just Where clause) with more of a DSL for filter [see examples](https://github.com/araddon/qlbridge/blob/master/vm/filterqlvm_test.go#L75)\n* Simple Expressions [see examples](https://github.com/araddon/qlbridge/blob/master/vm/vm_test.go#L59)\n\n### Example of Expression Evaluation Engine\n\nThese expressions can be used stand-alone embedded usage in your app.  But, \nare the same expressions which might be columns, where, group-by clauses in SQL.\n[see example](examples/expressions/main.go)\n```go\nfunc main() {\n\n\t// Add a custom function to the VM to make available to expression language\n\texpr.FuncAdd(\"email_is_valid\", \u0026EmailIsValid{})\n\n\t// This is the evaluation context which will be the data-source\n\t// to be evaluated against the expressions.  There is a very simple\n\t// interface you can use to create your own.\n\tevalContext := datasource.NewContextSimpleNative(map[string]interface{}{\n\t\t\"int5\":     5,\n\t\t\"str5\":     \"5\",\n\t\t\"created\":  dateparse.MustParse(\"12/18/2015\"),\n\t\t\"bvalt\":    true,\n\t\t\"bvalf\":    false,\n\t\t\"user_id\":  \"abc\",\n\t\t\"urls\":     []string{\"http://google.com\", \"http://nytimes.com\"},\n\t\t\"hits\":     map[string]int64{\"google.com\": 5, \"bing.com\": 1},\n\t\t\"email\":    \"bob@bob.com\",\n\t\t\"emailbad\": \"bob\",\n\t\t\"mt\": map[string]time.Time{\n\t\t\t\"event0\": dateparse.MustParse(\"12/18/2015\"),\n\t\t\t\"event1\": dateparse.MustParse(\"12/22/2015\"),\n\t\t},\n\t})\n\n\t// Example list of expressions\n\texprs := []string{\n\t\t\"int5 == 5\",\n\t\t`6 \u003e 5`,\n\t\t`6 \u003e 5.5`,\n\t\t`(4 + 5) / 2`,\n\t\t`6 == (5 + 1)`,\n\t\t`2 * (3 + 5)`,\n\t\t`todate(\"12/12/2012\")`,\n\t\t`created \u003e \"now-1M\"`, // Date math\n\t\t`created \u003e \"now-10y\"`,\n\t\t`user_id == \"abc\"`,\n\t\t`email_is_valid(email)`,\n\t\t`email_is_valid(emailbad)`,\n\t\t`email_is_valid(\"not_an_email\")`,\n\t\t`EXISTS int5`,\n\t\t`!exists(user_id)`,\n\t\t`mt.event0 \u003e now()`, // step into child of maps\n\t\t`[\"portland\"] LIKE \"*land\"`,\n\t\t`email contains \"bob\"`,\n\t\t`email NOT contains \"bob\"`,\n\t\t`[1,2,3] contains int5`,\n\t\t`[1,2,3,5] NOT contains int5`,\n\t\t`urls contains \"http://google.com\"`,\n\t\t`split(\"chicago,portland\",\",\") LIKE \"*land\"`,\n\t\t`10 BETWEEN 1 AND 50`,\n\t\t`15.5 BETWEEN 1 AND \"55.5\"`,\n\t\t`created BETWEEN \"now-50w\" AND \"12/18/2020\"`,\n\t\t`toint(not_a_field) NOT IN (\"a\",\"b\" 4.5)`,\n\t\t`\n\t\tOR (\n\t\t\temail != \"bob@bob.com\"\n\t\t\tAND (\n\t\t\t\tNOT EXISTS not_a_field\n\t\t\t\tint5 == 5 \n\t\t\t)\n\t\t)`,\n\t}\n\n\tfor _, expression := range exprs {\n\t\t// Same ast can be re-used safely concurrently\n\t\texprAst := expr.MustParse(expression)\n\t\t// Evaluate AST in the vm\n\t\tval, _ := vm.Eval(evalContext, exprAst)\n\t\tv := val.Value()\n\t\tu.Debugf(\"Output: %-35v T:%-15T expr:  %s\", v, v, expression)\n\t}\n}\n\n// Example of a custom Function, that we are making available in the Expression VM\ntype EmailIsValid struct{}\n\nfunc (m *EmailIsValid) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error) {\n\tif len(n.Args) != 1 {\n\t\treturn nil, fmt.Errorf(\"Expected 1 arg for EmailIsValid(arg) but got %s\", n)\n\t}\n\treturn func(ctx expr.EvalContext, args []value.Value) (value.Value, bool) {\n\t\tif args[0] == nil || args[0].Err() || args[0].Nil() {\n\t\t\treturn value.BoolValueFalse, true\n\t\t}\n\t\tif _, err := mail.ParseAddress(args[0].ToString()); err == nil {\n\t\t\treturn value.BoolValueTrue, true\n\t\t}\n\n\t\treturn value.BoolValueFalse, true\n\t}, nil\n}\nfunc (m *EmailIsValid) Type() value.ValueType { return value.BoolType }\n\n\n```\n### Example SQL Runtime for Reading a Csv via Stdio, File\n\nSee example in [qlcsv](https://github.com/araddon/qlbridge/tree/master/examples/qlcsv)\nfolder for a CSV reader, parser, evaluation engine.\n\n```sh\n\n./qlcsv -sql 'select \n\t\tuser_id, email, item_count * 2, yy(reg_date) \u003e 10 \n\tFROM stdin where email_is_valid(email);' \u003c users.csv\n\n```\n```go\n\nfunc main() {\n\n\tif sqlText == \"\" {\n\t\tu.Errorf(\"You must provide a valid select query in argument:    --sql=\\\"select ...\\\"\")\n\t\treturn\n\t}\n\n\t// load all of our built-in functions\n\tbuiltins.LoadAllBuiltins()\n\n\t// Add a custom function to the VM to make available to SQL language\n\texpr.FuncAdd(\"email_is_valid\", \u0026EmailIsValid{})\n\n\t// We are registering the \"csv\" datasource, to show that\n\t// the backend/sources can be easily created/added.  This csv\n\t// reader is an example datasource that is very, very simple.\n\texit := make(chan bool)\n\tsrc, _ := datasource.NewCsvSource(\"stdin\", 0, bytes.NewReader([]byte(\"##\")), exit)\n\tschema.RegisterSourceAsSchema(\"example_csv\", src)\n\n\tdb, err := sql.Open(\"qlbridge\", \"example_csv\")\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\tdefer db.Close()\n\n\trows, err := db.Query(sqlText)\n\tif err != nil {\n\t\tu.Errorf(\"could not execute query: %v\", err)\n\t\treturn\n\t}\n\tdefer rows.Close()\n\tcols, _ := rows.Columns()\n\n\t// this is just stupid hijinx for getting pointers for unknown len columns\n\treadCols := make([]interface{}, len(cols))\n\twriteCols := make([]string, len(cols))\n\tfor i := range writeCols {\n\t\treadCols[i] = \u0026writeCols[i]\n\t}\n\tfmt.Printf(\"\\n\\nScanning through CSV: (%v)\\n\\n\", strings.Join(cols, \",\"))\n\tfor rows.Next() {\n\t\trows.Scan(readCols...)\n\t\tfmt.Println(strings.Join(writeCols, \", \"))\n\t}\n\tfmt.Println(\"\")\n}\n\n// Example of a custom Function, that we are adding into the Expression VM\n//\n//         select\n//              user_id AS theuserid, email, item_count * 2, reg_date\n//         FROM stdin\n//         WHERE email_is_valid(email)\ntype EmailIsValid struct{}\n\nfunc (m *EmailIsValid) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error) {\n\tif len(n.Args) != 1 {\n\t\treturn nil, fmt.Errorf(\"Expected 1 arg for EmailIsValid(arg) but got %s\", n)\n\t}\n\treturn func(ctx expr.EvalContext, args []value.Value) (value.Value, bool) {\n\t\tif args[0] == nil || args[0].Err() || args[0].Nil() {\n\t\t\treturn value.BoolValueFalse, true\n\t\t}\n\t\tif _, err := mail.ParseAddress(args[0].ToString()); err == nil {\n\t\t\treturn value.BoolValueTrue, true\n\t\t}\n\n\t\treturn value.BoolValueFalse, true\n\t}, nil\n}\nfunc (m *EmailIsValid) Type() value.ValueType { return value.BoolType }\n\n```\n\n[x]QL languages are making a comeback.   It is still an easy, approachable\nway of working with data.   Also, we see more and more ql's that are xql'ish but\nun-apologetically non-standard.  This matches our observation that\ndata is stored in more and more formats in more tools, services that aren't\ntraditional db's but querying that data should still be easy.  Examples\n[Influx](http://influxdb.com/docs/v0.8/api/query_language.html), \n[GitQL](https://github.com/cloudson/gitql), \n[Presto](http://prestosql.io/), \n[Hive](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select), \n[CQL](http://www.datastax.com/documentation/cql/3.1/cql/cql_intro_c.html),\n[yql](https://developer.yahoo.com/yql/),\n[ql.io](http://ql.io/), etc\n\n\nProjects that access non-sql data via [x]ql\n----------------------------------------------------\n* http://prestosql.io/\n* https://crate.io/docs/current/sql/index.html\n* http://senseidb.com/\n* http://influxdb.com/docs/v0.8/api/query_language.html\n* https://github.com/crosbymichael/dockersql\n* http://harelba.github.io/q/\n* https://github.com/dinedal/textql\n* https://github.com/cloudson/gitql\n* https://github.com/brendandburns/ksql \n\nGo Script/VM interpreters\n---------------------------------------\n* https://github.com/robpike/ivy\n* https://github.com/yuin/gopher-lua\n* https://github.com/SteelSeries/golisp\n* [Complete List](https://github.com/golang/go/wiki/Projects#virtual-machines-and-languages)\n\n","funding_links":[],"categories":["开源类库","Go","Open source library","Repositories"],"sub_categories":["解释器","Interpreter"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faraddon%2Fqlbridge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faraddon%2Fqlbridge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faraddon%2Fqlbridge/lists"}