{"id":13879122,"url":"https://github.com/discourse/mini_sql","last_synced_at":"2025-05-15T05:06:13.118Z","repository":{"id":33875202,"uuid":"137448279","full_name":"discourse/mini_sql","owner":"discourse","description":"a minimal, fast, safe sql executor","archived":false,"fork":false,"pushed_at":"2024-09-17T21:41:26.000Z","size":225,"stargazers_count":399,"open_issues_count":1,"forks_count":17,"subscribers_count":27,"default_branch":"main","last_synced_at":"2025-05-13T14:59:16.267Z","etag":null,"topics":["rubygem"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/discourse.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-15T06:15:13.000Z","updated_at":"2025-04-30T18:13:40.000Z","dependencies_parsed_at":"2024-06-18T20:12:37.784Z","dependency_job_id":"e9619a43-18c9-45bb-aefc-4e51eddc946c","html_url":"https://github.com/discourse/mini_sql","commit_stats":{"total_commits":126,"total_committers":15,"mean_commits":8.4,"dds":"0.40476190476190477","last_synced_commit":"7e0d428f2a4ddfc1d313368b7bc2b7d393f0139f"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/discourse%2Fmini_sql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/discourse%2Fmini_sql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/discourse%2Fmini_sql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/discourse%2Fmini_sql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/discourse","download_url":"https://codeload.github.com/discourse/mini_sql/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254276447,"owners_count":22043867,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["rubygem"],"created_at":"2024-08-06T08:02:10.542Z","updated_at":"2025-05-15T05:06:09.023Z","avatar_url":"https://github.com/discourse.png","language":"Ruby","readme":"# MiniSql\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'mini_sql'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install mini_sql\n\n## Usage\n\nMiniSql is a very simple, safe and fast SQL executor and mapper for PG and Sqlite.\n\n```ruby\npg_conn = PG.connect(db_name: 'my_db')\nconn = MiniSql::Connection.get(pg_conn)\n\nputs conn.exec('update table set column = 1 where id in (1,2)')\n# returns 2 if 2 rows changed\n\nconn.query(\"select 1 id, 'bob' name\").each do |user|\n  puts user.name # bob\n  puts user.id # 1\nend\n\n# extend result objects with additional method\nmodule ProductDecorator\n  def amount_price\n    price * quantity\n  end\nend\n\nconn.query_decorator(ProductDecorator, \"select 20 price, 3 quantity\").each do |user|\n  puts user.amount_price # 60\nend\n\np conn.query_single('select 1 union select 2')\n# [1,2]\n\np conn.query_hash('select 1 as a, 2 as b union select 3, 4')\n# [{\"a\" =\u003e 1, \"b\"=\u003e 1},{\"a\" =\u003e 3, \"b\" =\u003e 4}\n \np conn.query_array(\"select 1 as a, '2' as b union select 3, 'e'\")\n# [[1, '2'], [3, 'e']]\n \np conn.query_array(\"select 1 as a, '2' as b union select 3, 'e'\").to_h\n# {1 =\u003e '2', 3 =\u003e 'e'}\n```\n\n## Auto Encode Arrays (only PostgreSQL)\n```ruby\npg_conn = PG.connect(db_name: 'my_db')\nconn = MiniSql::Connection.get(pg_conn, auto_encode_arrays: true)\n\n# select * from table where id = ANY('{1,2,3}')\nconn.query(\"select * from table where id = ANY(?)\", [1, 2, 3])\n```\n\n## The query builder\n\nYou can use the simple query builder interface to compose queries.\n\n```ruby\nbuilder = conn.build(\"select * from topics /*where*/ /*limit*/\")\n\nbuilder.where('created_at \u003e ?', Time.now - 400)\n\nif look_for_something\n  builder.where(\"title = :title\", title: 'something')\nend\n\nbuilder.limit(20)\n\nbuilder.query.each do |t|\n  puts t.id\n  puts t.title\nend\n```\n\nThe same builder's statement may occur multiple times.\n\n```ruby\nbuilder = conn.build('(/*select*/ from books) union (/*select*/ from movies)').select('title').query\n\n# =\u003e (SELECT title from books) union (SELECT title from movies)\n```\n\nThe builder predefined next _SQL Literals_\n\n| Method      | SQL Literal  |\n|-------------|--------------|\n| `select`    | `/*select*/` |\n| `count`     | `/*select*/` |\n| `where`     | `/*where*/`  |\n| `where_or`  | `/*where*/`  |\n| `where2`    | `/*where2*/` |\n| `where2_or` | `/*where2*/` |\n| `join`      | `/*join*/`   |\n| `left_join` | `/*left_join*/` |\n| `group_by`  | `/*group_by*/` |\n| `order_by`  | `/*order_by*/` |\n| `limit`     | `/*limit*/`  |\n| `offset`    | `/*offset*/` |\n| `set`       | `/*set*/`    |\n\n### Custom SQL Literals\nUse `sql_literal` to inject SQL into Builder from `String`, `Builder`, `ActiveRecord::Relation`, or any object that implements `to_sql` method.\n\n```ruby\nuser_builder = conn\n  .build(\"select date_trunc('day', created_at) day, count(*) from user_topics /*where*/\")\n  .where('type = ?', input_type)\n  .group_by(\"date_trunc('day', created_at)\")\n\nguest_relation = GuestTopic\n  .select(\"date_trunc('day', created_at) day, count(*)\")\n  .where(state: input_state)\n  .group_by(\"date_trunc('day', created_at)\")\n\nconn\n  .build(\u003c\u003c~SQL)\n     with as (/*user*/) u, (/*guest*/) as g\n     select COALESCE(g.day, u.day), g.count, u.count\n     from u\n     /*custom_join*/\n  SQL\n  .sql_literal(user: user_builder, guest: guest_relation) # Builder and ActiveRecord::Relation\n  .sql_literal(custom_join: \"#{input_cond ? 'FULL' : 'LEFT'} JOIN g on g.day = u.day\") # or String\n  .query\n```\n\n## Is it fast?\nYes, it is very fast. See benchmarks in [the bench directory](https://github.com/discourse/mini_sql/tree/master/bench).\n\n**Comparison mini_sql methods**\n```\nquery_array     1351.6 i/s\n      query      963.8 i/s - 1.40x  slower\n query_hash      787.4 i/s - 1.72x  slower\n\nquery_single('select id from topics limit 1000')             2368.9 i/s\n query_array('select id from topics limit 1000').flatten     1350.1 i/s - 1.75x  slower\n```\n\nAs a rule it will outperform similar naive PG code while remaining safe.\n\n```ruby\npg_conn = PG.connect(db_name: 'my_db')\n\n# this is slower, and less safe\nresult = pg_conn.async_exec('select * from table')\nresult.each do |r|\n  name = r['name']\nend\n# ideally you want to remember to run r.clear here\n\n# this is faster and safer\nconn = MiniSql::Connection.get(pg_conn)\nr = conn.query('select * from table')\n\nr.each do |row|\n  name = row.name\nend\n```\n\n## Safety\n\nIn PG gem version 1.0 and below you should be careful to clear results. If you do not you risk memory bloat.\nSee: [Sam's blog post](https://samsaffron.com/archive/2018/06/13/ruby-x27-s-external-malloc-problem).\n\nMiniSql is careful to always clear results as soon as possible.\n\n## Timestamp decoding\n\nMiniSql's default type mapper prefers treating `timestamp without time zone` columns as utc. This is done to ensure widest amount of compatability and is a departure from the default in the PG 1.0 gem. If you wish to amend behavior feel free to pass in a custom type_map.\n\n## Custom type maps\n\nWhen using Postgres, native type mapping implementation is used. This is roughly\nimplemented as:\n\n```ruby\ntype_map ||= PG::BasicTypeMapForResults.new(conn)\n# additional specific decoders\n```\n\nThe type mapper instansitated once on-demand at boot and reused by all mini_sql connections.\n\nInitializing the basic type map for Postgres can be a costly operation. You may\nwish to amend the type mapper so for example you only return strings:\n\n```\n# maybe you do not want Integer\np cnn.query(\"select a 1\").first.a\n\"1\"\n```\n\nTo specify a different type mapper for your results use:\n\n```\nMiniSql::Connections.get(pg_connection, type_map: custom_type_map)\n```\n\nIn the case of Rails you can opt to use the type mapper Rails uses with:\n\n```\npg_cnn = ActiveRecord::Base.connection.raw_connection\nmini_sql_cnn = MiniSql::Connection.get(pg_cnn, type_map: pg_cnn.type_map_for_results)\n```\n\nNote the type mapper for Rails may miss some of the mapping MiniSql ships with such as `IPAddr`, MiniSql is also careful to use the very efficient TimestampUtc decoders where available.\n\n## Streaming support\n\nIn some exceptional cases you may want to stream results directly from the database. This enables selection of 100s of thousands of rows with limited memory impact.\n\nTwo interfaces exists for this:\n\n`query_each` : which can be used to get materialized objects  \n`query_each_hash` : which can be used to iterate through Hash objects\n\nUsage:\n\n```ruby\nmini_sql_cnn.query_each(\"SELECT * FROM tons_of_cows limit :limit\", limit: 1_000_000) do |row|\n  puts row.cow_name\n  puts row.cow_age\nend\n\nmini_sql_cnn.query_each_hash(\"SELECT * FROM one_million_cows\") do |row|\n  puts row[\"cow_name\"]\n  puts row[\"cow_age\"]\nend\n```\n\nNote, in Postgres streaming is going to be slower than non-streaming options due to internal implementation in the pq gem, each row gets a full result object and additional bookkeeping is needed. Only use it if you need to optimize memory usage.\n\nStreaming support is only implemented in the postgres backend at the moment, PRs welcome to add to other backends.\n\n## Prepared Statements\nSee [benchmark mini_sql](https://github.com/discourse/mini_sql/tree/master/bench/prepared_perf.rb)\n[benchmark mini_sql vs rails](https://github.com/discourse/mini_sql/tree/master/bench/bilder_perf.rb).\n\n```ruby\nconn.prepared.query(\"select * from table where id = ?\", id: 10)\n```\n\n### Prepared Statement Bloat in PostgreSQL\nBy default prepared cache size is __500__ queries per connection. Use prepared queries only for frequent queries.\n\nThe following code will create 100 prepared statements in PostgreSQL for a single database connection:\n\n```ruby\n100.times do |i|\n  ids = (1..i).to_a\n  conn.prepared.query(\"SELECT * FROM table WHERE id IN (?)\", ids)\nend\n```\n\nThis can lead to high memory usage and performance issues due to the overhead of maintaining numerous prepared statements.\n\nTo improve performance, you can enable `auto_encode_arrays: true`. With this option, only one prepared statement is created, regardless of the size of ids:\n\n```ruby\npg_conn = PG.connect(db_name: 'my_db')\nconn = MiniSql::Connection.get(pg_conn, auto_encode_arrays: true)\n100.times do |i|\n  ids = (1..i).to_a\n  conn.prepared.query(\"select * from table where id = ANY (?)\", ids)\nend\n```\n\n## Active Record Postgres\n\nWhen using alongside ActiveRecord, passing in the ActiveRecord connection rather than the raw Postgres connection will allow mini_sql to lock the connection, thereby preventing concurrent use in other threads.\n\n```ruby\nar_conn = ActiveRecord::Base.connection\nconn = MiniSql::Connection.get(ar_conn)\n\nconn.query(\"select * from topics\")\n```\n\n## I want more features!\n\nMiniSql is designed to be very minimal. Even though the query builder and type materializer give you a lot of mileage, it is not intended to be a fully fledged ORM. If you are looking for an ORM I recommend investigating ActiveRecord or Sequel which provide significantly more features.\n\n## Development\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n### Local testing\n```bash\n  docker run --name mini-sql-mysql --rm -it -p 33306:3306 -e MYSQL_DATABASE=test_mini_sql -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -d mysql:5.7\n  export MINI_SQL_MYSQL_HOST=127.0.0.1\n  export MINI_SQL_MYSQL_PORT=33306\n  \n  docker run --name mini-sql-postgres --rm -it -p 55432:5432 -e POSTGRES_DB=test_mini_sql -e POSTGRES_HOST_AUTH_METHOD=trust -d postgres # or ankane/pgvector for testing vector type decoder\n  export MINI_SQL_PG_USER=postgres\n  export MINI_SQL_PG_HOST=127.0.0.1\n  export MINI_SQL_PG_PORT=55432\n\n  sleep 10 # waiting for up databases\n\n  bundle exec rake\n\n  # end working on mini-sql\n  docker stop mini-sql-postgres mini-sql-mysql\n```\n\nSqlite tests rely on the SQLITE_STMT view existing. This is enabled by default on most systems, however some may\nopt for a leaner install. See: https://bugs.archlinux.org/task/70072. You may have to recompile sqlite on such systems.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/discourse/mini_sql. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n\n## Code of Conduct\n\nEveryone interacting in the MiniSql project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/discourse/mini_sql/blob/master/CODE_OF_CONDUCT.md).\n","funding_links":[],"categories":["Ruby"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiscourse%2Fmini_sql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdiscourse%2Fmini_sql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiscourse%2Fmini_sql/lists"}