{"id":20127549,"url":"https://github.com/amauryt/crystal-duckdb","last_synced_at":"2025-05-06T18:31:35.948Z","repository":{"id":176841807,"uuid":"393803337","full_name":"amauryt/crystal-duckdb","owner":"amauryt","description":"Crystal language bindings for DuckDB","archived":false,"fork":false,"pushed_at":"2025-03-03T10:40:54.000Z","size":113,"stargazers_count":18,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-09T15:51:41.748Z","etag":null,"topics":["crystal","duckdb"],"latest_commit_sha":null,"homepage":"","language":"Crystal","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amauryt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-07T22:11:26.000Z","updated_at":"2025-03-03T10:40:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"db130b07-c0a9-49f8-8c5b-a9723a193c80","html_url":"https://github.com/amauryt/crystal-duckdb","commit_stats":null,"previous_names":["amauryt/crystal-duckdb"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amauryt%2Fcrystal-duckdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amauryt%2Fcrystal-duckdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amauryt%2Fcrystal-duckdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amauryt%2Fcrystal-duckdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amauryt","download_url":"https://codeload.github.com/amauryt/crystal-duckdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252744497,"owners_count":21797632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crystal","duckdb"],"created_at":"2024-11-13T20:22:19.084Z","updated_at":"2025-05-06T18:31:35.913Z","avatar_url":"https://github.com/amauryt.png","language":"Crystal","funding_links":[],"categories":[],"sub_categories":[],"readme":"# crystal-duckdb\n\n[![Crystal CI](https://github.com/amauryt/crystal-duckdb/actions/workflows/crystal.yml/badge.svg)](https://github.com/amauryt/crystal-duckdb/actions/workflows/crystal.yml)\n\nCrystal bindings for [DuckDB](https://duckdb.org/), an in-process SQL OLAP database management system.\n\nCheck [crystal-db](https://github.com/crystal-lang/crystal-db) for general DB driver documentation. This shard's driver is registered under the `duckdb://` URI.\n\n## Project status\n\nThis is an implementation primarily intended to fulfill my needs for Online Analytical Processing (OLAP) using DuckDB across different languages (Crystal, R, and JS). Therefore, only a subset of the DuckDB C API is implemented, but **it should more than enough for many OLAP applications in Crystal**.\nMoreover, with the release of DuckDB v1.0, this subset of features has now reached a high level of stability.\n\nPlease note that OLAP workloads and workflows are very different from OLTP (Online Transaction Processing), especially in an embedded context. Before using DuckDB be sure to understand the differences between the two to decide which option is more apt for your use case.\n\n## DuckDB compatibility\n\nThe DuckDB engine has recently released their first stable version (v1.0), hence breaking changes in the near future are now unlikely to occur.\n\nIf you are using a non-stable version of the engine, be sure to use the correct shard version. In addition, there might be DB file *storage incompability* across different versions of the DuckDB engines before v1.0. In this case you need to export your data with the old engine and import it with the new engine; see the [export/import documentation](https://duckdb.org/docs/sql/statements/export) for more details. If supported, I suggest using the parquet format.\n\n| Shard release   | DuckDB engine    | Notes                                                 |\n| --------------- | -----------------| ------------------------------------------------------|\n| 0.2.5           |  0.9.x – 1.2.x   | Storage compability. Updated `crystal-db` to v0.13.   |\n| 0.2.4           |  0.9.x – 0.10.x  | Storage incompability. Updated `crystal-db` to v0.12. |\n| 0.2.3           |  0.6.0 – 0.8.x   | Added support for hugeint. Changed varchar C API.     |\n| 0.2.2           |  0.5.1 – 0.6.0   | Added support for interval datatype and configuration.|\n| 0.2.1           |  0.3.4 – 0.5.1   | Storage incompability. Updated `crystal-db` to v0.11. |\n| 0.2.0           |  0.2.9 – 0.3.2   | Storage incompability.                                |\n| 0.1.x           |  0.2.8           |                                                       |\n\n\n## Prerequisites\n\nYou must have a **compatible DuckDB engine** installed and available as a dynamic library within your app.\n\nFor MacOS (and many Linux distributions) the simplest way to install DuckDB is via homebrew:\n\n```\nbrew install duckdb\n```\n\n## Installation\n\n1. Add the dependency to your `shard.yml`:\n\n   ```yaml\n   dependencies:\n     duckdb:\n       github: amauryt/crystal-duckdb\n       version: ~\u003e 0.2.5\n   ```\n\n2. Run `shards install`\n\n## Usage\n\nFor most use cases it is better to use the connection directly instead of a DB pool.\n\n```crystal\nrequire \"duckdb\"\n\nDB.connect \"duckdb://./data.db\" do |cnn|\n  cnn.exec \"create table contacts (name varchar, age integer)\"\n  cnn.exec \"insert into contacts values (?, ?)\", \"John Doe\", 30\n\n  args = [] of DB::Any\n  args \u003c\u003c \"Sarah\"\n  args \u003c\u003c 33\n  cnn.exec \"insert into contacts values (?, ?)\", args: args\n\n  puts \"max age:\"\n  puts cnn.scalar \"select max(age) from contacts\" # =\u003e 33\n\n  puts \"contacts:\"\n  cnn.query \"select name, age from contacts order by age desc\" do |rs|\n    puts \"#{rs.column_name(0)} (#{rs.column_name(1)})\"\n    # =\u003e name (age)\n    rs.each do\n      puts \"#{rs.read(String)} (#{rs.read(Int32)})\"\n      # =\u003e Sarah (33)\n      # =\u003e John Doe (30)\n    end\n  end\nend\n```\n\nFor an in-memory database you can use `DuckDB::IN_MEMORY`. \n\n```crystal\nrequire \"duckdb\"\n\nDB.connect DuckDB::IN_MEMORY do |cnn|\n  cnn.scalar(\"select 'hello world'\") # =\u003e \"hello world\"\nend\n```\n\n### Configuration\n\nYou can also modify the DuckDB configuration upon opening a database using URI params (together with the URI params available from `crystal-db`).\n\nFor more details refer to the [DuckDB configuration documentation](https://duckdb.org/docs/sql/configuration).\n\nBeware that an invalid configuration will raise a `DuckDB::Exception`.\n\n```crystal\nrequire \"duckdb\"\n\n# Connect to a database in read-only mode (file must already exists) and with NULL values ordered last by default\nDB.connect \"duckdb://./data.db?access_mode=read_only\u0026default_null_order=nulls_last\" do |cnn|\n  puts cnn.scalar \"SELECT current_setting('access_mode')\"  # =\u003e read_only\n  puts cnn.scalar \"SELECT current_setting('default_null_order')\"  # =\u003e nulls_last\nend\n```\n\nTo configure an in-memory database:\n\n```crystal\nrequire \"duckdb\"\nDB.connect \"#{DuckDB::IN_MEMORY}?default_null_order=nulls_last\" do |cnn|\n  puts cnn.scalar \"SELECT current_setting('default_null_order')\"  # =\u003e nulls_last\nend\n```\n\n### Appender\n\nTo efficiently load bulk data into a table use the appender instead of insert statements.\nThe [Appender](https://duckdb.org/docs/api/c/appender) is tied to a connection, and will use the transaction context of that connection when appending.\nAn Appender always appends to a single table in the database.\n\n```crystal\nrequire \"duckdb\"\n\nrecords = [\n  {name: \"Alice\", age: 20, is_active: true},\n  {name: \"Bob\", age: 30, is_active: false}\n  {name: \"Charles\", age: 25, is_active: nil}\n]\n\nDB.connect DuckDB::IN_MEMORY do |cnn|\n  cnn.exec \"create table contacts (name varchar, age integer, is_active boolean)\"\n\n  cnn.appender(\"contacts\") do |appender|\n    records.each do |record|\n      appender.row do |row|\n        row \u003c\u003c record.name\n        row \u003c\u003c record.age\n        row \u003c\u003c record.is_active\n      end\n    end\n  end \nend\n```\n\n## Implemented datatypes\n\nThe following DuckDB [SQL datatypes](https://duckdb.org/docs/sql/data_types/overview) plus 'NULL' are supported:\n\n| Crystal             | SQL Datatype        |\n| ------------------- | ------------------- |\n| `Nil`               | NULL                |\n| `Bool`              | BOOLEAN             |\n| `Int8`              | TINYINT             |\n| `Int16`             | SMALLINT            |\n| `Int32`             | INTEGER             |\n| `Int64`             | BIGINT              |\n| `Int128`            | HUGEINT             |\n| `UInt8`             | UTINYINT            |\n| `UInt16`            | USMALLINT           |\n| `UInt32`            | UINTEGER            |\n| `UInt64`            | UBIGINT             |\n| `Float32`           | FLOAT               |\n| `Float64`           | DOUBLE              |\n| `String`            | VARCHAR             |\n| `Bytes`             | BLOB                |\n| `DuckDB::Date`      | DATE                |\n| `DuckDB::TimeOfDay` | TIME                |\n| `DuckDB::Timestamp` | TIMESTAMP           |\n| `DuckDB::Interval`  | INTERVAL            |\n\n\nAll other DuckDB SQL datatypes are treated as `String`.\n\nYou can also use `DuckDB::Any`, which augments `DB::Any` according to the table above.\n\n### Time-related datatypes\n\nGiven the differences between time-related standard Crystal structs and DuckDB SQL datatypes, this shard implements lightweight Crystal structs to better interact with DuckDB.\n\nPlease note the following:\n\n* As DuckDB does not support timezones without an extension, all Crystal's `Time` instances **must be in UTC**\n* Crystal's `Time` and `Time::Span` resolutions are in nanoseconds while DuckDB's 'TIME' is in microseconds, thus beware of **loss of precision while converting between structs** (where integer division is used)\n* Creating a new `DuckDB::TimeOfDay` from a `Time::Span` greater or equal than a day raises a `DuckDB::Exception`\n* Converting a `DuckDB::Interval` with a non-zero month value to `Time::Span` without specifying the number of days per month raises a `DuckDB::Exception`\n\n```crystal\nrequire \"duckdb\"\n\ntime = Time.utc(1999, 12, 31, 10, 11, 59)\n\ndate = DuckDB::Date.new(1999, 12, 31)\ndate.year   # =\u003e 1999\ndate.month  # =\u003e 12\ndate.day    # =\u003e 31\ndate.to_s   # =\u003e \"1999-12-31\"\ndate == DuckDB::Date.new(\"1999-12-31\")  # =\u003e true\ndate == DuckDB::Date.new(time)          # =\u003e true\ndate.to_time == time                    # =\u003e false\n\ntime_of_day = DuckDB::TimeOfDay.new(10, 11, 59)\ntime_of_day.hour        # =\u003e 10\ntime_of_day.minute      # =\u003e 11\ntime_of_day.second      # =\u003e 59\ntime_of_day.microsecond # =\u003e 0\ntime_of_day.to_s        # =\u003e \"10:11:59\"\nspan = time_of_day.to_span\ntime_of_day == DuckDB::TimeOfDay.new(\"10:11:59\")  # =\u003e true\ntime_of_day == DuckDB::TimeOfDay.new(time)        # =\u003e true\ntime_of_day == DuckDB::TimeOfDay.new(span)        # =\u003e true\n\ntimestamp = DuckDB::Timestamp.new(date, time_of_day)\ntimestamp.date        # =\u003e \u003cDuckDB::Date\u003e\ntimestamp.time_of_day # =\u003e \u003cDuckDB::TimeOfDay\u003e\ntimestamp.to_span     # =\u003e \u003cTime::Span\u003e\ntime_of_day == DuckDB::Timestamp.new(\"1999-12-31 10:11:59\") # =\u003e true\ntimestamp == DuckDB::Timestamp.new(time)                    # =\u003e true\ntimestamp.to_time == time                                   # =\u003e true\n# Expected getters are delegated to date and time of_day\ntimestamp.year # =\u003e 1999\ntimestamp.hour # =\u003e 10\n\ninterval = DuckDB::Interval.new(0, 1, 2)\ninterval.months # =\u003e 2\ninterval.days # =\u003e 1\ninterval.microseconds # =\u003e 0\n# non-zero value for months; must indicate days per month while converting to Time::Span\ninterval.to_span(30) # =\u003e 61.00:00:00\ninterval.to_span(31) # =\u003e 63.00:00:00\ninterval.to_span(0) # =\u003e 1.00:00:00\ninterval.to_month_span # =\u003e Time::MonthSpan(@value=2)\ninterval.to_spans # =\u003e {1.00:00:00, Time::MonthSpan(@value=2)}\n```\n\nFor covenience you can also use `Time` to read from a result set, and a `Time` instance (in UTC) to bind to a prepared statement or append to a row; in this case it is automatically converted to `DuckDB::Timestamp`. However, when only reading a timestamp scalar you should use `#to_time` after reading the value in order to get a `Time` instance.\n\n```crystal\nrequire \"duckdb\"\n\nDB.connect DuckDB::IN_MEMORY do |cnn|\n  cnn.exec \"create table events (id integer, at timestamp)\"\n\n  cnn.exec \"insert into events values (?, ?)\", 1, Time::UNIX_EPOCH\n\n  cnn.appender(\"contacts\") do |appender|\n    appender.row do |row|\n      row \u003c\u003c 2\n      row \u003c\u003c Time.utc\n    end\n  end\n\n  cnn.query \"select * from events\" do |rs|\n    rs.each do\n      id = rs.read(Int32)\n      at = rs.read(Time)\n    end\n  end\n\n  timestamp = cnn.scalar \"select at from event where id = 1\"\n  timestamp.to_time == Time::UNIX_EPOCH # =\u003e true\nend\n```\n\n## Known issues\n\n* DuckDB v0.3.3 was a short-lived version —with v0.3.4 being a bug fix relase for it— hence there is no corresponding shard version.\n\n* For DuckDB v0.2.8 binding `UInt8` and `UInt16` values to prepared statements causes a crash due to a typo in the C header file. This is already fixed for later versions. See [this issue](https://github.com/duckdb/duckdb/issues/2105) for more information.\n\n\n## Contributing\n\n1. Fork it (\u003chttps://github.com/your-github-user/crystal-duckdb/fork\u003e)\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create a new Pull Request\n\nBe sure to run the specs with `crystal spec` before commiting and, if necessary, add the related specs for your new feature or change.\n\n## Contributors\n\n- [Amaury Trujillo](https://github.com/amauryt) - creator and maintainer\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famauryt%2Fcrystal-duckdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famauryt%2Fcrystal-duckdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famauryt%2Fcrystal-duckdb/lists"}