{"id":13695594,"url":"https://github.com/balance-platform/pillar","last_synced_at":"2025-04-05T21:07:43.089Z","repository":{"id":37941501,"uuid":"250782700","full_name":"balance-platform/pillar","owner":"balance-platform","description":"Elixir library client for work with ClickHouse","archived":false,"fork":false,"pushed_at":"2023-04-27T22:06:53.000Z","size":176,"stargazers_count":84,"open_issues_count":6,"forks_count":28,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-08-01T02:13:02.447Z","etag":null,"topics":["clickhouse","driver","elixir","hex","library","pillar"],"latest_commit_sha":null,"homepage":"https://hex.pm/packages/pillar","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/balance-platform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-28T11:55:16.000Z","updated_at":"2024-06-26T08:55:29.000Z","dependencies_parsed_at":"2024-05-01T17:33:21.565Z","dependency_job_id":null,"html_url":"https://github.com/balance-platform/pillar","commit_stats":{"total_commits":153,"total_committers":23,"mean_commits":"6.6521739130434785","dds":0.3529411764705882,"last_synced_commit":"3fcff9750bdaab67558ab30206e8deb6237d7b9c"},"previous_names":["sofakingworld/pillar"],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balance-platform%2Fpillar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balance-platform%2Fpillar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balance-platform%2Fpillar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balance-platform%2Fpillar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/balance-platform","download_url":"https://codeload.github.com/balance-platform/pillar/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247399877,"owners_count":20932876,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","driver","elixir","hex","library","pillar"],"created_at":"2024-08-02T18:00:30.427Z","updated_at":"2025-04-05T21:07:43.060Z","avatar_url":"https://github.com/balance-platform.png","language":"Elixir","funding_links":[],"categories":["Language bindings","Elixir"],"sub_categories":["Elixir"],"readme":"# Pillar\n\n[![github.com](https://github.com/balance-platform/pillar/workflows/build/badge.svg?branch=master)](https://github.com/balance-platform/pillar/actions)\n[![hex.pm](https://img.shields.io/badge/docs-hexpm-blue.svg)](https://hexdocs.pm/pillar)\n[![hex.pm](https://img.shields.io/hexpm/v/pillar.svg)](https://hex.pm/packages/pillar)\n[![hex.pm](https://img.shields.io/hexpm/dt/pillar.svg)](https://hex.pm/packages/pillar)\n[![hex.pm](https://img.shields.io/hexpm/l/pillar.svg)](https://hex.pm/packages/pillar)\n[![github.com](https://img.shields.io/github/last-commit/balance-platform/pillar.svg)](https://github.com/balance-platform/pillar/commits/master)\n\nElixir client for [ClickHouse](https://clickhouse.tech/), a fast open-source\nOnline Analytical Processing (OLAP) database management system.\n\n# Features\n\n  - [Direct Usage with connection structure](#direct-usage-with-connection-structure)\n  - [Pool of workers](#pool-of-workers)\n  - [Async insert](#async-insert)\n  - [Buffer for periodical bulk inserts](#buffer-for-periodical-bulk-inserts)\n  - [Migrations](#migrations)\n  - [DateTime Timezones](#timezones)\n  - [Switching between HTTP adapters](#http-adapters)\n\n## Usage\n\n### Direct Usage with connection structure\n\n```elixir\nconn = Pillar.Connection.new(\"http://user:password@localhost:8123/database\")\n\n# Params are passed in brackets {} in SQL query, and map strtucture does fill\n# query by values.\nsql = \"SELECT count(*) FROM users WHERE lastname = {lastname}\"\n\nparams = %{lastname: \"Smith\"}\n\n{:ok, result} = Pillar.query(conn, sql, params)\n\nresult\n#=\u003e [%{\"count(*)\" =\u003e 347}]\n\n```\n\n### Pool of workers\n\nRecommended usage, because of limited connections and supervised workers.\n\n```elixir\ndefmodule ClickhouseMaster do\n  use Pillar,\n    connection_strings: [\n      \"http://user:password@host-master-1:8123/database\",\n      \"http://user:password@host-master-2:8123/database\"\n    ],\n    name: __MODULE__,\n    pool_size: 15\nend\n\nClickhouseMaster.start_link()\n\n{:ok, result} = ClickhouseMaster.select(sql, %{param: value})\n```\n\n### Async insert\n\n```elixir\nconnection = Pillar.Connection.new(\"http://user:password@host-master-1:8123/database\")\n\nPillar.async_insert(connection, \"INSERT INTO events (user_id, event) SELECT {user_id}, {event}\", %{\n  user_id: user.id,\n  event: \"password_changed\"\n}) # =\u003e :ok\n```\n\n### Buffer for periodical bulk inserts\n\nFor this feature required [Pool of workers](#pool-of-workers).\n\n```elixir\ndefmodule BulkToLogs do\n  use Pillar.BulkInsertBuffer,\n    pool: ClickhouseMaster,\n    table_name: \"logs\",\n    # interval_between_inserts_in_seconds, by default -\u003e 5\n    interval_between_inserts_in_seconds: 5,\n    # on_errors is optional\n    on_errors: \u0026__MODULE__.dump_to_file/2\n\n  @doc \"\"\"\n  dump to file function store failed inserts into file \n  \"\"\"\n  def dump_to_file(_result, records) do\n    File.write(\"bad_inserts/#{DateTime.utc_now()}\", inspect(records))\n  end\n\n  @doc \"\"\"\n  retry insert is dangerous (but it is possible and listed as proof of concept)\n\n  this function may be used in `on_errors` option\n  \"\"\"\n  def retry_insert(_result, records) do\n    __MODULE__.insert(records)\n  end\nend\n```\n\n```elixir\n:ok = BulkToLogs.insert(%{value: \"online\", count: 133, datetime: DateTime.utc_now()})\n:ok = BulkToLogs.insert(%{value: \"online\", count: 134, datetime: DateTime.utc_now()})\n:ok = BulkToLogs.insert(%{value: \"online\", count: 132, datetime: DateTime.utc_now()})\n....\n\n# All this records will be inserted with 5 second interval.\n```\n\n*on_errors* parameter allows you to catch any error of bulk insert (for example: one of batch is bad or clickhouse was not available )\n\n\n### Migrations\n\nMigrations can be generated with mix task `mix pillar.gen.migration migration_name`.\n\nMulti-statement migration Example for this [UseCase](https://github.com/balance-platform/pillar/issues/61)\n\n```elixir\ndefmodule Pillar.Migrations.CreateMultipleTables do\n  def up do\n    # for MultiStatement migration result of this function should be List of Strings  \n    (0..4) |\u003e Enum.map(fn i -\u003e\n      \"CREATE TABLE IF NOT EXISTS shard_#{i} (field FixedString(10)) ENGINE = Memory\"\n    end)\n  end\nend\n```\n\n```bash\nmix pillar.gen.migration events_table\n```\n\nBut for launching them we have to write own task, like this:\n\n```elixir\ndefmodule Mix.Tasks.MigrateClickhouse do\n  use Mix.Task\n  def run(_args) do\n    connection_string = Application.get_env(:my_project, :clickhouse_url)\n    conn = Pillar.Connection.new(connection_string)\n    Pillar.Migrations.migrate(conn)\n  end\nend\n```\n\nAnd launch this via command.\n\n```bash\nmix migrate_clickhouse\n```\n\n### Timezones\n\nIn order to be able to use Timezones add timezones database to your project and configure your app:\n\n```elixir\nconfig :elixir, :time_zone_database, Tzdata.TimeZoneDatabase\n```\n\nDetails here https://hexdocs.pm/elixir/1.12/DateTime.html#module-time-zone-database\n\n### HTTP Adapters\n\nIf you have problems with default Pillar HTTP Adapter (Mint over Tesla), you can use alternative one, based on :httpc or define your own and pass it\nthrough config.\n\n```\nconfig :pillar, Pillar.HttpClient, http_adapter: Pillar.HttpClient.TeslaMintAdapter\n```\n\nAdapter should define one function `post/3` and return 2 possible results (`%Pillar.HttpClient.Response{}`, `%Pillar.HttpClient.TransportError{}`)\n\n# Contribution\n\nFeel free to make a pull request. All contributions are appreciated!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbalance-platform%2Fpillar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbalance-platform%2Fpillar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbalance-platform%2Fpillar/lists"}