{"id":13695691,"url":"https://github.com/apla/node-clickhouse","last_synced_at":"2025-04-02T03:08:19.615Z","repository":{"id":9986936,"uuid":"64013655","full_name":"apla/node-clickhouse","owner":"apla","description":"Yandex ClickHouse driver for nodejs","archived":false,"fork":false,"pushed_at":"2023-02-09T11:52:28.000Z","size":158,"stargazers_count":217,"open_issues_count":30,"forks_count":48,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-25T16:19:41.634Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apla.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2016-07-23T11:41:47.000Z","updated_at":"2024-10-07T12:34:15.000Z","dependencies_parsed_at":"2024-04-13T00:48:36.195Z","dependency_job_id":null,"html_url":"https://github.com/apla/node-clickhouse","commit_stats":{"total_commits":146,"total_committers":14,"mean_commits":"10.428571428571429","dds":0.3356164383561644,"last_synced_commit":"46dc6b70d42c9dd5a0712cfeb77bc39d9a428ee3"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apla%2Fnode-clickhouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apla%2Fnode-clickhouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apla%2Fnode-clickhouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apla%2Fnode-clickhouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apla","download_url":"https://codeload.github.com/apla/node-clickhouse/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246746932,"owners_count":20827061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T18:00:32.408Z","updated_at":"2025-04-02T03:08:19.585Z","avatar_url":"https://github.com/apla.png","language":"JavaScript","funding_links":[],"categories":["Language bindings"],"sub_categories":["JavaScript/Typescript"],"readme":"Simple and powerful interface for [ClickHouse](https://clickhouse.yandex/) [![travis](https://travis-ci.org/apla/node-clickhouse.svg)](https://travis-ci.org/apla/node-clickhouse) [![codecov](https://codecov.io/gh/apla/node-clickhouse/branch/master/graph/badge.svg)](https://codecov.io/gh/apla/node-clickhouse)\n===\n```sh\nnpm install @apla/clickhouse\n```\n\nSynopsis\n---\n```javascript\nconst ClickHouse = require('@apla/clickhouse')\nconst ch = new ClickHouse({ host, port, user, password })\n\nconst stream = ch.query(\"SELECT 1\", (err, data) =\u003e {})\nstream.pipe(process.stdout)\n\n// promise interface, not recommended for selects\n// (requires 'util.promisify' for node \u003c 8, Promise shim for node \u003c 4)\nawait ch.querying(\"CREATE DATABASE test\")\n```\nExamples:\n- [Selecting large dataset](README.md#selecting-large-dataset)\n- [Inserting large dataset](README.md#inserting-large-dataset)\n- [Inserting single row](README.md#insert-single-row-of-data)\n\n\nAPI\n---\n\n### `new ClickHouse(options: Options)`\n\n#### `Options`\n\n|                  | required | default       | description\n| :--------------- | :------: | :------------ | :----------\n| `host`           | ✓        |               | Host to connect.\n| `user`           |          |               | Authentication user.\n| `password`       |          |               | Authentication password.\n| `path` (`pathname`) |       | `/`           | Pathname of ClickHouse server.\n| `port`           |          | `8123`        | Server port number.\n| `protocol`       |          | `'http:'`     | `'https:'` or `'http:'`.\n| `dataObjects`    |          | `false`       | By default (`false`), you'll receive array of values for each row. \u003cbr /\u003e If you set `dataObjects: true`, every row will become an object with format: `{ fieldName: fieldValue, … }`. \u003cbr /\u003e Alias to `format: 'JSON'`.\n| `format`         |          | `JSONCompact` | Adds the `FORMAT` statement for query if it did not have one. \u003cbr /\u003e Specifies format of [selected](https://clickhouse.yandex/docs/en/query_language/select/#format-clause) or [inserted](https://clickhouse.yandex/docs/en/query_language/insert_into/#insert) data. \u003cbr /\u003e See [\"Formats for input and output data\"](https://clickhouse.yandex/docs/en/interfaces/formats/#formats) to find out possible values.\n| `queryOptions`   |          |               | Object, can contain any ClickHouse option from [Settings](https://clickhouse.yandex/docs/en/operations/settings/index.html), [Restrictions](https://clickhouse.yandex/docs/en/operations/settings/query_complexity/) and [Permissions](https://clickhouse.yandex/docs/en/operations/settings/permissions_for_queries/). \u003cbr /\u003e See [example](README.md#settings-for-connection).\n| `readonly`       |          | `false`       | Tells driver to send query with HTTP GET method. Same as [`readonly=1` setting](https://clickhouse.yandex/docs/en/operations/settings/permissions_for_queries/#settings_readonly). [More details](https://clickhouse.yandex/docs/en/interfaces/http/).\n| `timeout`, \u003cbr /\u003e `headers`, \u003cbr /\u003e `agent`, \u003cbr /\u003e `localAddress`, \u003cbr /\u003e `servername`, \u003cbr /\u003e etc… |   |   |  Any [http.request](https://nodejs.org/api/http.html#http_http_request_options_callback) or [https.request](https://nodejs.org/api/https.html#https_https_request_options_callback) options are also available.\n\n\u003c!--\nThis are dangerous for using by end user\n\n| `syncParser`     |          | `false`       | **Not recommended for large amounts of data!** \u003cbr /\u003e Collects all data, then parse entire response. \u003cbr /\u003e May be faster, but for large datasets all your dataset goes into memory (actually, entire response + entire dataset).\n# Might be completely replaced with promise interface.\n\n| `omitFormat`     |          | `false`       | By default `FORMAT JSONCompact` statement will be added to the query if it did not have it. \u003cbr /\u003e You can change disable this behaviour by providing this option.\n# Looks like internal option\n--\u003e\n\n##### Options example:\n```javascript\nconst ch = new ClickHouse({\n  host: \"clickhouse.msk\",\n  dataObjects: true,\n  readonly: true,\n  queryOptions: {\n    profile: \"web\",\n    database: \"test\",\n  },\n})\n```\n\n\n### `clickHouse.query(query, [options], [callback])`\nSends a query statement to a server.\n\n##### `query: string`\nSQL query statement.\n\n##### `options: Options`\nThe same [`Options`](README.md#options), excluding connection options.\n\n##### `callback: (error, result) =\u003e void`\nWill be always called upon completion.\n\n##### Returns: [`DuplexStream`](https://nodejs.org/api/stream.html#stream_duplex_and_transform_streams)\nIt supports [`.pipe`](https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options) to process records. \u003cbr/\u003e\nYou should have at least one error handler listening. Via query callback or via stream `error` event.\n\n| Stream event | Description\n| ------------ | -----------\n| `'error'`      | Query execution finished with error. \u003cbr /\u003e If you have both query `callback` and stream `error` listener, you'll have error notification in both listeners.\n| `'metadata'`   | When a column information is parsed.\n| `'data'`       | When a row is available.\n| `'end'`        | When entire response is processed. \u003cblockquote\u003eRegardless of whether there is an `'end'` listener, the query `callback` are always called.\u003c/blockquote\u003e \u003cblockquote\u003eYou should always listen to `'data'` event together with `'end'` event. \u003cbr/\u003e[\"The 'end' event will not be emitted unless the data is completely consumed.\"](https://nodejs.org/api/stream.html#stream_event_end) \u003cbr/\u003e If you don't need to handle `'data'` event prefer to use only `callback` or [Promise interface](#promise-interface).\u003c/blockquote\u003e\n\n##### `stream.supplemental`\nAfter response is processed, you can read a supplemental response data from it, such as row count.\n\n\nExamples:\n- [Selecting with stream](README.md#selecting-with-stream)\n- [Inserting with stream](README.md#inserting-with-stream)\n\n### `clickHouse.ping(callback)`\nSends an empty query.\nDoesn't requires authorization.\n\n##### `callback: (error, result) =\u003e void`\nWill be called upon completion.\n\n\u003cbr /\u003e\n\n## Promise interface\n\nPromise interface **is not recommended** for `INSERT` and `SELECT` queries.\n* `INSERT` can't do bulk load data with promise interface.\n* `SELECT` will collect entire query result in the memory. See the [Memory size](README.md#memory-size) section.\n\nWith promise interface query result are parsed synchronously.\nThis means that large query result in promise interface:\n* Will synchronously block JS thread/event loop.\n* May lead to memory leaks in your app due peak GC loads.\n\nUse it only for queries where resulting data size is is known and extremely small.\u003cbr/\u003e\nThe good cases to use it is `DESCRIBE TABLE` or `EXISTS TABLE`\n\n### `clickHouse.querying(query, [options])`\nSimilar to `ch.query(query)` but collects entire response in memory and resolves with complete query result. \u003cbr /\u003e\nSee the [Memory size](README.md#memory-size) section.\n##### `options: Options`\nThe same [`Options`](README.md#options), excluding connection options.\n\n##### Returns: `Promise`\nWill be resolved with entire query result.\n\nExample of [promise interface](README.md#promise-interface).\n\n### `clickHouse.pinging()`\nPromise interface for [`.ping`](README.md#clickhousepingcallback).\n\n##### Returns: `Promise`\n\n\u003cbr /\u003e\n\nHow it works\n-----\n\n### Bulk data loading with `INSERT` statements\n\n`INSERT` can be used for bulk data loading. There is a 2 formats easily implementable\nwith javascript: CSV and TabSeparated/TSV.\n\nCSV is useful for loading from file, thus you can read and `.pipe` into clickhouse\nfile contents. \u003cbr /\u003e\nTo activate CSV parsing you should set `format` driver option or query `FORMAT` statement to `CSV`:\n\n```javascript\n\nvar csvStream = fs.createReadStream('data.csv')\nvar clickhouseStream = ch.query(statement, { format: CSV })\n\ncsvStream.pipe(clickhouseStream)\n\n```\n\nTSV is useful for loading from file and bulk loading from external sources, such as other databases.\nOnly `\\\\`, `\\t` and `\\n` need to be escaped in strings; numbers, nulls,\nbools and date objects need some minor processing. You can send prepared TSV data strings\n(line ending will be appended automatically), buffers (always passed as is) or Arrays with fields.\n\nInternally, every field will be converted to the format which ClickHouse can accept.\nThen escaped and joined with delimiter for the particular format.\nIf you ever need to store rows (in arrays) and send preformatted data, you can do it.\n\nClickHouse also supports [JSONEachRow](https://clickhouse.yandex/docs/en/formats/jsoneachrow.html) format\nwhich can be useful to insert javascript objects if you have such recordset.\n\n```js\nconst stream = ch.query(statement, { format: 'JSONEachRow' })\n\nstream.write(object) // Do write as many times as possible\nstream.end() // And don't forget to finish insert query\n```\n\n### Memory size\n\nYou can read all the records into memory in single call like this:\n\n```javascript\n\nvar ch = new ClickHouse({ host: host, port: port })\nch.querying(\"SELECT number FROM system.numbers LIMIT 10\", (err, result) =\u003e {\n  // result will contain all the data you need\n})\n\n```\n\nIn this case whole JSON response from the server will be read into memory,\nthen parsed into memory hogging your CPU. Default parser will parse server response\nline by line and emits events. This is slower, but much more memory and CPU efficient\nfor larger datasets.\n\n\u003cbr /\u003e\n\n## Examples\n#### Selecting with stream:\n```javascript\nconst readableStream = ch.query(\n  'SELECT * FROM system.contributors FORMAT JSONEachRow',\n  (err, result) =\u003e {},\n)\nconst writableStream = fs.createWriteStream('./contributors.json')\nreadableStream.pipe(writableStream)\n```\n\n#### Inserting with stream:\n```javascript\nconst readableStream = fs.createReadStream('./x.csv')\nconst writableStream = ch.query('INSERT INTO table FORMAT CSV', (err, result) =\u003e {})\nreadableStream.pipe(writableStream)\n```\n\n#### Insert single row of data:\n```javascript\nconst ch = new ClickHouse(options)\nconst writableStream = ch.query(`INSERT INTO table FORMAT TSV`, (err) =\u003e {\n  if (err) {\n    console.error(err)\n  }\n  console.log('Insert complete!')\n})\n\n// data will be formatted for you\nwritableStream.write([1, 2.22, \"erbgwerg\", new Date()])\n\n// prepare data yourself\nwritableStream.write(\"1\\t2.22\\terbgwerg\\t2017-07-17 17:17:17\")\n\nwritableStream.end()\n\n```\n\n#### Selecting large dataset:\n\n```javascript\nconst ch = new ClickHouse(options)\n// it is better to use stream interface to fetch select results\nconst stream = ch.query(\"SELECT * FROM system.numbers LIMIT 10000000\")\n\nstream.on('metadata', (columns) =\u003e { /* do something with column list */ })\n\nlet rows = []\nstream.on('data', (row) =\u003e rows.push(row))\n\nstream.on('error', (err) =\u003e { /* handler error */ })\n\nstream.on('end', () =\u003e {\n  console.log(\n    rows.length,\n    stream.supplemental.rows,\n    stream.supplemental.rows_before_limit_at_least, // how many rows in result are set without windowing\n  )\n})\n```\n\n#### Inserting large dataset:\n```javascript\nconst ch = new ClickHouse(options)\n// insert from file\nconst tsvStream = fs.createReadStream('data.tsv')\nconst clickhouseStream = ch.query('INSERT INTO table FORMAT TSV')\n\ntsvStream.pipe(clickhouseStream)\n```\n\n#### Settings for connection:\n```javascript\nconst ch = new ClickHouse({\n  host: 'clickhouse.msk',\n  queryOptions: {\n    database: \"test\",\n    profile: \"web\",\n    readonly: 2,\n    force_index_by_date: 1,\n    max_rows_to_read: 10 * 1e6,\n  },\n})\n```\n\n#### Settings for query:\n```javascript\nconst ch = new ClickHouse({ host: 'clickhouse.msk' })\nconst stream = ch.query('INSERT INTO table FORMAT TSV', {\n  queryOptions: {\n    database: \"test\",\n    insert_quorum: 2,\n  },\n})\n```\n\n#### Promise interface:\n```js\nconst ch = new ClickHouse(options)\n// Check connection to server. Doesn't requires authorization.\nawait ch.pinging()\n```\n```js\nconst { data } = await ch.querying(\"SELECT 1\")\n// [ [ 1 ] ]\nconst { data } = await ch.querying(\"DESCRIBE TABLE system.numbers\", { dataObjects: true })\n// [ { name: 'number', type: 'UInt64', default_type: '', default_expression: '' } ]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapla%2Fnode-clickhouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapla%2Fnode-clickhouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapla%2Fnode-clickhouse/lists"}