{"id":17398436,"url":"https://github.com/vweevers/lento","last_synced_at":"2025-04-16T03:49:40.773Z","repository":{"id":29243092,"uuid":"120283439","full_name":"vweevers/lento","owner":"vweevers","description":"Streaming client for Presto HTTP protocol v1.","archived":false,"fork":false,"pushed_at":"2023-11-22T21:59:22.000Z","size":105,"stargazers_count":24,"open_issues_count":5,"forks_count":12,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-06T08:43:22.389Z","etag":null,"topics":["nodejs","npm-package","presto","prestodb","readable-stream"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vweevers.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-05T09:31:41.000Z","updated_at":"2024-05-17T16:47:04.000Z","dependencies_parsed_at":"2024-06-21T16:35:41.220Z","dependency_job_id":"f9de9578-45fc-4f7e-9140-27663a98e21b","html_url":"https://github.com/vweevers/lento","commit_stats":{"total_commits":96,"total_committers":7,"mean_commits":"13.714285714285714","dds":"0.11458333333333337","last_synced_commit":"eeaa14efc5245182da7a8354fcf5fd852246c6ed"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vweevers%2Flento","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vweevers%2Flento/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vweevers%2Flento/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vweevers%2Flento/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vweevers","download_url":"https://codeload.github.com/vweevers/lento/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249192553,"owners_count":21227804,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nodejs","npm-package","presto","prestodb","readable-stream"],"created_at":"2024-10-16T14:56:25.457Z","updated_at":"2025-04-16T03:49:40.743Z","avatar_url":"https://github.com/vweevers.png","language":"JavaScript","readme":"# lento \u003csup id=\"a1\"\u003e[1](#f1)\u003c/sup\u003e\n\n**Streaming Node.js client for [Presto](https://prestosql.io/), the \"Distributed SQL Query Engine for Big Data\".**\n\n[![npm status](http://img.shields.io/npm/v/lento.svg)](https://www.npmjs.org/package/lento)\n[![node](https://img.shields.io/node/v/lento.svg)](https://www.npmjs.org/package/lento)\n[![Travis build status](https://img.shields.io/travis/vweevers/lento.svg?label=travis)](http://travis-ci.org/vweevers/lento)\n[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)\n\n## Table of Contents\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\n- [Features](#features)\n- [Example](#example)\n- [API](#api)\n- [Install](#install)\n- [License](#license)\n\n\u003c/details\u003e\n\n## Features\n\n- Stream rows or pages of rows\n- Cancelation through [`stream.destroy()`](https://nodejs.org/api/stream.html#stream_readable_destroy_error)\n- Set session properties\n- Set time limit (enforced by Presto)\n- Get rows as objects or arrays\n- Supports both callbacks \u0026 promises\n- Uses [Presto HTTP protocol v1](https://github.com/prestosql/presto/wiki/HTTP-Protocol)\n- Keep-alive HTTP connections\n- Supports Gzip and Deflate content encoding\n- Retries [HTTP 503 and other failures](#builtin-retry).\n\n## Example\n\nConvert a Presto table to CSV:\n\n```js\nconst lento = require('lento')\nconst csvWriter = require('csv-write-stream')\nconst stdout = require('stdout-stream')\nconst pipeline = require('readable-stream').pipeline\n\nconst client = lento({\n  hostname: 'example',\n  port: 8080,\n  catalog: 'hive',\n  schema: 'test',\n  user: 'test'\n})\n\nconst source = client.createRowStream('SELECT * FROM events')\nconst transform = csvWriter()\n\npipeline(source, transform, stdout, (err) =\u003e {\n  if (err) throw err\n})\n```\n\nIf the destination streams close or error, the source stream is destroyed (courtesy of [`pipeline`](https://nodejs.org/docs/latest-v10.x/api/stream.html#stream_stream_pipeline_streams_callback)) and the Presto query canceled.\n\n## API\n\n- [`lento([options])`](#lentooptions)\n- [`createPageStream(sql[, options])`](#createpagestreamsql-options)\n  - [Cancelation](#cancelation)\n  - [Events](#events)\n- [`createRowStream(sql[, options])`](#createrowstreamsql-options)\n- [`query(sql[, options], callback)`](#querysql-options-callback)\n- [`setTimeout(duration[, options], callback)`](#settimeoutduration-options-callback)\n- [`resetTimeout([options, ]callback)`](#resettimeoutoptions-callback)\n- [`set(key, value[, options], callback)`](#setkey-value-options-callback)\n- [`reset(key[, options], callback)`](#resetkey-options-callback)\n- [`session([options, ]callback)`](#sessionoptions-callback)\n- [Errors](#errors)\n- [Builtin Retry](#builtin-retry)\n- [Debug](#debug)\n\n### `lento([options])`\n\nOptions:\n\n- `hostname`: string, default is `localhost`\n- `port`: number, default 8080\n- `protocol`: string, one of `http:` (default) or `https:`\n- `user`: string, default none. Sent as `X-Presto-User` header.\n- `timezone`: string, for example `UTC`, default none. Sent as `X-Presto-Time-Zone` header.\n- `headers`: object containing custom headers to set on every request such as `Authorization` (case-insensitive). Headers specified here take precedence over other options that set headers.\n- `parametricDatetime`: boolean, default false. Opt-in to datetime types with variable precision, for example `timestamp(6)`. When not set, datetime types are returned with a precision of 3.\n\nYou can specify a [catalog](https://prestosql.io/docs/current/overview/concepts.html#catalog) and [schema](https://prestosql.io/docs/current/overview/concepts.html#schema) to avoid writing fully-qualified table names in queries:\n\n- `catalog`: string, for example `hive`, default none. Sent as `X-Presto-Catalog` header.\n- `schema`: string, for example `logs`, default none. Sent as `X-Presto-Schema` header.\n\nControl delays and retries:\n\n- `pollInterval`: number (milliseconds) or string with unit (e.g. `1s`, `500ms`). How long to wait for server-side state changes, before sending another HTTP request. Default is 1 second.\n- `maxRetries`: number of retries if Presto responds with [HTTP 503 or other failures](#builtin-retry). Default is 10.\n- `socketTimeout`: number (milliseconds) or string with unit. When to timeout after inactivity on the socket. Default is 2 minutes.\n\n\u003ca name=\"createpagestream\"\u003e\u003c/a\u003e\n\n### `createPageStream(sql[, options])`\n\nExecute a query. Takes `sql` as a string or Buffer and returns a [readable stream](https://nodejs.org/api/stream.html#stream_readable_streams) that yields pages of rows.\n\n```js\nconst through2 = require('through2')\n\nclient\n  .createPageStream('SELECT * FROM events')\n  .pipe(through2.obj((page, enc, next) =\u003e {\n    for (let row of page) {\n      // ..\n    }\n\n    // Process next page\n    next()\n  }))\n```\n\nOptions:\n\n- `pageSize`: number, default 500\n- `highWaterMark`: number, default 0\n- `rowFormat`: string, one of `object` (default) or `array`\n- `deserialize`: boolean, default true\n- `headers`: custom request headers. Merged with headers that were set in the constructor, if any.\n\nThe `pageSize` specifies the maximum number of rows per page. Presto may return less per page. If Presto returns more rows than `pageSize`, the surplus is buffered and the stream will not make another HTTP request to Presto until fully drained. Note that if the (remainder of) rows fit in Presto's buffers, Presto will not block (until another HTTP request is made) but instead go into the `FINISHED` state after which you have 15 minutes (by default) to fetch the remaining results. If `pageSize` is \u0026lt;= 0 the stream emits pages as returned by Presto, without slicing them up.\n\nThe `highWaterMark` specifies how many pages to fetch preemptively. The maximum numbers of rows held in memory is approximately `(highWaterMark || 1) * pageSize`, plus any surplus from the last HTTP request. Because Presto can return thousands of rows per request, the default `highWaterMark` is 0 so that we _don't_ preemptively fetch and only hold the number of rows contained in the last HTTP request.\n\nIf you care more about throughput, you can opt to increase `highWaterMark`. Additionally you can increase `pageSize` if processing time is minimal or if you don't mind blocking the rest of your app while processing a page.\n\nFor tuning the Presto side of things, use [`set()`](#set).\n\n#### Cancelation\n\n[Destroying the stream](https://nodejs.org/api/stream.html#stream_readable_destroy_error) will cancel the query with a `DELETE` request to Presto, unless no requests were made yet. If the initial request is in flight the stream will wait for a response, which contains the query id that can then be cancelled. If cancelation fails the stream will emit an `error` (open an issue if you think it shouldn't). Regardless of success, the streams emits `close` as the last event.\n\n#### Events\n\nBesides the usual [Node.js stream events](https://nodejs.org/api/stream.html#stream_class_stream_readable), the stream emits:\n\n- `id`: emitted with query id once known\n- `info`: emitted with fully qualified info URL\n- `columns`: emitted with an array of column metadata as returned by Presto\n- `stats`: emitted for each HTTP response, with raw data\n- `state_change`: emitted with a string state (e.g. `RUNNING`, `FINISHED`) when Presto state changes. Should not be relied upon, only meant for debugging.\n- `raw_page_size`: emitted with a count of rows (e.g. 21829) for each HTTP response that has rows. For debugging.\n\n**Note** The `id`, `info` and `columns` events may be emitted more than once due to retries. Subsequent `stats` and `state_change` events pertain to that retried query as well.\n\n\u003ca name=\"createrowstream\"\u003e\u003c/a\u003e\n\n### `createRowStream(sql[, options])`\n\nExecute a query. Takes `sql` as a string or Buffer and returns a [readable stream](https://nodejs.org/api/stream.html#stream_readable_streams) that yields rows. Options:\n\n- `highWaterMark`: number, default 16\n- `rowFormat`: string, one of `object` (default) or `array`\n- `deserialize`: boolean, default true\n- `headers`: custom request headers. Merged with headers that were set in the constructor, if any.\n\n\u003ca name=\"query\"\u003e\u003c/a\u003e\n\n### `query(sql[, options][, callback])`\n\nSame as above but non-streaming, meant for simple queries. The `callback` function will receive an error if any and an array of rows. If no callback is provided, a promise is returned.\n\n```js\nclient.query('DESCRIBE events', (err, rows) =\u003e {\n  // ..\n})\n```\n\nWith async/await:\n\n```js\nconst rows = await client.query('DESCRIBE events')\n```\n\n\u003ca name=\"set-timeout\"\u003e\u003c/a\u003e\n\n### `setTimeout(duration[, options][, callback])`\n\nSet `query_max_run_time` for subsequent queries. If those take longer than `duration`, Presto will return an error with code `EXCEEDED_TIME_LIMIT` (see [errors](#errors)). The `duration` can be a number (in milliseconds) or a string parsed by Presto with the format `\u003cvalue\u003e\u003cunit\u003e` - for example `5d` or `100ms`. Options are passed to [`query()`](#query) via [`set()`](#set), as this method is a shortcut for:\n\n```js\nclient.set('query_max_run_time', duration[, options], callback)\n```\n\nIf no callback is provided, a promise is returned.\n\n\u003ca name=\"reset-timeout\"\u003e\u003c/a\u003e\n\n### `resetTimeout([options][, callback])`\n\nReset `query_max_run_time` to Presto's default. Options are passed to [`query()`](#query) via [`reset()`](#reset). If no callback is provided, a promise is returned.\n\n\u003ca name=\"set\"\u003e\u003c/a\u003e\n\n### `set(key, value[, options][, callback])`\n\nSet a session property. Executes `SET SESSION ..` to prevalidate input, then sets `X-Presto-Session` header on subsequent queries. Value can be a boolean, number or string. Options are passed to [`query()`](#query). If no callback is provided, a promise is returned.\n\n```js\nclient.set('redistribute_writes', false, (err) =\u003e {\n  if (err) return console.error('failed to set', err)\n\n  // Subsequent queries now use redistribute_writes=false\n})\n```\n\n\u003ca name=\"reset\"\u003e\u003c/a\u003e\n\n### `reset(key[, options][, callback])`\n\nReset a session property to its default value. Options are passed to [`query()`](#query). If no callback is provided, a promise is returned.\n\n```js\nclient.reset('redistribute_writes', (err) =\u003e {\n  if (err) return console.error('failed to reset', err)\n\n  // Subsequent queries now use the default value of redistribute_writes\n})\n```\n\n\u003ca name=\"show-session\"\u003e\u003c/a\u003e\n\n### `session([options, ][callback])`\n\nConverts the result of `SHOW SESSION` into a tree, coerces boolean and integer values to JSON types. Options are passed to [`query()`](#query). Callback signature is `(err, session)`.  If no callback is provided, a promise is returned.\n\nPartial example of a `session`:\n\n```json\n{\n  \"execution_policy\": {\n    \"key\": \"execution_policy\",\n    \"value\": \"all-at-once\",\n    \"default\": \"all-at-once\",\n    \"type\": \"varchar\",\n    \"description\": \"Policy used for scheduling query tasks\"\n  },\n  \"hash_partition_count\": {\n    \"key\": \"hash_partition_count\",\n    \"value\": 100,\n    \"default\": 100,\n    \"type\": \"integer\",\n    \"description\": \"Number of partitions for distributed joins and aggregations\"\n  },\n  \"hive\": {\n    \"bucket_execution_enabled\": {\n      \"key\": \"hive.bucket_execution_enabled\",\n      \"value\": true,\n      \"default\": true,\n      \"type\": \"boolean\",\n      \"description\": \"Enable bucket-aware execution: only use a single worker per bucket\"\n    }\n  }\n}\n```\n\nSee [Presto Properties](https://prestosql.io/docs/current/admin/properties.html) for a detailed description of (some of) the properties.\n\n### Errors\n\nErrors are enriched with a `code` and `type` (string) and optionally `info`. For example:\n\n```js\nclient.setTimeout('1ms', (err) =\u003e {\n  if (err) throw err\n\n  client.query('SELECT * FROM big_table', (err) =\u003e {\n    console.error(err.message) // 'EXCEEDED_TIME_LIMIT: Query exceeded maximum time limit of 1.00ms'\n    console.error(err.code) // 'EXCEEDED_TIME_LIMIT'\n    console.error(err.type) // 'INSUFFICIENT_RESOURCES'\n  })\n})\n```\n\n### Builtin Retry\n\nIf Presto responds to an HTTP request with 503 or if the TCP connection is refused, `lento` retries the request with an exponential delay between 1 and 10 seconds.\n\nIn addition, a query (consisting of one or more HTTP requests) will be retried if Presto returns an error like `SERVER_STARTING_UP` or `HIVE_METASTORE_ERROR`, but only if no data was received yet, to avoid emitting duplicates.\n\nI wish retries could be handled at a higher level, but as it stands, `lento` is both a low-level HTTP client and a streaming client, so retries have to be handled here. This may change in the future.\n\n### Debug\n\nEnable debug output with `DEBUG=lento`. Mostly logs HTTP requests and retries, no usernames, SQL or other potentially sensitive data. Beware, it can log hundreds of lines per query.\n\n## Install\n\nWith [npm](https://npmjs.org) do:\n\n```\nnpm install lento\n```\n\n## License\n\n[MIT](LICENSE) © 2018-present Vincent Weevers\n\n---\n\n\u003csup\u003e\u003cb id=\"f1\"\u003e1\u003c/b\u003e\u003c/sup\u003e Because streams are about [slowing down](https://www.youtube.com/watch?v=Yn-U6ygClyU)! [↩](#a1)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvweevers%2Flento","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvweevers%2Flento","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvweevers%2Flento/lists"}