Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tomchen/fetchline
Read text file (remote over HTTP(S) or local) line by line as async iterator, with Node, browsers and Deno
https://github.com/tomchen/fetchline
async-iterator browser deno fetch file isomorphic iterable iterator lerna line monorepo node nodejs promise readline
Last synced: 23 days ago
JSON representation
Read text file (remote over HTTP(S) or local) line by line as async iterator, with Node, browsers and Deno
- Host: GitHub
- URL: https://github.com/tomchen/fetchline
- Owner: tomchen
- License: mit
- Created: 2021-02-04T16:29:09.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2021-02-14T15:23:36.000Z (almost 4 years ago)
- Last Synced: 2024-04-29T09:40:34.191Z (7 months ago)
- Topics: async-iterator, browser, deno, fetch, file, isomorphic, iterable, iterator, lerna, line, monorepo, node, nodejs, promise, readline
- Language: TypeScript
- Homepage:
- Size: 1.62 MB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Fetch Line JavaScript packages: read remote or local file line by line as async iterator
Read text file (remote over HTTP(S) or local) line by line as async iterator, with Node, browsers and Deno.
[This GitHub monorepo](https://github.com/tomchen/fetchline/) hosts 6 npm packages and 1 Deno module. They all serve a similar, simple purpose: read text file line by line and return an asynchronous iterable iterator of strings. They all try to be efficient and fast, and are written in TypeScript. However, their environment / platforms and exact purpose, features and behavior differ.
[](https://github.com/tomchen/fetchline/actions) [](https://nodejs.org/) [](https://deno.land/) [](https://lerna.js.org/) [](https://github.com/tomchen/fetchline/blob/main/LICENSE)
**TLDR: read the ["Purpose & environment" table](#purpose--environment), pick the package you need, have a look at the [Usage section](#usage), know at least a little bit about JavaScript's [async / await](#async--await), and go ahead to use them.**
## Comparison
### Purpose & environment
Package / Module Name
Rec?
Fetch remote file over HTTP(S)
Read local file
Version
fetchline
👍
✅
✅
nodefetchline
👍
✅
isomorphic-fetchline
✅
✅
naivefetchline
✅
✅
getfileline
✅
readlineiter
👍
✅
readlineiter for Deno
👍
✅
If you are not sure, just use the recommended one that has the correct environment and purpose you need.
**Tested:**
* Node.js: ≥ 12
* Deno: ≥ v1.2.0
* Modern browsers (Google Chrome, Firefox, Safari, Microsoft Edge, Opera, Samsung Internet): all latest
## Usage
### fetchline, nodefetchline, isomorphic-fetchline, and naivefetchline
#### Examples
```bash
npm install fetchline
```
```js
import fetchline from 'fetchline'
const lineIterator =
fetchline('https://raw.githubusercontent.com/tomchen/fetchline/main/testfile/crlf_finalnewline')
// This is the same as:
// fetchline(
// 'https://raw.githubusercontent.com/tomchen/fetchline/main/testfile/crlf_finalnewline',
// {
// includeLastEmptyLine: true,
// encoding: 'utf-8',
// delimiter: /\r?\n/g,
// }
// )
;(async () => {
for await (const line of lineIterator) {
// do something with `line`
}
})()
```
Change `'fetchline'` to `nodefetchline`, `isomorphic-fetchline`, or `naivefetchline` if you use these packages instead.
For Deno: `import fetchline from 'https://github.com/tomchen/fetchline/blob/main/packages/fetchline/src/index.ts'`
Change the `import` line to syntax like `const nodefetchline = require('nodefetchline')` if you use nodefetchline or isomorphic-fetchline package in Node's CommonJS.
For browsers:
```html
fetchline(...) // same as above
```
#### Details
These four packages have exactly the same interface (parameters and return value):
| Parameter Name | Required? | Type | Default Value | Description |
| :----------------------------- | :-------- | :------------------- | :------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filepath` | Required | *string* | *N/A* | URL or path of the text file |
| `options` | Optional | *object* | `{}` | options, including the following three |
| `options.includeLastEmptyLine` | Optional | *boolean* | `true` | Should it count the last empty line? |
| `options.encoding` | Optional | *string* | `'utf-8'` | File encoding |
| `options.delimiter` | Optional | *string* or *RegExp* | `/\r?\n/g` | Line (or other item)'s delimiter / separator.
**NOTE:** do not set it as something like `/\r\n\|\n\|\r/g`, it causes trouble when one of the chunks of a CRLF (`\r\n`)-EOL file ends with CR (`\r`) |
**Return value:** { *AsyncIterableIterator\* } An asynchronous iterable iterator containing each line in string from the text file
### readlineiter, readlineiter for Deno, and getfileline
They have **similar** interface as the aforementioned fetchline, nodefetchline, isomorphic-fetchline, and naivefetchline, **but do not have the second parameter, `options`**, and everything `options` contains.
```bash
npm install readlineiter
```
```js
import readlineiter from 'readlineiter' // For Deno: import readlineiter from 'https://raw.githubusercontent.com/tomchen/fetchline/main/packages/readlineiter-deno/mod.ts'
const lineIterator = readlineiter('./crlf_finalnewline')
;(async () => {
for await (const line of lineIterator) {
// do something with `line`
}
})()
```
## Further comparison
### Characteristics
| | ASAP | 0 dependencies | TypeScript |
| --------------------- | :---: | :----------: | :--------: |
| fetchline | ✅ | ✅ | ✅ |
| nodefetchline | ✅ | ✅ | ✅ |
| isomorphic-fetchline | ✅ | ✅ | `.d.ts` |
| naivefetchline | ❌ | ✅ | ✅ |
| getfileline | ✅ | ✅ | ✅ |
| readlineiter | ✅ | ✅ | ✅ |
| readlineiter for Deno | ✅ | ✅ | ✅ |
**ASAP:**
* These remote file requesting libs should resolve with the line text string as soon as possible, i.e. as soon as the chunks that have arrived can form the next complete line
* Except for naivefetchline that is, well, naïve, I really can't blame it
* The local file reading libs read the file with pointer, rather than get a whole string in memory then split the string
**0 dependencies:** no external non-dev dependency for npm packages. Note that:
* Node libraries inevitably use native Node libraries `http` and `https`, or `fs`
* getfileline and readlineiter also use `readline` native lib directly thus are just wrappers, but other packages here use own low-level method
* "readlineiter for Deno" uses Deno Standard Module [`bufio.ts`](https://deno.land/std/io/bufio.ts).
**TypeScript:** the source code is in TypeScript, except for isomorphic-fetchline's source which is in JavaScript but has type definition (`.d.ts`).
As for the production / dist files, these packages are all compiled into different [module versions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) where possible: Node.js' [Common.js](https://en.wikipedia.org/wiki/CommonJS) (which uses `require()`), ES Module (native JS module with `import`), and minified [UMD](https://github.com/umdjs/umd) that is good for browser.
### Parameters amd return value
| | `filepath` parameter | `includeLastEmptyLine` option | `encoding` option | `delimiter` option | Return `AsyncIterableIterator` |
| --------------------- | :------------------: | :---------------------------: | :---------------: | :--------------------------------------------------------------------------: | :------------------------------------: |
| fetchline | ✅ | ✅ | ✅ | ✅ | ✅ |
| nodefetchline | ✅ | ✅ | ✅ | ✅ | ✅ |
| isomorphic-fetchline | ✅ | ✅ | ✅ | ✅ | ✅ |
| naivefetchline | ✅ | ✅ | ✅ | ✅ | ✅ |
| getfileline | ✅ | ❌, always doesn't | ❌, always utf-8 | ❌, always EOL detected by [`readline`](https://nodejs.org/api/readline.html) | ✅ |
| readlineiter | ✅ | ❌, always doesn't | ❌, always utf-8 | ❌, always EOL detected by [`readline`](https://nodejs.org/api/readline.html) | ✅ |
| readlineiter for Deno | ✅ | ❌, always does | ❌, always utf-8 | ❌, always EOL detected by [`bufio.ts`](https://deno.land/std/io/bufio.ts) | ✅ |
getfileline and readlineiter's delimiter is EOL character detected by [`readline`](https://nodejs.org/api/readline.html) native module with its [`crlfDelay` option](https://nodejs.org/api/readline.html#readline_readline_createinterface_options) set to `Infinity`.
## Tips & thoughts
### `async` / `await`
Of course, you should at least know a little bit about [async / await](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function), `await asyncIterator.next()` or [`for await of`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of) before using the packages here. If you don't, click the links to read the article from MDN Web Docs. Basically, you can do this:
```js
;(async () => {
for await (const line of lineIterator) {
// do something with `line`
}
})()
```
Or this:
```js
;(async () => {
let line
let isDone
while (1) {
;({ value: line, done: isDone } = await lineIterator.next())
// do something with `line`
if (isDone) {
break
}
}
})()
```
### Line-delimited JSON
These packages, especially 'fetchline' (the first one) for browsers, could be helpful for [line-delimited JSON](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON) (aka. [ndjson](http://ndjson.org/) (Newline Delimited JSON), [JSON Lines](https://jsonlines.org/)) parsing. You could write something like:
```js
import fetchline from 'fetchline'
const lineIterator = fetchline(lineDelimitedJsonUrl)
;(async () => {
for await (const line of lineIterator) {
const lineJson = JSON.parse(line)
// do something with `lineJson`
}
})()
```
### Development
This is a [Lerna](https://lerna.js.org/) powered monorepo with mixed code (TypeScript / JavaScript), mixed module version (CommonsJS, ES Module, UMD) and cross-environment (Node.js, Deno, browsers) support, automated tests with GitHub Actions CI. Look at the root [package.json](https://github.com/tomchen/fetchline/blob/main/package.json) and individual packages' package.json for available scripts, to get started, `yarn` then `yarn bootstrap`. You could also use the repo as an example of Lerna monorepo.