https://github.com/uwdata/arquero-sql

Database backend support for Arquero
https://github.com/uwdata/arquero-sql

Last synced: 7 days ago
JSON representation

Database backend support for Arquero

• Host: GitHub
• URL: https://github.com/uwdata/arquero-sql
• Owner: uwdata
• Created: 2020-11-22T09:59:44.000Z (almost 4 years ago)
• Default Branch: main
• Last Pushed: 2022-10-31T20:09:42.000Z (about 2 years ago)
• Last Synced: 2024-04-16T06:17:38.652Z (7 months ago)
• Language: JavaScript
• Homepage:
• Size: 301 KB
• Stars: 23
• Watchers: 6
• Forks: 1
• Open Issues: 4
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Arquero-SQL

[Arquero](https://github.com/uwdata/arquero) is a query processing JavaScript library.

So, your local machine memory is its limitation when dealing with large amount of data.

Arquero-SQL is a SQL backend support for Arquero.

Using the same syntax as Arquero in JavaScript, Aquero-SQL use your SQL server as an execution engine for transforming your data table.

With Arquero-SQL, your data is stored in a disk and can be transformed with a powerful remote machine, suited for big data analysis tasks.

Demo video: https://youtu.be/RO0luOvpiOY

## Setup

### Prerequisite

Set up at PostgreSQL server. We recommend using [Docker](https://www.docker.com/) with the official PostgreSQL [image](https://hub.docker.com/_/postgres/).

```sh

yarn

yarn build

# OR

npm install

npm run build

```

## Test

### Prerequisite

Set the following environment variables to your PostgreSQL server's credential

  - `PGDB`: Database

  - `PGUSER`: User name

  - `PGPASSWORD`: Password

  - `PGHOST`: Host name

  - `PGPORT`: Port

```sh

yarn test

# OR

npm test

```

## Usage

```js

import * as aq from 'arquero';

import * as fs from 'fs';

import {db} from 'arquero-sql';

// Connect to a database

const pg = new db.Postgres({

  name,

  password,

  host,

  database,

  port

});

// Create a data table

const dt1 = pg.fromArquero(aq.table(...));

// OR

const dt1 = pg.fromCSV(fs.readFileSync('path-to-file.csv'));

// Transform the data table

const dt2 = dt1

  .filter(d => d.Seattle > 200)

  .derive({new_col: d => d.Seattle + 10});

// Observe the transformation result

const output = await dt2.objects();

// OR

dt2.print();

```