Arquero is a query processing JavaScript library. So, your local machine memory is its limitation when dealing with large amount of data. Arquero-SQL is a SQL backend support for Arquero. Using the same syntax as Arquero in JavaScript, Aquero-SQL use your SQL server as an execution engine for transforming your data table. With Arquero-SQL, your data is stored in a disk and can be transformed with a powerful remote machine, suited for big data analysis tasks.

Demo video: https://youtu.be/RO0luOvpiOY

Setup

Prerequisite

Set up at PostgreSQL server. We recommend using Docker with the official PostgreSQL image.

yarn
yarn build
# OR
npm install
npm run build

Test

Prerequisite

Set the following environment variables to your PostgreSQL server's credential

• PGDB: Database
• PGUSER: User name
• PGPASSWORD: Password
• PGHOST: Host name
• PGPORT: Port

yarn test
# OR
npm test

Usage

import * as aq from 'arquero';
import * as fs from 'fs';
import {db} from 'arquero-sql';
// Connect to a database
const pg = new db.Postgres({
  name,
  password,
  host,
  database,
  port
});
// Create a data table
const dt1 = pg.fromArquero(aq.table(...));
// OR
const dt1 = pg.fromCSV(fs.readFileSync('path-to-file.csv'));
// Transform the data table
const dt2 = dt1
  .filter(d => d.Seattle > 200)
  .derive({new_col: d => d.Seattle + 10});
// Observe the transformation result
const output = await dt2.objects();
// OR
dt2.print();