Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/scratchdata/scratchdata
Scratch is a swiss army knife for big data.
https://github.com/scratchdata/scratchdata
bigquery clickhouse data-warehouse duckdb hacktoberfest motherduck olap redshift snowflake
Last synced: about 1 month ago
JSON representation
Scratch is a swiss army knife for big data.
- Host: GitHub
- URL: https://github.com/scratchdata/scratchdata
- Owner: scratchdata
- License: agpl-3.0
- Created: 2023-07-10T20:15:01.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-07-19T17:29:04.000Z (4 months ago)
- Last Synced: 2024-09-27T04:42:53.695Z (about 2 months ago)
- Topics: bigquery, clickhouse, data-warehouse, duckdb, hacktoberfest, motherduck, olap, redshift, snowflake
- Language: Go
- Homepage: https://www.scratchdata.com
- Size: 13.6 MB
- Stars: 1,079
- Watchers: 6
- Forks: 54
- Open Issues: 16
-
Metadata Files:
- Readme: readme.md
- License: LICENSE
Awesome Lists containing this project
- awesome-clickhouse - scratchdata/scratchdata - Scratch Data is a versatile tool for streaming data into and out of analytics databases. (Integrations / ETL and Data Processing)
README
# Scratch Data
Scratch Data is a wrapper that lets you stream data into and
out of your analytics database.
It takes arbitrary JSON as input and lets you perform analytical queries.
## Quickstart
### 1. Run the server
Clone the repo:
```bash
$ git clone [email protected]:scratchdata/scratchdata.git
$ cd scratchdata
```
Start the service:
``` bash
$ go run .
```
With no configuration, this will automatically set up a local DuckDB
database ready for reading and writing.
#### Run with custom config
Create a `config.yaml` file with all of your settings and run:
``` bash
$ go run . config.yaml
```
### 2. Insert JSON data
``` bash
$ curl -X POST "http://localhost:8080/api/data/insert/events?api_key=local" \
--data '{"user": "alice", "event": "click"}'
```
The "events" table and columns are automatically
created.
### 3. Query
```bash
curl -G "http://localhost:8080/api/data/query" \
--data-urlencode "api_key=local" \
--data-urlencode "query=select * from events"
```
## Other Features
### Share Data
You can share data as CSV or JSON by creating "share links".
``` bash
$ curl -X POST "http://localhost:8080/api/data/query/share?api_key=local" \
--data '{"query": "select * from events", "duration": 120}'
```
This will produce a query ID that expires in 120 seconds. From there, send the following link to users:
```
http://localhost:8080/share//data.csv
http://localhost:8080/share//data.json
```
### Copy Data
You can set up multiple databases and copy data between them.
You can run a SQL query against your source database and
Scratch will automatically create a table and insert data into
a destination.
``` bash
$ curl -X POST "http://localhost:8080/api/data/copy?api_key=local" \
--data '{"query": "select * from events", "destination_id": 3, "destination_table": "events"}'
```
## Next Steps
To see the full list of options, look at:
https://docs.scratchdata.com