Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/asg017/sqlite-http
A SQLite extension for making HTTP requests purely in SQL
https://github.com/asg017/sqlite-http
sqlite sqlite-extension
Last synced: 5 days ago
JSON representation
A SQLite extension for making HTTP requests purely in SQL
- Host: GitHub
- URL: https://github.com/asg017/sqlite-http
- Owner: asg017
- License: mit
- Created: 2021-10-11T21:27:49.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-10-07T03:45:23.000Z (about 1 year ago)
- Last Synced: 2024-10-25T09:31:03.488Z (11 days ago)
- Topics: sqlite, sqlite-extension
- Language: Go
- Homepage:
- Size: 2.5 MB
- Stars: 228
- Watchers: 4
- Forks: 11
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-sqlite - `sqlite-http` - Making HTTP requests and querying responses (People / As Main Database)
README
# sqlite-http
A SQLite extension for making HTTP requests purely in SQL.
- Create GET, POST, and other HTTP requests, like `curl`, `wget`, and `fetch`
- Download response bodies, header, status codes, timing info
- Set rate limits, timeouts
## Usage
```sql
.load ./http0
select http_get_body('https://text.npr.org/');
/*
NPR : National Public Radio
....
*/
```
Query for all custom headers in an endpoint.
```sql
select name, value
from http_headers_each(
http_get_headers('https://api.github.com/')
)
where name like 'X-%';
/*
┌────────────────────────┬────────────────────────────────────┐
│ name │ value │
├────────────────────────┼────────────────────────────────────┤
│ X-Ratelimit-Limit │ 60 │
│ X-Ratelimit-Used │ 8 │
│ X-Content-Type-Options │ nosniff │
│ X-Github-Media-Type │ github.v3; format=json │
│ X-Github-Request-Id │ CCCA:5FDF:1014BC2:10965F9:62F3DE4E │
│ X-Ratelimit-Remaining │ 52 │
│ X-Ratelimit-Resource │ core │
│ X-Frame-Options │ deny │
│ X-Ratelimit-Reset │ 1660152798 │
│ X-Xss-Protection │ 0 │
└────────────────────────┴────────────────────────────────────┘
*/
```
Scrape data from a JSON endpoint.
```sql
select http_get_body('https://api.github.com/repos/sqlite/sqlite')
->> '$.description' as description;
/*
┌───────────────────────────────────────────────┐
│ description │
├───────────────────────────────────────────────┤
│ Official Git mirror of the SQLite source tree │
└───────────────────────────────────────────────┘
*/
```
Pass in specific headers in a request.
```sql
select
value
from json_each(
http_get_body(
'https://api.github.com/issues',
http_headers(
'Authorization', 'token ghp_16C7e42F292c6912E7710c8'
)
)
);
```
## Documentation
See [`docs.md`](./docs.md) for a full API reference.
## Installing
| Language | Install | |
| -------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Python | `pip install sqlite-http` | [](https://pypi.org/project/sqlite-http/) |
| Datasette | `datasette install datasette-sqlite-http` | [](https://datasette.io/plugins/datasette-sqlite-http) |
| Node.js | `npm install sqlite-http` | [](https://www.npmjs.com/package/sqlite-http) |
| Deno | [`deno.land/x/sqlite_http`](https://deno.land/x/sqlite_http) | [](https://deno.land/x/sqlite_http) |
| Ruby | `gem install sqlite-http` |  |
| Github Release | |  |
The [Releases page](https://github.com/asg017/sqlite-http/releases) contains pre-built binaries for Linux amd64, MacOS amd64 (no arm), and Windows.
### As a loadable extension
If you want to use `sqlite-http` as a [Runtime-loadable extension](https://www.sqlite.org/loadext.html), Download the `http0.dylib` (for MacOS), `http0.so` (Linux), or `http0.dll` (Windows) file from a release and load it into your SQLite environment.
> **Note:**
> The `0` in the filename (`http0.dylib`/ `http0.so`/`http0.dll`) denotes the major version of `sqlite-http`. Currently `sqlite-http` is pre v1, so expect breaking changes in future versions.
For example, if you are using the [SQLite CLI](https://www.sqlite.org/cli.html), you can load the library like so:
```sql
.load ./http0
select http_version();
-- v0.0.1
```
Or in Python, using the builtin [sqlite3 module](https://docs.python.org/3/library/sqlite3.html):
```python
import sqlite3
con = sqlite3.connect(":memory:")
con.enable_load_extension(True)
con.load_extension("./http0")
print(con.execute("select http_version()").fetchone())
# ('v0.0.1',)
```
Or in Node.js using [better-sqlite3](https://github.com/WiseLibs/better-sqlite3):
```javascript
const Database = require("better-sqlite3");
const db = new Database(":memory:");
db.loadExtension("./http0");
console.log(db.prepare("select http_version()").get());
// { 'http_version()': 'v0.0.1' }
```
Or with [Datasette](https://datasette.io/), with the "no network" option to limit DDoS attacks:
```
datasette data.db --load-extension ./http0-no-net
```
## See also
- [sqlite-html](https://github.com/asg017/sqlite-html), for parsing and querying HTML using CSS selectors in SQLite (pairs great with this tool)
- [pgsql-http](https://github.com/pramsey/pgsql-http), a similar yet very different HTTP library for POstgreSQL (didn't know about this before I started this, but interestingly enough came up with a very similar API)
- [riyaz-ali/sqlite](https://github.com/riyaz-ali/sqlite), the brilliant Go library that this library depends on
- [nalgeon/sqlean](https://github.com/nalgeon/sqlean), several pre-compiled handy SQLite functions, in C