https://github.com/alpstable/gidari

Transport web data to local/remote storage using Gidari
https://github.com/alpstable/gidari

api-wrapper csv data-science go http mongodb storage

Last synced: 13 days ago
JSON representation

Transport web data to local/remote storage using Gidari

• Host: GitHub
• URL: https://github.com/alpstable/gidari
• Owner: alpstable
• License: apache-2.0
• Created: 2022-07-25T00:45:41.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2023-12-18T22:47:32.000Z (about 2 years ago)
• Last Synced: 2025-09-08T21:17:18.683Z (5 months ago)
• Topics: api-wrapper, csv, data-science, go, http, mongodb, storage
• Language: Go
• Homepage:
• Size: 31.8 MB
• Stars: 29
• Watchers: 1
• Forks: 28
• Open Issues: 8
• Metadata Files:
  • Readme: README.md
  • Contributing: docs/CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Gidari

[![PkgGoDev](https://img.shields.io/badge/go.dev-docs-007d9c?logo=go&logoColor=white)](https://pkg.go.dev/github.com/alpstable/gidari)

![Build Status](https://github.com/alpstable/gidari/actions/workflows/ci.yml/badge.svg)

[![Go Report Card](https://goreportcard.com/badge/github.com/alpstable/gidari)](https://goreportcard.com/report/github.com/alpstable/gidari)

[![Discord](https://img.shields.io/discord/987810353767403550)](https://discord.gg/3jGYQz74s7)



Gidari is a library for batch querying data and persisting the results to local storage.

## Installation

```sh

go get github.com/alpstable/gidari@latest

```

For information on using the CLI, see [here](https://github.com/alpstable/gidari-cli).

## Usage

Gidari supports HTTP and network socket services. There are two ways to use an HTTP service:

1. Iterate over [`http.Response`](https://pkg.go.dev/net/http#Response) data, for pre-defined [`http.Request`](https://pkg.go.dev/net/http#Request)s.

2. Define a writer to concurrently "write" response data for pre-defined `http.Request`s.

See the Go Docs or [Web-to-Storage Examples](#web-to-storage-examples) section for examples.

Network sockets involves subscribing to a socket (such a web socket) and continuously iterating over the data via ReadWriter interface. The results would then be sent to a user-defined ListWriter to be stored. See the Go Docs or [Network Socket Examples](#network-socket-examples) section for examples.

### Authenticating HTTP Requests

| Protocol                                                                                   | Parameters                                                                    | Description                                                                                                                                                                                                       |

|--------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

| [Basic](https://www.w3.org/Protocols/HTTP/1.0/spec.html#BasicAA)                           | Requires a username/email and password                                        | The server will authorize the request only if it can validate the user-ID and password for the protection space of the Request-URI                                                                                |

| [Coinbase](https://docs.cloud.coinbase.com/exchange/docs/authorization-and-authentication) | Requires a key, passphrase and secrete generated in the Coinbase Exchange GUI | The Coinbase API requires a CB-ACCESS-SIGN header that is generated by creating a sha256 HMAC using the base64-decoded secret key on the prehash string from a timestamp + method + requestPath + body combination |

| [Kraken](https://docs.kraken.com/rest/#section/Authentication) | Requires key and secret generated in the Kraken Pro GUI | The Kraken spot API uses a custom authentication algorithm that is based on a combination of API key, nonce, and message signature. The signature is generated using a hash-based message authentication code (HMAC) with SHA-512 as the hash function. |

### Web-to-Storage Examples

| Data Type | Writer                                          | Example                                      | Description                                               |

|-----------|-------------------------------------------------|----------------------------------------------|-------------------------------------------------------------------|

| [CSV](https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml)       | [csvpb](https://github.com/alpstable/csvpb)     | [examples/csvp](examples/csvpb/main.go)      | Use the HTTPService to write web API data to stdout as CSV        |

| [MongoDB Document](https://www.mongodb.com/docs/manual/core/document/)  | [mongopb](https://github.com/alpstable/mongopb) | [exmaples/mongopb](examples/mongopb/main.go) | Use the HTTPService to write web API data to a MongoDB Collection |

### Network Socket Examples

| Type | Writer        | Example                                  | Description                                               |

|------|---------------|------------------------------------------|-----------------------------------------------------------|

| WebSocket |`fmt.Println` | [examples/websocket](examples/websocket) | Subscribe to the Coinbase WebSocket and print the results |

## Contributing

Follow [this guide](docs/CONTRIBUTING.md) for information on contributing.

## Resources

- Public REST APIs from [Postman Documenter](https://documenter.getpostman.com/view/8854915/Szf7znEe)

- Go Gopher artwork by [Victoria Trum](https://www.fiverr.com/victoria_trum?source=order_page_user_message_link)

- The original Go gopher was designed by the awesome [Renee French](http://reneefrench.blogspot.com/)