https://github.com/precog/quasar-datasource-azure

https://github.com/precog/quasar-datasource-azure

Last synced: 6 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/precog/quasar-datasource-azure
• Owner: precog
• License: apache-2.0
• Created: 2018-10-24T10:36:35.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2022-06-13T19:40:48.000Z (about 4 years ago)
• Last Synced: 2024-04-15T03:19:09.300Z (over 2 years ago)
• Language: Scala
• Size: 671 KB
• Stars: 0
• Watchers: 8
• Forks: 9
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# quasar-datasource-azure [![Discord](https://img.shields.io/discord/373302030460125185.svg?logo=discord)](https://discord.gg/QNjwCg6)

## Usage

```sbt

libraryDependencies += "com.precog" %% "quasar-datasource-azure" % 

```

## Configuration

Configuration for the Azure datasource has the following JSON format

```

{

  "container": String,

  "storageUrl": String,

  "format": {

    "type": "json" | "separated-values"

    // for "json"

    "precise": Boolean,

    "variant" "array-wrapped" | "line-delimited"

    // for "separated-values", all strings must be one symbol length

    "header": Boolean,

    // The first char of row break

    "row1": String,

    // The second char of row break, empty string if row break has only one symbol

    "row2": String,

    // Column separator (char)

    "record": String,

    "openQuote": String,

    "closeQuote": String,

    "escape": String

  },

  ["credentials": Object,]

  ["maxQueueSize": Number]

}

```

* `container` the name of the Azure blobstore container to use.

* `storageUrl` the Azure storage URL to use. Typically this will be an URL of the form `https://.blob.core.windows.net/`.

* `format` the format of the resource referred to by `url`. CSV/TSV, array wrapped json and line delimited jsons are supported

* `credentials` (optional, default = empty) Azure credentials to use for access. Object has the following format: `{ "accountName": String, "accountKey": String }`.

* `maxQueueSize` (optional, default = 10) maximum amount of `ByteBuffer`s that can be kept in a queue when downloading a resource.

  When the queue is full, downloading will halt. Downloading will continue again when a `ByteBuffer` is dequeued.

  Usually this value does not need to be overridden, but it can be increased in case downloading halts too often, or decreased to reduce memory use.