https://github.com/leafo/cloud_storage
A Lua library for communicating with Google Cloud Storage
https://github.com/leafo/cloud_storage
Last synced: 2 months ago
JSON representation
A Lua library for communicating with Google Cloud Storage
- Host: GitHub
- URL: https://github.com/leafo/cloud_storage
- Owner: leafo
- Created: 2012-11-05T16:04:50.000Z (about 13 years ago)
- Default Branch: master
- Last Pushed: 2025-09-03T19:15:26.000Z (4 months ago)
- Last Synced: 2025-09-03T21:23:02.369Z (4 months ago)
- Language: Lua
- Size: 106 KB
- Stars: 39
- Watchers: 9
- Forks: 13
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# `cloud_storage`
A library for connecting to [Google Cloud Storage](https://cloud.google.com/products/cloud-storage) through Lua.
## Tutorial
You can learn more about authenticating with Google Cloud Storage here:
.
Here's a quick guide to get you started:
The easiest way use this library is to create a service account for your
project. You'll need to generate a private key and store it alongside your
configuration.
Go to the cloud console: . Enable Cloud
Storage if you haven't done so already. You may also need to enter billing
information.
Navigate to **IAM & admin** » **Service accounts**, located on the sidebar.
Click **Create service account**.
Select **Furnish a new private key** and select **JSON** as the type. After
creating the service account your browser will download a `.json` file. Store
that securely, since it grants access to your project.
The `.json` is used to create a new API client with this library:
```lua
local google = require "cloud_storage.google"
local storage = google.CloudStorage:from_json_key_file("path/to/my-key.json")
-- you can now use any of the methods on the storage object
local files = assert(storage:get_bucket("my_bucket"))
```
## Using a p12/pem secret key
Use these directions only if you have a key created with the `P12` type.
The private key downloaded from the Cloud Console is a `.p12` file. The key
uses a hard coded password `notasecret`.
In order to use it, it must be converted to a `.pem` file. Run the following
command (replacing `key.p12` and `key.pem` with the input filename and desired
output filename). Enter `notasecret` for the password.
```bash
openssl pkcs12 -in key.p12 -out key.pem -nodes -clcerts
```
One more piece of information is needed: the service account email address.
You'll find it labeled **Service account ID** on the service account list. It
might look something like `cloud-storage@my-project.iam.gserviceaccount.com`.
You can now connect to the API. Create a new client like this:
```lua
local oauth = require "cloud_storage.oauth"
local google = require "cloud_storage.google"
-- replace with your service account ID
local o = oauth.OAuth("cloud-storage@my-project.iam.gserviceaccount.com", "path/to/key.pem")
-- use your id as the second argument, everything before the @ in your service account ID
local storage = google.CloudStorage(o, "cloud-storage")
local files = assert(storage:get_bucket("my_bucket"))
```
## HTTP Client Customization
By default, this library uses `socket.http` for making HTTP requests. You can provide a custom HTTP client that supports the LuaSocket interface:
```lua
local http = require("cloud_storage.http")
-- Replace with your custom HTTP client
-- Must implement a request() function compatible with LuaSocket
http.set(require("lapis.http"))
-- Now all future requests will use the `request` function in your http client...
```
This allows for custom timeout handling, proxy configuration, or mock clients for testing.
## Reference
### Module `cloud_storage.oauth`
This module contains the OAuth implementation used by the storage API.
```lua
local oauth = require "cloud_storage.oauth"
```
#### `ouath_instance = oauth.OAuth(service_email, path_to_pem_file=nil)`
Create a new OAuth object. Handles OAuth authenticated requests.
It's not necessary to use this module directly if loading private key from a
`.json` file.
### Module `cloud_storage.google`
Communicates with the Google cloud storage API.
```lua
local google = require "cloud_storage.google"
```
#### Error handling
Any methods that fail to execute will return `nil`, an error message, and an
object that represents the error. Successful responses will return a Lua table
containing the details of the operation.
#### `storage = google.CloudStorage(ouath_instance, project_id)`
Create a new instance of a storage object from an OAuth instance. Note that the
`from_json_key_file` interface is a simpler way of constructing the storage
interface.
```lua
local oauth = require "cloud_storage.oauth"
local google = require "cloud_storage.google"
local o = oauth.OAuth("me@my-project.iam.gserviceaccount.com", "key.pem")
local storage = google.CloudStorage(o, "111111111111")
```
#### `storage = google.CloudStorage:from_json_key_file(json_key_file_name)`
Create a new instance of a storage object from a `.json` private key file
generated by the Google Cloud Console.
```lua
local google = require "cloud_storage.google"
local storage = google.CloudStorage:from_json_key_file("my-storage-auth.json")
```
#### `storage:get_service()`
#### `storage:get_bucket(bucket)`
#### `storage:get_file(bucket, key)`
#### `storage:delete_file(bucket, key)`
#### `storage:head_file(bucket, key)`
#### `storage:copy_file(source_bucket, source_key, dest_bucket, dest_key, options={})`
Options:
* `acl`: value to use for `x-goog-acl`, defaults to `public-read`
* `headers`: table of additional headers to include in request
#### `storage:compose(bucket, key, source_keys, options={})`
Composes a new file from multiple source files in the same bucket by
concatenating them.
`source_keys` is an array of keys as strings or an array of key declarations as
tables.
The table format for a source key is structured like this:
```lua
{
name = "file.png",
-- optional fields:
generation = "1361471441094000",
if_generation_match = "1361471441094000",
}
```
Options:
* `acl`: value to use for `x-goog-acl`, defaults to `public-read`
* `mimetype`: sets `Content-type` header for the composed file
* `headers`: table of additional headers to include in request
#### `storage:put_file_string(bucket, key, data, opts={})`
> **Note:** this API previously had `key` as an option but it was moved to
> second argument
Uploads the string `data` to the bucket at the specified key.
Options include:
* `mimetype`: sets `Content-type` header for the file, defaults to not being set
* `acl`: sets the `x-goog-acl` header for file, defaults to `public-read`
* `headers`: an optional array table of any additional headers to send
```lua
storage:put_file_string("my_bucket", "message.txt", "hello world!", {
mimetype = "text/plain",
acl = "private"
})
```
#### `storage:put_file(bucket, fname, opts={})`
Reads `fname` from disk and uploads it. The key of the file will be the name of
the file unless `opts.key` is provided. The mimetype of the file is guessed
based on the extension unless `opts.mimetype` is provided.
```lua
storage:put_file("my_bucket", "source.lua", {
mimetype = "text/lua"
})
```
All the same options from `put_file_string` are available for this method.
> **Note:** This method is currently inefficient . The whole file is loaded
> into memory and sent at once in the request. An alternative implementation
> will be added in the future. Open a ticket if you need it.
#### `storage:start_resumable_upload(bucket, key, options={})`
Options:
* `acl`: value to use for `x-goog-acl`, defaults to `public-read`
* `mimetype`: sets `Content-type` header for the composed file
* `headers`: table of additional headers to include in request
#### `storage:bucket_url(bucket, opts={})`
Returns full URL for a bucket
Options:
* `subdomain`: use the `{bucket}.domain` format instead of default `domain/{bucket}` format
* `scheme`: protocol of returned URL, default `https`
#### `storage:bucket_url(bucket, opts={})`
Returns full URL for a key on a bucket. Supports the same options as `storage:bucket_url` method.
```lua
storage:file_url("my-bucket", "pics/leafo.png") --> "https://commondatastorage.googleapis.com/my-bucket/pics/leafo.png"
storage:file_url("my-bucket", "pics/leafo.png", {
scheme = "http",
subdomain = true,
}) --> "http://my-bucket.commondatastorage.googleapis.com/pics/leafo.png"
```
Options:
* `subdomain`: use the `{bucket}.domain` format instead of default `domain/{bucket}` format
* `scheme`: protocol of returned URL, default `https`
#### `storage:signed_url(bucket, key, expiration, opts={})`
Creates a temporarily URL for downloading an object regardless of it's ACL.
`expiration` is a Unix timestamp in the future, like one generated from
`os.time()`.
```lua
print(storage:signed_url("my_bucket", "message.txt", os.time() + 100))
```
Options:
* `headers`: table of additional `x-goog` headers to embed into the signature
* `verb`: HTTP verb to embed into the signature, deafult `GET`
* `scheme`: protocol of returned URL, default `https`
#### `storage:upload_url(bucket, key, opts={})`
Generate a signed URL for browser-based upload.
Options:
* `content_disposition`
* `filename` If provided, will append to content disposition as `filename=` (and set content disposition to `attachment` if none is provided)
* `acl` default `project-private`
* `success_action_redirect`
* `expires` unix timestamp. Deafult: 1 hour from now
* `size_limit`
* `scheme` protocol of returned URL, default `https`
#### `storage:put_file_acl(bucket, key, acl)`
## Changelog
### Future updates
All future changelogs will be post on github release notes: https://github.com/leafo/cloud_storage/releases
### `1.0.0` Mar 31, 2018
* **Changed** `put_file_string` now takes `key` as second argument, instead of within options table
* Replace `luacrypto` with `luaossl`
* Add support for `json` private keys
* Better error handling for all API calls. (Failures return `nil`, error message, and error object)
* Add `copy_file` and `compose` methods
* Add `upload_url` method to create upload URLs
* Add `start_resumable_upload` method for creating resumable uploads
* Headers are canonicalized and sorted to make them consistent between calls
* Fix bug where special characters in key were not being URL encoded
* Fix bug where some special characters were not being encoded for signed URLs
* Rewrite documentation tutorial
### `0.1.0` Sep 29, 2013
* Initial release
[0]: https://developers.google.com/storage/docs/accesscontrol