Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/captaincodeman/afero-datastore
AppEngine Datastore filesystem implementation for Afero
https://github.com/captaincodeman/afero-datastore
appengine datastore filesystem google-cloud hugo hugo-blog
Last synced: 14 days ago
JSON representation
AppEngine Datastore filesystem implementation for Afero
- Host: GitHub
- URL: https://github.com/captaincodeman/afero-datastore
- Owner: CaptainCodeman
- Created: 2017-03-21T17:44:12.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2017-03-22T01:40:55.000Z (almost 8 years ago)
- Last Synced: 2024-12-18T05:39:50.285Z (about 1 month ago)
- Topics: appengine, datastore, filesystem, google-cloud, hugo, hugo-blog
- Language: Go
- Size: 16.6 KB
- Stars: 3
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# Afero Datastore FileSystem
_work in progress_
AppEngine Datastore filesystem implementation for [Afero](https://github.com/spf13/afero)
Allows the datastore to be treated as a filesystem by Go code which would, for instance, allow [Hugo](https://github.com/spf13/hugo) blog engine to run in the cloud - add a UI to edit the source files and regenerate the site when done, Afero also provides an Http filesystem which can be used to serve it.
Build flags are used to allow it to work with the `google.golang.org/appengine/datastore` package (for AppEngine Standard) or remotely via the newer `cloud.google.com/go/datastore` package (for AppEngine Flex, GCE or standalone.
## Usage
Depending on whether you are using AppEngine Standard or Flex / GCS or some other platform, the constructor will vary (this is due to there being different datastore clients for the different platforms).
### AppEngine Standard
Call `NewFileSystem` passing a `context.Context` sourced from the request (`appengine.NewContext(r)`) the namespace and kind to use for datastore entities and either `dfs.Standard` to use raw datastore calls or `dfs.Memcache` to use memcache. Note that due to AppEngine Standard restrictions, a filedsystem instance is only valid while associated with it's request.
```go
fs := dfs.NewFileSystem(ctx, "captaincodeman", "drafts", dfs.Memcache)
```
### AppEngine Flex / GCE
Create a datastore client with the `google.golang.org/appengine/datastore` package and pass to the `NewFileSystem` method along with a `context.Context` and the namespace and kind to use. Example using the default namespace and
file entities stored as `file` in the datastore:
```go
ctx := context.Background()
serviceAccount := option.WithServiceAccountFile("service-account.json")
client, _ := datastore.NewClient(ctx, "app-name", serviceAccount)
fs = NewFileSystem(client, "captaincodeman", "drafts")
```
## Notes
The namespacing feature of datastore can be used in a similar way to having separate volumes.
Currently files are only saved to the datastore so they are limited to just under 1Mb each (usually plenty for a blog). in future this could be enhanced to use Google Cloud Storage for larger files.
To avoid too many datastore writes, the datastore entities are only written on close. Multiple filesystem sessions will therefore see inconsistent results. Access to files within the same fileysystem session will use the same file references for consistency with the same approach used as per the Afero memory file system (locks).
## Testing
To test the standalone version, first download a `service-account.json` file for your project and set the project name in the `file_system_standalone_test.go` file, then run:
go test -v
To test AppEngine standard version, install the AppEngine SDK for Go and run:
goapp test -v
## Known Issues
Afero isn't (yet) compatible with AppEngine Standard due to direct import of "syscall" but I have a fork that hides these behind build flags so that it can be used on the platform.
Some of the operations currently don't keep the session cache of files updated (but the tests pass and publishing via Hugo runs fine). Also, closed files should be removed to avoid excessive memory use (more critical if running on the low-memory AppEngine frontend instances).
Datastore is eventually consistent so some operations may not be immediately visible. Files are written on Close, so Directory operations will not always show newly created files (it typically works fine with Hugo).
## Enhancements
* Flags for test configuration (e.g. project name)
* Fully clean-up tests and use more unique names for standalone namespace + entity to avoid conflicts
* Improve usefulness of logging
* Add storage-level compression (i.e. transparent to the caller)