{"id":19522219,"url":"https://github.com/captaincodeman/afero-datastore","last_synced_at":"2026-04-29T01:04:53.194Z","repository":{"id":66357868,"uuid":"85734454","full_name":"CaptainCodeman/afero-datastore","owner":"CaptainCodeman","description":"AppEngine Datastore filesystem implementation for Afero","archived":false,"fork":false,"pushed_at":"2017-03-22T01:40:55.000Z","size":17,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-06-01T07:05:33.611Z","etag":null,"topics":["appengine","datastore","filesystem","google-cloud","hugo","hugo-blog"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CaptainCodeman.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-21T17:44:12.000Z","updated_at":"2018-09-11T07:53:11.000Z","dependencies_parsed_at":"2023-02-22T03:45:13.450Z","dependency_job_id":null,"html_url":"https://github.com/CaptainCodeman/afero-datastore","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CaptainCodeman/afero-datastore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fafero-datastore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fafero-datastore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fafero-datastore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fafero-datastore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CaptainCodeman","download_url":"https://codeload.github.com/CaptainCodeman/afero-datastore/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fafero-datastore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32405905,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T19:38:08.556Z","status":"ssl_error","status_checked_at":"2026-04-28T19:37:55.688Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["appengine","datastore","filesystem","google-cloud","hugo","hugo-blog"],"created_at":"2024-11-11T00:37:49.043Z","updated_at":"2026-04-29T01:04:53.169Z","avatar_url":"https://github.com/CaptainCodeman.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Afero Datastore FileSystem\n\n_work in progress_\n\nAppEngine Datastore filesystem implementation for [Afero](https://github.com/spf13/afero)\n\nAllows the datastore to be treated as a filesystem by Go code which would, for instance, allow [Hugo](https://github.com/spf13/hugo) blog engine to run in the cloud - add a UI to edit the source files and regenerate the site when done, Afero also provides an Http filesystem which can be used to serve it.\n\nBuild flags are used to allow it to work with the `google.golang.org/appengine/datastore` package (for AppEngine Standard) or remotely via the newer `cloud.google.com/go/datastore` package (for AppEngine Flex, GCE or standalone.\n\n## Usage\n\nDepending on whether you are using AppEngine Standard or Flex / GCS or some other platform, the constructor will vary (this is due to there being different datastore clients for the different platforms).\n\n### AppEngine Standard\n\nCall `NewFileSystem` passing a `context.Context` sourced from the request (`appengine.NewContext(r)`) the namespace and kind to use for datastore entities and either `dfs.Standard` to use raw datastore calls or `dfs.Memcache` to use memcache. Note that due to AppEngine Standard restrictions, a filedsystem instance is only valid while associated with it's request.\n\n```go\nfs := dfs.NewFileSystem(ctx, \"captaincodeman\", \"drafts\", dfs.Memcache)\n```\n\n### AppEngine Flex / GCE\n\nCreate a datastore client with the `google.golang.org/appengine/datastore` package and pass to the `NewFileSystem` method along with a `context.Context` and the namespace and kind to use. Example using the default namespace and\nfile entities stored as `file` in the datastore:\n\n```go\nctx := context.Background()\nserviceAccount := option.WithServiceAccountFile(\"service-account.json\")\nclient, _ := datastore.NewClient(ctx, \"app-name\", serviceAccount)\n\nfs = NewFileSystem(client, \"captaincodeman\", \"drafts\")\n```\n\n## Notes\n\nThe namespacing feature of datastore can be used in a similar way to having separate volumes.\n\nCurrently files are only saved to the datastore so they are limited to just under 1Mb  each (usually plenty for a blog). in future this could be enhanced to use Google Cloud Storage for larger files.\n\nTo avoid too many datastore writes, the datastore entities are only written on close. Multiple filesystem sessions will therefore see inconsistent results. Access to files within the same fileysystem session will use the same file references for consistency with the same approach used as per the Afero memory file system (locks).\n\n## Testing\n\nTo test the standalone version, first download a `service-account.json` file for your project and set the project name in the `file_system_standalone_test.go` file, then run:\n\n    go test -v\n\nTo test AppEngine standard version, install the AppEngine SDK for Go and run:\n\n    goapp test -v\n\n## Known Issues\n\nAfero isn't (yet) compatible with AppEngine Standard due to direct import of \"syscall\" but I have a fork that hides these behind build flags so that it can be used on the platform.\n\nSome of the operations currently don't keep the session cache of files updated (but the tests pass and publishing via Hugo runs fine). Also, closed files should be removed to avoid excessive memory use (more critical if running on the low-memory AppEngine frontend instances).\n\nDatastore is eventually consistent so some operations may not be immediately visible. Files are written on Close, so Directory operations will not always show newly created files (it typically works fine with Hugo).\n\n## Enhancements\n\n* Flags for test configuration (e.g. project name)\n* Fully clean-up tests and use more unique names for standalone namespace + entity to avoid conflicts\n* Improve usefulness of logging\n* Add storage-level compression (i.e. transparent to the caller)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaptaincodeman%2Fafero-datastore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaptaincodeman%2Fafero-datastore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaptaincodeman%2Fafero-datastore/lists"}