{"id":23231142,"url":"https://github.com/alvarios/gcfs","last_synced_at":"2025-04-05T19:24:42.857Z","repository":{"id":134272896,"uuid":"268352117","full_name":"Alvarios/gcfs","owner":"Alvarios","description":"Metadata management tool for file server.","archived":false,"fork":false,"pushed_at":"2020-06-15T10:18:41.000Z","size":301,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-11T17:16:28.256Z","etag":null,"topics":["couchbase","golang","parsing","search-engine"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Alvarios.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-31T19:46:21.000Z","updated_at":"2020-06-15T09:40:46.000Z","dependencies_parsed_at":"2024-06-19T01:28:53.428Z","dependency_job_id":"e1d47703-13e0-4c9b-8482-c85b0834c759","html_url":"https://github.com/Alvarios/gcfs","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alvarios%2Fgcfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alvarios%2Fgcfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alvarios%2Fgcfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alvarios%2Fgcfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Alvarios","download_url":"https://codeload.github.com/Alvarios/gcfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247388012,"owners_count":20930939,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["couchbase","golang","parsing","search-engine"],"created_at":"2024-12-19T02:13:18.325Z","updated_at":"2025-04-05T19:24:42.835Z","avatar_url":"https://github.com/Alvarios.png","language":"Go","readme":"# Go-Couchbase File Server (GCFS)\n\nGCFS is a metadata management tool for file servers.\n\n## Summary\n\n- [About](#about)\n- [GCFS Usage](#gcfs-usage)\n- [Configuration](#configuration)\n    - [Prerequisite](#prerequisite)\n        - [Couchbase instance](#couchbase-instance)\n        - [Server instance](#server-instance)\n    - [Basic setup](#basic-setup)\n    - [Api mode](#api-mode)\n    - [Extended setup](#extended-setup)\n        - [Database](#database)\n        - [Global](#global)\n            - [AutoProvide](#autoprovide)\n            - [Strict](#strict)\n        - [Server](#server)\n            - [Port](#port)\n        - [Metadata](#metadata-advanced)\n            - [Understanding metadata](#understanding-metadata)\n            - [Adding your own metadata](#adding-your-own-metadata)\n            - [Checking for required metadata](#checking-for-required-metadata)\n- [Methods](#methods)\n    - [Insert](#insert)\n    - [InsertF](#insertf)\n        - [Insert flags](#insert-flags)\n            - [AutoProvide flag](#autoprovide-flag)\n            - [Strict flag](#strict-flag)\n            - [Force flag](#force-flag)\n    - [Get](#get)\n    - [Update](#update)\n        - [Update specs](#update-specs)\n            - [Remove](#remove)\n            - [Upsert](#upsert)\n            - [Append](#append)\n    - [Delete](#delete)\n    - [AutoProvide (method)](#autoprovide-method)\n    - [CheckIntegrity](#checkintegrity)\n- [Error handling](#error-handling)\n- [Developers section](#developers-section)\n    - [Setting up test environment](#setting-up-test-environment)\n        - [Setting up Couchbase from scratch](#setting-up-couchbase-from-scratch)\n        - [Using your own Couchbase instance](#using-your-own-couchbase-instance)\n- [Upcoming features](#upcoming-features)\n\n## About\n\nGCFS is a tool for building cloud file systems. It provides default Couchbase\nutilities to store data about your files, located on a distant server.\n\nGCFS is the link between your UI and the distant file. It will provide client\nadequate information for retrieving any files stored on your custom server.\n\n## GCFS Usage\n\nYou can use GCFS in two modes : api mode and methods mode. The first gives you\nmore easy setups, while the second aims for deeper integration in your server\napplication.\n\nYou can use both modes together, depending on your configuration.\n\n## Configuration\n\n### Prerequisite\n\nBefore using GCFS, you need a minimal setup.\n\n#### Couchbase instance\n\nYou need a running Couchbase instance. If you don't want to provide a\ncluster configuration, you need to setup a local Couchbase instance with\nthe following defaults :\n\n- a server running on 127.0.0.1\n- empty username and password to access cluster\n- one bucket named \"metadata\"\n\nTo provide custom cluster configuration, please refer to the [database](#database)\nsection below.\n\n#### Server instance\n\nFor Api mode, you need your go app to run in server mode on a port.\nBy default and for local development, this port is set to 8080.\n\n### Basic setup\n\n```go\npackage my_package\n\nimport \"github.com/Alvarios/gcfs\"\n\nfunc main() {\n    gcfs.Setup(gcfs.Configuration{})\n}\n```\n\nConfiguration interface can be left empty. However, it is required to fill it\nif you want to use the package in api mode.\n\n### Api mode\n\n*coming soon*\n\n### Extended setup\n\n```go\npackage my_package\n\nimport \"github.com/Alvarios/gcfs\"\n\nfunc main() {\n    gcfs.Setup(gcfs.Configuration{\n        Database: gcfs.DbConfig{\n            BucketName: \"metadata\",\n            Address: \"couchbase://127.0.0.1\",\n            Username: \"\",\n            Password: \"\",\n            Bucket: nil,\n        },\n        Global: gcfs.GlobalConfig{\n            AutoProvide: false,\n            Strict: false,\n        },\n        Server: gcfs.ServerConfig{\n            Port: \"8080\",\n        },\n        Metadata: nil,\n    })\n}\n```\n\n#### Database\n\nProvide information to connect to your Couchbase cluster. See [Couchbase official documentation](https://docs.couchbase.com/go-sdk/current/hello-world/start-using-sdk.html#hello-couchbase)\nfor more information.\n\nAlternatively, you can do the setup on your own and pass a Bucket pointer\ninto the Database configuration. You can then ignore other configuration\narguments :\n\n```go\npackage my_package\n\nimport (\n    \"github.com/Alvarios/gcfs\"\n    \"github.com/couchbase/gocb/v2\"\n)\n\nfunc main() {\n    var myClust *gocb.Cluster\n    var myBucket *gocb.Bucket\n\n    myClust = gocb.Connect(...) // Enter your cluster configuration here.\n    myBucket = myClust.Bucket(bucketName)\n\n    gcfs.Setup(gcfs.Configuration{\n        Database: gcfs.Database{\n            Bucket: myBucket,\n        },\n    })\n}\n```\n\n#### Global\n\nConfiguration of flags to set the package default behaviors.\n\n##### AutoProvide\n\nProvides a default behavior for the [insert method](#insert-method). Default is\nfalse.\n\nAutoProvide mode allows insert method to automatically fill up some metadata\nwhen not specified in the newly created document.\n\nOnly 2 fields currently support the autofill mode : `general.creation_time` and\n`general.modification_time`. Both are automatically set to current date.\n\nFor more detail about metadata, please refer to [this section](#metadata).\n\n##### Strict\n\nStrict mode only enables required metadata for insertion and update of a\ndocument. It will reject any attempt to add unauthorized metadata.\n\n#### Server\n\nProvide server configuration for Api mode.\n\n##### Port\n\nThe port to run the server on. Running a server on a port from any domain\nmakes it accessible via `domain:port` url. Domain is `localhost` for local\nserver, or `127.x.x.x` (range from `127.0.0.1` to `127.255.255.254`).\n\n#### Metadata (advanced)\n\n##### Understanding metadata\n\nGCFS is a metadata utility, which stores textual information about a distant\nfile. Metadata is used to represent a file without having to load it.\n\nAs a service, GCFS provides some pre-configured metadata. This metadata is\npresent on every file handled by GCFS, and serves as a standard representation.\n\nSome pre-configured metadata is required, while the rest can be left blank.\nThey are listed below.\n\n| Metadata | Required | AutoFillable | Type | Description |\n| :--- | :--- | :--- | :--- | :--- |\n| url | true | - | string | url pointing to the actual file. |\n| general.name | true | - | string | name of the file. |\n| general.format | true | - | string | format of the file. |\n| general.size | - | - | int | size of the file in bytes. |\n| general.creation_time | true | true | uint64 | date of the file creation. **(1)** |\n| general.modification_time | - | true | uint64 | date of the file last modification. **(1)** |\n\n\u003e **(1)** In milliseconds since January 1st, 1970, 00:00:00 UTC. Be careful as this\nformat is different from the one used by Go and UNIX systems, calculated from\nthe same date but using nanoseconds. The reason we use milliseconds is to keep\nconsistent with javascript and other web standards (since Couchbase is a JSON\ndocument database system).\n\n##### Adding your own metadata\n\nAs Couchbase works with the very permissive JSON format, you are totally free\nto add any metadata to your file, as long as you leave the default provided\nones. \n\nYou can even add some metadata under the general key. For example, a general.author\nto keep track of the file owner. GCFS treats a document like a basic interface{},\nso you have total freedom.\n\nNow you may want to work with your own standards. Your system will grow up in\na specific direction, and maybe you'd like to require some more metadata. You\ncan do a check on your own in methods mode, but that's time and processor\ncostly. Instead, GCFS provides you a prebuilt and efficient solution, that\nalso works in Api mode.\n\n##### Checking for required metadata\n\nWhen adding a new document, GCFS will perform an integrity check to ensure\nevery required field is present with a correctly typed value.\n\nYou can add some required fields to check, with the Metadata field inside the\nConfiguration interface.\n\n```go\npackage my_package\n\nimport \"github.com/Alvarios/gcfs\"\n\nfunc main() {\n    gcfs.Setup(gcfs.Configuration{\n        Metadata: map[string]interface{}{\n            \"general\": map[string]interface{}{\n                \"author\": \"string\",\n            },\n            \"version\": \"string\",\n        },\n    })\n}\n```\n\nFor example, the above configuration will force every document to have a\n`general.author` and a `version` field, both of type string. If a document\nmiss one of the above fields, it will be refused and the insertion will\nreturn an error.\n\nMetadata consist of key-value pairs : key points to a field, and value is a\nstring representing a go type. To access nested fields, declare the parent\nfield as a hardcoded `map[string]interface{}`, then every key declared\ninside will be considered as a child of its parent.\n\n## Methods\n\nGCFS provides methods for interacting with your metadata. They wrap the\ngocb methods and add some useful checks and syntax.\n\n### Insert\n\n`fileId, err := gcfs.Insert(fileMetadata, fileId)`\n\nInsert metadata and returns the id of the generated document.\n\n*arguments*\n\n| Name | Type | Description |\n| :--- | :--- | :--- |\n| fileMetadata | map[string]interface{} **(1)** | metadata document. |\n| fileId | string | optional file id. |\n\n**(1)** The map has to contain the required metadata, if not autofilled.\n\n```go\npackage my_package\n\nimport (\n    \"github.com/Alvarios/gcfs\"\n    \"log\"\n)\n\nfunc main() {\n    // Works with AutoFill = true.\n    data := map[string]interface{}{\n        \"url\": \"/path/to/my/file\",\n        \"general\": gcfs.GeneralMetadata{ // You can also pass a map[string]interface{}, as long as it contains every required field.\n            Name: \"my awesome file\",\n            Format: \"txt\",\n        },\n        \"another key\": 123456,\n    }\n    \n    // Leave id blank will generate a unique id.\n    fileId, err := gcfs.Insert(data, \"\")\n\n    log.Println(err == (*gcfs.Error)(nil)) // true\n\n    // Fails with missing url error.\n    data := map[string]interface{}{\n        \"general\": map[string]interface{}{\n            \"name\": \"my awesome file\",\n            \"format\": \"txt\",\n        },\n        \"another key\": 123456,\n    }\n    \n    fileId, err = gcfs.Insert(data, \"\")\n\n    log.Println(err == (*gcfs.Error)(nil)) // false\n}\n```\n\n*return value*\n\n| Name | Type | Description |\n| :--- | :--- | :--- |\n| fileId | string | id pointing to the newly created document. |\n| err | [Error object](#error-handling) | - |\n\n### InsertF\n\nShortcut for InsertFlagged. Allows to set custom flag and ignore Global\nconfiguration for a specific action.\n\n`fileId, err := gcfs.InsertF(fileMetadata, fileId, flags)`\n\nInsert metadata and returns the id of the generated tuple.\n\n*arguments*\n\n| Name | Type | Description |\n| :--- | :--- | :--- |\n| fileMetadata | map[string]interface{} | file metadata object. |\n| fileId | string | optional file id. |\n| flags | [gcfs.InsertFlags](#insert-flags) | override default configuration. |\n\n*return value*\n\n| Name | Type | Description |\n| :--- | :--- | :--- |\n| fileId | string | id pointing to the newly created document. |\n| err | [Error object](#error-handling) | - |\n\n#### Insert Flags\n\n```go\npackage my_package\n\nimport (\n    \"github.com/Alvarios/gcfs\"\n    \"log\"\n)\n\nfunc main() {\n    // Works with AutoFill = true.\n    data := map[string]interface{}{\n        \"url\": \"/path/to/my/file\",\n        \"general\": gcfs.GeneralMetadata{ // You can also pass a map[string]interface{}, as long as it contains every required field.\n            Name: \"my awesome file\",\n            Format: \"txt\",\n        },\n        \"another key\": 123456,\n    }\n    \n    // Leave id blank will generate a unique id.\n    fileId, err := gcfs.InsertF(data, \"\", gcfs.InsertFlags{\n        AutoProvide: false,\n        Strict: false,\n        Force: false,\n    })\n\n    log.Println(err == (*gcfs.Error)(nil)) // true\n}\n```\n\n##### AutoProvide flag\n\nAutofill the missing metadata when possible.\n\n##### Strict flag\n\nToggle strict mode for the current request.\n\n##### Force flag\n\nForce document to be inserted in the database by skipping integrity checks.\nThe only way to fail in Force mode is to try an update with a non JSON marshable\ninterface.\n\n### Get\n\n`fileMetadata, err := gcfs.Get(fileId)`\n\nRetrieve metadata from database.\n\n*arguments*\n\n| Name | Type | Description |\n| :--- | :--- | :--- |\n| fileId | string | id pointing to the document. |\n\n*return value*\n\n| Name | Type | Description |\n| :--- | :--- | :--- |\n| fileMetadata | map[string]interface{} | file metadata object. |\n| err | [Error object](#error-handling) | - |\n\n### Update\n\n`timestamp, err := gcfs.Update(fileId, updateSpecs)`\n\nPerform a partial update of a document.\n\n*arguments*\n\n| Name | Type | Description |\n| :--- | :--- | :--- |\n| fileId | string | id pointing to the document. |\n| fileMetadata | [Update specs](#update-specs) | - |\n\n*return value*\n\n| Name | Type | Description |\n| :--- | :--- | :--- |\n| timestamp | uint64 | timestamp of the update. |\n| err | [Error object](#error-handling) | - |\n\n#### Update specs\n\nGCFS provides an easy, declarative way to perform a partial update of your\nmetadata, using [gocb MutateIn](https://docs.couchbase.com/go-sdk/2.1/howtos/subdocument-operations.html#mutating) function.\n\n```go\nupdateSpecs := gcfs.UpdateSpec{\n    Remove: []string{\"key1\", \"key2\", \"key4.key4-2\"},\n    Upsert: map[string]interface{}{\n        \"key3\": \"new value\",\n        \"key4\": map[string]interface{}{\n            \"key4-1\": \"new value\",\n        },\n    },\n    Append: map[string]interface{}{\n        \"array-key\": [\"newValue1\", \"newValue2\"],\n    },\n    Force: false,\n}\n```\n\n##### Remove\n\nA list of keys to remove from the document. Support the short dot syntax\nfor nested keys.\n\n\u003e 💡 Tip : remove specs are forbidden in strict mode, when Force flag is set\nto false.\n\n##### Upsert\n\nUpdate a list of values. Each value in the given map will replace the\nequivalent one in the original document. If a path doesn't exist, it will be\ncreated.\n\nYou can also use short dot syntax to access nested keys.\n\n```go\nupdateSpecs := gcfs.UpdateSpec{\n    Upsert: map[string]interface{}{\n        \"key3\": \"new value\",\n        \"key4.key4-1\": \"new value\",\n    },\n}\n```\n\n##### Append\n\nAppend a list of values to an array key.\n\n\u003e 💡 Tip : you can declare a non array value to append a single value. An\n array value will always be treaten with the `HasMultiple` flag set to true\n (see [gocb specs](https://docs.couchbase.com/go-sdk/2.1/howtos/subdocument-operations.html#array-append-and-prepend)).\n Add a nested array if you want to append an array as an array, and not a list\n of values.\n\n### Delete\n \n`err := gcfs.Delete(fileId)`\n\nDelete metadata from database. Only returns an error.\n\n*arguments*\n\n| Name | Type | Description |\n| :--- | :--- | :--- |\n| fileId | string | Id pointing to the document. |\n\n*return value*\n\n| Name | Type | Description |\n| :--- | :--- | :--- |\n| err | [Error object](#error-handling) | - |\n\n### AutoProvide (method)\n\n`autoProvidedData, err = gcfs.AutoProvide(fileMetadata)`\n\nAuto fill metadata with default values. More details in the [Metadata section](#metadata-advanced).\n\n### CheckIntegrity\n\n`coreMetadata, err := gcfs.CheckIntegrity(fileMetadata)`\n\nCheck integrity of a dataset.\n\n\u003e 💡 Tip : coreMetadata represents the default provided metadata.\n More details in the [Metadata section](#metadata-advanced).\n\n## Error handling\n \nGCFS returns a pointer to an error object adapted to web servers. This error\nobject is retrieved from [kushuh-go-utils](https://www.github.com/Alvarios/kushuh-go-utils) package.\n\n| Key | Type | Description |\n| :--- | :--- | :--- |\n| Code | int | The http status of the error. |\n| Message | string | Describes the nature of the error. |\n \n \u003e 💡 Tip : as a custom error object, `err == nil` wont work to check for errors.\nInstead, use the following guard clause : `err == (*gcfsresponses.Error)(nil)`.\n\n## Developers section\n\nThis section provides you information if you want to participate in the project.\n\n### Setting up test environment\n\nOnce you pulled the project, navigate through your terminal to the root folder\n(usually `gcfs/`). To launch the tests, run :\n \n `go test -v ./...`\n\nTest command will fail if you don't have a little setup environment.\n\nFirst of all, you need a Couchbase instance running. You can use one of\nyour existing instances. If so, you can skip the following section and go\n[here](#using-your-own-couchbase-instance).\n\n#### Setting up Couchbase from scratch\n\nIf you don't have any running instance, the easiest way is to download the\n[Couchbase Server from source](https://www.couchbase.com/downloads?family=couchbase-server).\n\n\u003cimg src=\"https://github.com/Alvarios/gcfs/blob/master/resources/images/Capture%20d%E2%80%99%C3%A9cran%202020-06-14%20%C3%A0%2016.38.03.png\" width=\"800\" alt=\"download couchbase\"\u003e\n\nYou can go for the free community edition. Select your platform and make sure\nyou go for the last version (marked as current). Then follow the instruction.\n\nAt the end, you should have a Couchbase service running on your computer.\nConnect to `http://127.0.0.1:8091/ui/index.html` from any web browser.\n\nAgain, follow the instruction to setup your local Cluster. You can leave\nUsername and Password blank, since we don't need any security here. Setup\nyour Cluster on localhost (`127.0.0.1`).\n\nFinally, create a single bucket named 'metadata'. You can leave every parameter\nto default.\n\n#### Using your own Couchbase instance\n\nIf you want to use a pre-configured Couchbase instance, or just don't want to\ncomply to default configuration above, then you'll need to set some\nenvironment variables before running any test, to tell GCFS where to look\nfor.\n\nYou don't need to provide each environment variables : only set one when\nits value is different in your configuration, from the default configuration\nbelow.\n\n| ENV | Default | Description |\n| :--- | :--- | :--- |\n| GCFS_TEST_ADDRESS | \"couchbase://127.0.0.1\" | address of your server. |\n| GCFS_TEST_USERNAME | \"\" | username to access the cluster. |\n| GCFS_TEST_PASSWORD | \"\" | password to access the cluster. |\n| GCFS_TEST_BUCKETNAME | \"metadata\" | bucket for running tests. |\n\nFor example, if you have a local instance with some credentials, but you\nrun on localhost with a metadata bucket, then you only need to export the\nfollowing before running any test :\n\n`export GCFS_TEST_USERNAME=\"admin\" GCFS_TEST_PASSWORD=\"123456\"`\n\n## Upcoming features\n\n- **Configuration \u003e Metadata**\n    - Add a short syntax for nested fields \"general.author\"\n    - Ignore spaces and case for type declaration\n- **Methods**\n    - Search method and api with nefts package\n \n ## Copyright\n 2020 Alvarios - [MIT license](https://github.com/Alvarios/gcfs/blob/master/LICENSE)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falvarios%2Fgcfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falvarios%2Fgcfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falvarios%2Fgcfs/lists"}