{"id":19402656,"url":"https://github.com/cscfi/sda-filesystem","last_synced_at":"2025-04-24T07:32:15.820Z","repository":{"id":40486769,"uuid":"415794190","full_name":"CSCfi/sda-filesystem","owner":"CSCfi","description":"FUSE implementation for working with files in SD Desktop https://research.csc.fi/sensitive-data","archived":false,"fork":false,"pushed_at":"2024-08-29T18:03:23.000Z","size":2602,"stargazers_count":4,"open_issues_count":8,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-08-29T20:03:43.116Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSCfi.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-11T05:48:42.000Z","updated_at":"2024-08-29T18:03:28.000Z","dependencies_parsed_at":"2023-02-16T06:31:27.721Z","dependency_job_id":"7a2c5010-099a-4bf4-accf-d364def50846","html_url":"https://github.com/CSCfi/sda-filesystem","commit_stats":{"total_commits":464,"total_committers":8,"mean_commits":58.0,"dds":0.3469827586206896,"last_synced_commit":"5d8fb87dc10ac5f320afc4d613ab2da85e0bbba1"},"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fsda-filesystem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fsda-filesystem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fsda-filesystem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fsda-filesystem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSCfi","download_url":"https://codeload.github.com/CSCfi/sda-filesystem/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223945245,"owners_count":17229600,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T11:25:00.527Z","updated_at":"2024-11-10T11:25:01.109Z","avatar_url":"https://github.com/CSCfi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SDA-Filesystem / Data Gateway\n\n[![Linting go code](https://github.com/CSCfi/sda-filesystem/actions/workflows/linting.yml/badge.svg)](https://github.com/CSCfi/sda-filesystem/actions/workflows/linting.yml)\n[![Unit Tests](https://github.com/CSCfi/sda-filesystem/actions/workflows/unittest.yml/badge.svg)](https://github.com/CSCfi/sda-filesystem/actions/workflows/unittest.yml)\n[![Coverage Status](https://coveralls.io/repos/github/CSCfi/sda-filesystem/badge.svg?branch=master)](https://coveralls.io/github/CSCfi/sda-filesystem?branch=master)\n\n**This project has been rebranded as Data Gateway**\n\nData Gateway makes use of the:\n\n- [SD Connect Proxy API](docs/SD-Connect-API.md) \n- [SD Apply/SD Submit Download API](docs/SD-Submit-API.md) \n\nIt builds a FUSE (Filesystem in Userspace) layer and uses Airlock to export files to SD Connect. Software currently supports Linux, macOS and Windows for:\n- [Graphical User Interface](#graphical-user-interface)\n- [Command Line Interface](#command-line-interface)\n\nBinaries are built on each release for all supported Operating Systems.\n\n### Requirements\n\nGo version 1.21\n\nSet these environment variables before running the application:\n- `FS_SD_CONNECT_API` - API for SD-Connect\n- `FS_SD_SUBMIT_API` – a comma-separated list of APIs for SD Apply/SD Submit\n- `SDS_ACCESS_TOKEN` - a JWT for authenticating to the SD APIs\n- `FS_CERTS` - path to a file that contains certificates required by SD Connect, SD Apply/SD Submit, and SDS AAI \n\nOptional envronment variables:\n\n- `CSC_USERNAME` - username for SDA-Filesystem\n- `CSC_PASSWORD` - password for SDA-Filesystem and Airlock CLI\n\nFor test environment follow instructions at https://gitlab.ci.csc.fi/sds-dev/local-proxy\n\n## Graphical User Interface\n\n###  Dependencies\n\n`cgofuse` and its [dependencies on different operating systems](https://github.com/billziss-gh/cgofuse#how-to-build).\n\nInstall [Wails](https://wails.io/docs/gettingstarted/installation) and its dependencies.\n\nInstall [pnpm](https://pnpm.io/installation)\n\n### Build and Run\n\nBefore running/building the repository for the first time, generate the frontend assests by running:\n```bash\npnpm install --prefix frontend\npnpm --prefix frontend run build\n```\n\nTo run in development mode:\n```bash\ncd cmd/gui\nwails dev\n``` \n\nTo build for production:\n```bash\ncd cmd/gui\n\n# For Linux and macOS\nwails build -upx -trimpath -clean -s\n\n# For Windows\nwails build -upx -trimpath -clean -s -webview2=embed\n```\n\n### Deploy\n\nSee [Linux setup](docs/linux-setup.md).\n\n## Command Line Interface\n\nTwo command line binaries are released, one for SDA-Filesystem and one for Airlock. \n\n### SDA-Fileystem\n\nThe CLI binary will require a username and password for accessing the SD-Connect Proxy API. Username is given as input. Password is either given as input or in an environmental variable.\n\n#### Build and Run\n```bash\ngo build -o ./go-fuse ./cmd/fuse/main.go\n```\nTest install.\n```bash\n./go-fuse -help                        \nUsage of ./go-fuse:\n  -alsologtostderr\n    \tlog to standard error as well as files\n  -http_timeout int\n    \tNumber of seconds to wait before timing out an HTTP request (default 20)\n  -log_backtrace_at value\n    \twhen logging hits line file:N, emit a stack trace\n  -log_dir string\n    \tIf non-empty, write log files in this directory\n  -loglevel string\n    \tLogging level. Possible values: {debug,info,warning,error} (default \"info\")\n  -logtostderr\n    \tlog to standard error instead of files\n  -mount string\n    \tPath to Data Gateway mount point\n  -project string\n    \tSD Connect project if it differs from that in the VM\n  -sdapply\n      Connect only to SD Apply\n  -stderrthreshold value\n    \tlogs at or above this threshold go to stderr\n  -v value\n    \tlog level for V logs\n  -vmodule value\n    \tcomma-separated list of pattern=N settings for file-filtered logging\n\n```\nExample run: `./go-fuse -mount=$HOME/ExampleMount` will create the FUSE layer in the directory `$HOME/ExampleMount` for both 'SD Connect' and 'SD Apply'.\n\n#### User input\n\nUser can update the filesystem by inputting the command `update`. This requires that no files inside the filesystem are being used. Update also clears cache. As a result of this operation, new files may be added and some old ones removed.\n\nThe filesystem can be also updated programatically with the `SIGUSR2` signal.\n\nTo update filesystem on bash in SD Desktop:\n```bash\n# Update CLI version\nkill -s SIGUSR2 $(pgrep go-fuse)\n\n# Update GUI version\nkill -s SIGUSR2 $(pgrep sda-fuse)\n```\n\nIf the user wants to update particular SD Connect files inside the filesystem, the user can input command `clear \u003cpath\u003e`. `\u003cpath\u003e` is the path to the file/folder that the user wishes to update. `\u003cpath\u003e` must at least contain a bucket, i.e. `SD-Connect/project/bucket` or `SD-Connect/project/bucket/file` would be acceptable paths, but not, e.g., `SD-Connect/project`. If the user gives a path to a folder, all files inside this folder are updated but no files are added or removed. This operation clears the cache for all the neccessary files so that the new content is read from the database and sizes of these files are updated in the filesystem.\n\n### Airlock\n\nThe CLI binary will require a username, a bucket and a filename. Password is either given as input or in an environmental variable.\n\n#### Build and Run\n```bash\ngo build -o ./airlock ./cmd/airlock/main.go\n```\nTest install.\n```bash\n./airlock -help\nUsage of ./airlock:\n  -alsologtostderr\n    \tlog to standard error as well as files\n  -debug\n    \tEnable debug prints\n  -journal-number string\n    \tJournal Number/Name specific for Findata uploads\n  -log_backtrace_at value\n    \twhen logging hits line file:N, emit a stack trace\n  -log_dir string\n    \tIf non-empty, write log files in this directory\n  -logtostderr\n    \tlog to standard error instead of files\n  -original-file string\n    \tFilename of original unecrypted file when uploading pre-encrypted file from Findata vm\n  -project string\n    \tSD Connect project if it differs from that in the VM\n  -quiet\n    \tPrint only errors\n  -segment-size int\n    \tMaximum size of segments in Mb used to upload data. Valid range is 10-4000. (default 4000)\n  -stderrthreshold value\n    \tlogs at or above this threshold go to stderr\n  -v value\n    \tlog level for V logs\n  -vmodule value\n    \tcomma-separated list of pattern=N settings for file-filtered logging\n``` \n\nExample run: `./airlock username ExampleBucket ExampleFile` will export file `ExampleFile` to bucket `ExampleBucket`.\n\n## Troubleshooting\nSee [troubleshooting](docs/troubleshooting.md) for fixes to known issues.\n\n## License\n\nData Gateway is released under `MIT`, see [LICENSE](LICENSE).\n\n[Wails](https://wails.io) is released under [MIT](https://github.com/wailsapp/wails/blob/master/LICENSE)\n\n[CgoFuse](https://github.com/billziss-gh/cgofuse) is released under [MIT](https://github.com/billziss-gh/cgofuse/blob/master/LICENSE.txt)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fsda-filesystem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcscfi%2Fsda-filesystem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fsda-filesystem/lists"}