{"id":20092693,"url":"https://github.com/lendup/fs2-blobstore","last_synced_at":"2025-10-07T20:32:51.683Z","repository":{"id":48584800,"uuid":"126075011","full_name":"lendup/fs2-blobstore","owner":"lendup","description":"Minimal, idiomatic, stream-based Scala interface for key/value store implementations","archived":false,"fork":false,"pushed_at":"2020-02-22T12:42:44.000Z","size":175,"stargazers_count":91,"open_issues_count":9,"forks_count":26,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-07-20T03:21:46.367Z","etag":null,"topics":["fs2","scala"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lendup.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-20T19:59:26.000Z","updated_at":"2024-12-26T11:24:42.000Z","dependencies_parsed_at":"2022-09-17T13:51:19.869Z","dependency_job_id":null,"html_url":"https://github.com/lendup/fs2-blobstore","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/lendup/fs2-blobstore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lendup%2Ffs2-blobstore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lendup%2Ffs2-blobstore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lendup%2Ffs2-blobstore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lendup%2Ffs2-blobstore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lendup","download_url":"https://codeload.github.com/lendup/fs2-blobstore/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lendup%2Ffs2-blobstore/sbom","scorecard":{"id":584172,"data":{"date":"2025-08-11","repo":{"name":"github.com/lendup/fs2-blobstore","commit":"86bc933e5d801a0c684ff770947ad4faf1438bac"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":5,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-20T19:58:48.084Z","repository_id":48584800,"created_at":"2025-08-20T19:58:48.084Z","updated_at":"2025-08-20T19:58:48.084Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278840725,"owners_count":26055191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fs2","scala"],"created_at":"2024-11-13T16:44:19.971Z","updated_at":"2025-10-07T20:32:51.668Z","avatar_url":"https://github.com/lendup.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fs2-blobstore\n\n[![Build Status](https://travis-ci.org/lendup/fs2-blobstore.svg?branch=master)](https://travis-ci.org/lendup/fs2-blobstore)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.lendup.fs2-blobstore/core_2.12/badge.svg)](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.lendup.fs2-blobstore%22)\n[![codecov](https://codecov.io/gh/lendup/fs2-blobstore/branch/master/graph/badge.svg)](https://codecov.io/gh/lendup/fs2-blobstore)\n[![Join the chat at https://gitter.im/fs2-blobstore/Lobby](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/fs2-blobstore/Lobby)\n\n\nMinimal, idiomatic, stream-based Scala interface for key/value store implementations.\nIt provides abstractions for S3-like key/value store backed by different persistence \nmechanisms (i.e. S3, FileSystem, sftp, etc).\n\n### Installing\n\nfs2-blobstore is deployed to maven central, add to build.sbt:\n\n```sbtshell\nlibraryDependencies ++= Seq(\n  \"com.lendup.fs2-blobstore\" %% \"core\" % \"0.6.+\",\n  \"com.lendup.fs2-blobstore\" %% \"sftp\" % \"0.6.+\",\n  \"com.lendup.fs2-blobstore\" %% \"s3\" % \"0.6.+\"\n)\n```\n\n`core` module has minimal dependencies and only provides `FileStore` implementation.\n`sftp` module provides `SftpStore` and depends on [Jsch client](http://www.jcraft.com/jsch/). \n`s3` module provides `S3Store` and depends on [AWS S3 SDK](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3.html)\n\n\n### Store Abstraction\n\nThe goal of the [Store interface](core/src/main/scala/blobstore/Store.scala) is to \nhave a common representation of key/value functionality (get, put, list, etc) as \nstreams that can be composed, transformed and piped just like any other `fs2.Stream` \nor `fs2.Sink` regardless of the underlying storage mechanism.\n\nThis is especially useful for unit testing if you are building a S3 or SFTP backed \nsystem as you can provide a filesystem based implementation for tests that is \nguaranteed to work the same way as you production environment.\n\nThe three main activities in a key/value store are modeled like:\n\n ```scala\n def list(path: Path): fs2.Stream[F, Path]\n def get(path: Path, chunkSize: Int): fs2.Stream[F, Byte]\n def put(path: Path, contentLength: Long): fs2.Sink[F, Byte]  \n ```  \n\nNote that `list` and `get` are modeled as streams since they are reading \n(potentially) very large amounts of data from storage, while `put` is \nrepresented as a sink of byte so that any stream of bytes can by piped \ninto it to upload data to storage.\n\n### Implicit Ops\n\n`import blobstore.implicits._`\n\n[StoreOps](core/src/main/scala/blobstore/StoreOps.scala) and \n[PathOps](core/src/main/scala/blobstore/Path.scala) provide functionality on \nboth `Store` and `Path` for commonly performed tasks (i.e. upload/download a \nfile from/to local filesystem, collect all returned paths when listing, composing \npaths or extracting filename of the path).\n\nMost of these common tasks encapsulate stream manipulation and provide a simpler \ninterface that return the corresponding effect monad.  These are also very good \nexamples of how to use blobstore streams and sink in different scenarios.\n\n### Tests\n\nAll store implementations must support and pass the suite of tests in \n[AbstractStoreTest](core/src/test/scala/blobstore/AbstractStoreTest.scala). \nIt is expected that each store implementation (like s3, sftp, file) should \ncontain the `Store` implementation and at least one test suite that inherits \nfrom `AbstractStoreTest` and overrides store and root attributes:\n\n```scala\nclass MyStoreImplTest extends blobstore.AbstractStoreTest {\n  override val store: blobstore.Store[cats.effect.IO] = MyStoreImpl( ... )\n  override val root: String = \"my_store_impl_tests\"\n}\n```  \n\nThis test suite will guarantee that basic operations are supported properly and \nconsistent with all other `Store` implementations.\n\n**Running Tests:**\n\nTests are set up to run via docker-compose:\n\n```bash\ndocker-compose run --rm sbt \"testOnly * -- -l blobstore.IntegrationTest\"\n```\n\nThis will start a [minio](https://www.minio.io/docker.html) (Amazon S3 compatible \nobject storage server) and SFTP containers and run all tests not annotated as \n`@IntegrationTest`.\n\nYes, we understand `SftpStoreTest` and `S3StoreTest` are also _integration tests_ \nbecause they connect to external services, but we don't mark them as such because \nwe found these containers that allow to run them along unit tests and we want to \nexercise as much of the store code as possible.  \n\nCurrently, tests for `SftpStore` and `BoxStore` are annotated with `@IntegrationTest`\nbecause: (1) SFTP tests fail to run against sftp container in travis, and (2) we\nhave not found a box docker image. To run `BoxStore` integration tests locally\nyou need to provide env vars for `BOX_TEST_BOX_DEV_TOKEN` and `BOX_TEST_ROOT_FOLDER_ID`.\n\nRun box/sftp tests with:\n\n```bash\nsbt box/test\ndocker-compose run --rm sbt sftp/test\n```\n\n**Note:** this will exercise `AbstractStoreTest` tests against your box.com account.\n\n\n### Path Abstraction\n\n`blobstore.Path` is the representation of `key` in the key/value store. The key \nrepresentation is based on S3 that has a `root` (or bucket) and a `key` string.\n\nWhen functions in the `Store` interface that receive a `Path` should assume that only\nroot and key values are set, there is no guarantee that the other attributes of `Path`\nwould be filled: size, isDir, lastModified. On the other hand, when a `Store` implements\nthe list function, it is expected that all 3 fields will be present in the response.\n\nBy importing implicit `PathOps` into the scope you can make use of path composition `/`\nand `filename` function that returns the substring of the path's key after the last path\nseparator.\n\n**NOTE:** a good improvement to the path abstraction would be to handle OS specific \nseparators when referring to filesystem paths.\n\n### Store Implementations\n\n   * [FileStore](fs/src/main/scala/blobstore/fs/FileStore.scala) backed by local\n   FileSystem. FileStore is provided as part of core module because it doesn't\n   include any additional dependencies and it is used as the default source store\n   in TransferOps tests. It only requires root path in the local file system:\n     ```scala\n     import blobstore.Store, blobstore.fs.FileStore \n     import java.nio.file.Paths\n     import cats.effect.IO \n     val store: Store[IO] = FileStore[IO](Paths.get(\"tmp/\"))\n     ```\n   * [S3Store](s3/src/main/scala/blobstore/s3/S3Store.scala) backed by \n   [AWS S3](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3.html). \n   It requires an authenticated `AmazonS3` client:\n     ```scala\n     import blobstore.Store, blobstore.s3.S3Store\n     import com.amazonaws.services.s3.transfer.TransferManagerBuilder\n     import cats.effect.IO\n     val store: Store[IO] = S3Store[IO](TransferManagerBuilder.standard().build())\n     ```\n   * [SftpStore](sftp/src/main/scala/blobstore/sftp/SftpStore.scala) backed by \n   SFTP server with [Jsch client](http://www.jcraft.com/jsch/). It requires a \n   connected `ChannelSftp`:\n     ```scala\n     import blobstore.Store, blobstore.sftp.SftpStore\n     import com.jcraft.jsch.{ChannelSftp, JSch}\n     import cats.effect.IO\n\n     val jsch = new JSch()\n     val session = jsch.getSession(\"sftp.domain.com\")\n     session.connect()\n \n     val channel = session.openChannel(\"sftp\").asInstanceOf[ChannelSftp]\n     channel.connect(5000)\n\n     val store: Store[IO] = SftpStore(\"root/server/path\", channel)\n     ```\n   * [BoxStore](box/src/main/scala/blobstore/box/BoxStore.scala) backed by\n   a [BoxAPIConnection](https://github.com/box/box-java-sdk/blob/master/src/main/java/com/box/sdk/BoxAPIConnection.java),\n   which has multiple options for authentication. This requires that you have a Box app set up already. \n   See [Box SDK documentation](https://github.com/box/box-java-sdk) for more details:\n     ```scala\n     import blobstore.Store, blobstore.box.BoxStore\n     import com.box.sdk.BoxAPIConnection\n\n     val api = new BoxAPIConnection(\"myDeveloperToken\")\n     val store: Store[IO] = BoxStore[IO](api, \"rootFolderId\")\n     ```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flendup%2Ffs2-blobstore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flendup%2Ffs2-blobstore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flendup%2Ffs2-blobstore/lists"}