{"id":20724511,"url":"https://github.com/laserdisc-io/fs2-aws","last_synced_at":"2025-04-08T12:09:41.641Z","repository":{"id":39743552,"uuid":"117368091","full_name":"laserdisc-io/fs2-aws","owner":"laserdisc-io","description":"fs2 utilities to interact with AWS","archived":false,"fork":false,"pushed_at":"2025-03-01T16:53:48.000Z","size":1268,"stargazers_count":195,"open_issues_count":33,"forks_count":50,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-01T11:03:09.700Z","etag":null,"topics":["hacktoberfest"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/laserdisc-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-13T18:32:52.000Z","updated_at":"2025-03-31T07:10:13.000Z","dependencies_parsed_at":"2023-10-17T04:37:40.568Z","dependency_job_id":"6e5e08ec-b4d2-4468-8d2d-3a585999fdcf","html_url":"https://github.com/laserdisc-io/fs2-aws","commit_stats":{"total_commits":803,"total_committers":32,"mean_commits":25.09375,"dds":0.3511830635118306,"last_synced_commit":"050493ebf34e6a66253ad6ad72febab7813c4482"},"previous_names":["dmateusp/fs2-aws"],"tags_count":113,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laserdisc-io%2Ffs2-aws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laserdisc-io%2Ffs2-aws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laserdisc-io%2Ffs2-aws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laserdisc-io%2Ffs2-aws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/laserdisc-io","download_url":"https://codeload.github.com/laserdisc-io/fs2-aws/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247838444,"owners_count":21004580,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest"],"created_at":"2024-11-17T04:15:06.661Z","updated_at":"2025-04-08T12:09:41.621Z","avatar_url":"https://github.com/laserdisc-io.png","language":"Scala","funding_links":[],"categories":["Table of Contents"],"sub_categories":["Misc"],"readme":"# fs2-aws\n![Build](https://github.com/laserdisc-io/fs2-aws/workflows/Build/badge.svg)\n![Release](https://github.com/laserdisc-io/fs2-aws/workflows/Release/badge.svg)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/io.laserdisc/fs2-aws_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/io.laserdisc/fs2-aws_2.12)\n[![Coverage Status](https://coveralls.io/repos/github/laserdisc-io/fs2-aws/badge.svg?branch=main)](https://coveralls.io/github/laserdisc-io/fs2-aws?branch=main)\n\nfs2 Streaming utilities for interacting with AWS\n\n## Scope of the project\n\nfs2-aws provides an [fs2](https://github.com/functional-streams-for-scala/fs2) interface to AWS services\n\nThe design goals are the same as fs2:\n\u003e compositionality, expressiveness, resource safety, and speed\n\n## Using:\n\nFind [the latest release version](https://github.com/laserdisc-io/fs2-aws/releases) and add the following dependency:\n\n```sbt\nlibraryDependencies +=  \"io.laserdisc\" %% \"fs2-aws\" % \"VERSION\"\n```\n\n## S3\n\nThe module `fs2-aws-s3` provides a purely functional API to operate with the AWS-S3 API. It defines four functions:\n\n```scala\ntrait S3[F[_]] {\n  def delete(bucket: BucketName, key: FileKey): F[Unit]\n  def uploadFile(bucket: BucketName, key: FileKey): Pipe[F, Byte, ETag]\n  def uploadFileMultipart(bucket: BucketName, key: FileKey, partSize: PartSizeMB): Pipe[F, Byte, ETag]\n  def readFile(bucket: BucketName, key: FileKey): Stream[F, Byte]\n  def readFileMultipart(bucket: BucketName, key: FileKey, partSize: PartSizeMB): Stream[F, Byte]\n}\n```\n\nYou can find out more in the scaladocs for each function, but as a rule of thumb for:\n\n- Small files: use `readFile` and `uploadFile`.\n- Big files: use `readFileMultipart` and `uploadFileMultipart`.\n\nYou can also combine them as you see fit. For example, use `uploadFileMultipart` and then read it in one shot using `readFile`.\n\n### Getting started with the S3 module\n\nIn order to create an instance of `S3` we need to first create an `S3Client`. Here's an example of the former:\n\n```scala\ndef s3StreamResource: Resource[IO, S3AsyncClientOp[IO]] =\n  for {\n    credentials = AwsBasicCredentials.create(\"accesskey\", \"secretkey\")\n    port        = 4566\n    s3 \u003c- S3Interpreter[IO](blocker).S3AsyncClientOpResource(\n      S3AsyncClient\n        .builder()\n        .credentialsProvider(StaticCredentialsProvider.create(credentials))\n        .endpointOverride(URI.create(s\"http://localhost:$port\"))\n        .region(Region.US_EAST_1)\n    )\n  } yield s3\n```\n\nNow we can create our `S3[IO]` instance:\n\n```scala\n\ns3StreamResource.map(S3.create[IO]).use { s3 =\u003e\n  // do stuff with s3 here (or just share it with other functions)\n}\n```\n\nCreate it once and share it as an argument, as any other resource.\n\nFor more details on how to work with S3 streams follow [link](fs2-aws-examples/src/main/scala/fs2/aws/examples/S3Example.scala)\n\n### Reading a file from S3\n\nThe simple way:\n\n```scala\ns3.readFile(BucketName(\"test\"), FileKey(\"foo\"))\n  .through(fs2.text.utf8Decode)\n  .through(fs2.text.lines)\n  .evalMap(line =\u003e IO(println(line)))\n```\n\nThe streaming way in a multipart fashion (part size is indicated in MBs and must be 5 or higher):\n\n```scala\ns3.readFileMultipart(BucketName(\"test\"), FileKey(\"foo\"), partSize = 5)\n  .through(fs2.text.utf8Decode)\n  .through(fs2.text.lines)\n  .evalMap(line =\u003e IO(println(line)))\n```\n\n### Writing to a file in S3\n\nThe simple way:\n\n```scala\nStream.emits(\"test data\".getBytes(\"UTF-8\"))\n  .through(s3.uploadFile(BucketName(\"foo\"), FileKey(\"bar\"))\n  .evalMap(t =\u003e IO(println(s\"eTag: $t\")))\n```\n\nThe streaming way in a multipart fashion. Again, part size is indicated in MBs and must be 5 or higher.\n\n```scala\nStream.emits(\"test data\".getBytes(\"UTF-8\"))\n  .through(s3.uploadFileMultipart(BucketName(\"foo\"), FileKey(\"bar\"), partSize = 5))\n  .evalMap(t =\u003e IO(println(s\"eTag: $t\")))\n```\n\n### Deleting a file in S3\n\nThere is a simple function to delete a file.\n\n```scala\ns3.delete(BucketName(\"foo\"), FileKey(\"bar\"))\n```\n\n## Kinesis\n### Streaming records from Kinesis with KCL\nExample using IO for effects (any monad `F \u003c: ConcurrentEffect` can be used):\n```scala\nval stream: Stream[IO, CommittableRecord] = readFromKinesisStream[IO](\"appName\", \"streamName\")\n```\n\nThere are a number of other stream constructors available where you can provide more specific configuration for the KCL worker.\n\n#### Testing\nTODO: Implement better test consumer\n\nFor now, you can stub CommitableRecord and create a fs2.Stream to emit these records:\n```scala\nval record = new Record()\n  .withApproximateArrivalTimestamp(new Date())\n  .withEncryptionType(\"encryption\")\n  .withPartitionKey(\"partitionKey\")\n  .withSequenceNumber(\"sequenceNum\")\n  .withData(ByteBuffer.wrap(\"test\".getBytes))\n\nval testRecord = CommittableRecord(\n  \"shardId0\",\n  mock[ExtendedSequenceNumber],\n  0L,\n  record,\n  mock[RecordProcessor],\n  mock[IRecordProcessorCheckpointer])\n```\n\n#### Checkpointing records\nRecords must be checkpointed in Kinesis to keep track of which messages each consumer has received. Checkpointing a record in the KCL will automatically checkpoint all records upto that record. To checkpoint records, a Pipe and Sink are available. To help distinguish whether a record has been checkpointed or not, a CommittableRecord class exists to denote a record that hasn't been checkpointed, while the base Record class denotes a committed record.\n\n```scala\nreadFromKinesisStream[IO](\"appName\", \"streamName\")\n  .through(someProcessingPipeline)\n  .to(checkpointRecords_[IO]())\n```\n\n### Publishing records to Kinesis with KPL\nA Pipe and Sink allow for writing a stream of tuple2 (partitionKey, ByteBuffer) to a Kinesis stream.\n\nExample:\n```scala\nStream(\"testData\")\n  .map { d =\u003e (\"partitionKey\", ByteBuffer.wrap(d.getBytes))}\n  .to(writeToKinesis_[IO](\"streamName\"))\n```\n\nAWS credential chain and region can be configured by overriding the respective fields in the KinesisProducerClient parameter to `writeToKinesis`. Defaults to using the default AWS credentials chain and `us-east-1` for region.\n\n### Use with LocalStack\n\nIn some situations (e.g. local dev and automated tests), it may be desirable to be able to consume from and publish to Kinesis running inside LocalStack.\nEnsure that the `endpoint` setting is set correctly (e.g. http://localhost:4566) and that `retrievalMode` is set to `Polling` (LocalStack doesn't support `FanOut`).\n\n## Kinesis Firehose\n**TODO:** Stream get data, Stream send data\n\n## SQS\nExample\n```scala\nimplicit val messageDecoder: Message =\u003e Either[Throwable, Quote] = { sqs_msg =\u003e\n    io.circe.parser.decode[Quote](sqs_msg.asInstanceOf[TextMessage].getText)\n}\nfs2.aws\n      .sqsStream[IO, Quote](\n        sqsConfig,\n        (config, callback) =\u003e SQSConsumerBuilder(config, callback))\n      .through(...)\n      .compile\n      .drain\n      .as(ExitCode.Success)\n```\n\nTesting\n```scala\n//create stream for testing\ndef stream(deferredListener: Deferred[IO, MessageListener]) =\n            aws.testkit\n              .sqsStream[IO, Quote](deferredListener)\n              .through(...)\n              .take(2)\n              .compile\n              .toList\n\n//create the program for testing the stream\nimport io.circe.fs2.aws.examples.syntax._\nimport io.circe.generic.auto._\nval quote = Quote(...)\nval program : IO[List[(Quote, MessageListener)]] = for {\n            d \u003c- Deferred[IO, MessageListener]\n            r \u003c- IO.racePair(stream(d), d.get).flatMap {\n              case Right((streamFiber, listener)) =\u003e\n                //simulate SQS stream fan-in here\n                listener.onMessage(new SQSTextMessage(Printer.noSpaces.pretty(quote.asJson)))\n                streamFiber.join\n              case _ =\u003e IO(Nil)\n            }\n          } yield r\n\n//Assert results\nval result = program\n            .unsafeRunSync()\nresult should be(...)\n```\n**TODO:** Stream send SQS messages\n\n## Support\n\n![YourKit Image](https://www.yourkit.com/images/yklogo.png \"YourKit\")\n\nThis project is supported by YourKit with monitoring and profiling Tools. YourKit supports open source with innovative and intelligent tools for monitoring and profiling Java and .NET applications. YourKit is the creator of [YourKit Java Profiler](https://www.yourkit.com/java/profiler/), [YourKit .NET Profiler](https://www.yourkit.com/.net/profiler/), and [YourKit YouMonitor](https://www.yourkit.com/youmonitor/).\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaserdisc-io%2Ffs2-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flaserdisc-io%2Ffs2-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaserdisc-io%2Ffs2-aws/lists"}