{"id":13683558,"url":"https://github.com/matt-filion/s3-db","last_synced_at":"2025-04-30T13:31:18.981Z","repository":{"id":40783739,"uuid":"81096600","full_name":"matt-filion/s3-db","owner":"matt-filion","description":"Document DB API for AWS S3","archived":false,"fork":false,"pushed_at":"2023-01-07T14:41:48.000Z","size":744,"stargazers_count":109,"open_issues_count":23,"forks_count":12,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-11-06T11:04:38.341Z","etag":null,"topics":["bucket","database","lambda","s3","serverless"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matt-filion.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-06T14:40:32.000Z","updated_at":"2024-09-25T18:50:04.000Z","dependencies_parsed_at":"2023-02-07T11:01:39.029Z","dependency_job_id":null,"html_url":"https://github.com/matt-filion/s3-db","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matt-filion%2Fs3-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matt-filion%2Fs3-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matt-filion%2Fs3-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matt-filion%2Fs3-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matt-filion","download_url":"https://codeload.github.com/matt-filion/s3-db/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224212253,"owners_count":17274383,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bucket","database","lambda","s3","serverless"],"created_at":"2024-08-02T13:02:15.405Z","updated_at":"2024-11-12T03:31:26.506Z","avatar_url":"https://github.com/matt-filion.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"![Node.js CI](https://github.com/matt-filion/s3-db/workflows/Node.js%20CI/badge.svg)\n\n# Document DB API for AWS S3\n\n[AWS S3](https://aws.amazon.com/s3) is incredibly cheap, has 2 9's of availability, 12 9s of resiliency, triggers via [AWS Lambda](https://aws.amazon.com/lambda/), cross region replication, versioning and pretty decent [Performance](./docs/Performance.md). Its a pretty compelling database solution for a lot of scenarios. A few other people agree, see [Pete Warden's blog](https://petewarden.com/2010/10/01/how-i-ended-up-using-s3-as-my-database/) and [this interesting solution](http://www.s3nosql.com.s3.amazonaws.com/infinitedata.html).\n\n# Whats New\n\nThis library has been changed drastically from the 1.x.x version as this was written to be TypeScript first. Also:\n\n- It makes use of async/await patterns.\n- TypeScript decorators signficantly improved/reduced configuration.\n- Model classes are no longer decorated with convenience functions (just metadata.)\n- A new collection instance is no longer async/promisified.\n- No need to create a 'Database' object.\n\n## Next?\n\n- JavaScript Examples\n- eTag verification on save, for collission detection.\n- Carry over 'copy'\n- Iterator pattern on ResourceList, or instead of.\n- Performance docs on Lambda @ 128mb, 512mb, 1024mb. 2024mb, 3036mb.\n- Bug: MD5 does not appear to be persisting or returning when doing .head() check on a document.\n\n# Usage\n\n## Installation\n\n`npm install s3-db --save` into the project where S3-DB will be used. If you dont already have the aws-sdk node module then `npm install aws-sdk` as well.\n\n## Configure S3DB\n\nThere are very reaonsable out of the box configurations, but if you would like to change any of them you can do so using the `S3DB` class. The only values that you need to worry about at the global level are the following.\n\nHere is a basic example overwriting all default values.\n\n```\nS3DB.update({\n    baseName: 'myapp',\n    stage: 'quality',\n    region: 'us-east-2',\n    bucketPattern: '{{stage}}-{{region}}-{{baseName}}-{{bucketName}}'\n});\n```\n\nNote: Below in the configuration section each of these values is explained in detail.\n\n## Decorate Model/Types\n\nCreate your model class and decorate it with `@collection()` and `@id()` appropriately. So that when an instance of that class type is passed into the appropriate `Collection` instance, it will know how it should be configured.\n\nIf you do not specify an argument for `@collection()` then the class type will be lower cased, and used as the name and all collection configuration defaults used.\n\nHere is a very basic example where you are fine with the name being generated by s3-db.\n\n```\n@collection()\nexport class User {\n\n    @id()\n    private id?: string;\n    private name?: string;\n    private age?: name;\n    private address?: Address;\n\n    constructor(){\n    }\n}\n```\n\nSo once you have your model decorated, you can create an instance of a `Collection` and begin creating/updating/deleting objects in a Bucket. Function on `Collection` is async/promisified so you can use either pattern.\n\nAsync Example.\n\n```\nconst collection: Collection\u003cUser\u003e = new Collection(User);\nfunction async doStuff(){\n    const user: User = await collection.save({name:'Testing',age:21});\n    const checkedUser: User = await collection.load(user.id);\n    await collection.delete(user.id);\n}\n```\n\nPromise Example.\n\n```\nconst collection: Collection\u003cUser\u003e = new Collection(User);\n\n/* Creates a user and generates an ID for the user record. */\ncollection.save({name:'Testing',age:21})\n    .then( (user: User) =\u003e collection.load(user.id) )\n    .then( (user: User) =\u003e collection.delete(user.id) );\n```\n\n# Configuration\n\nA complete list of all the configuration points and what values you can use.\n\n## S3DB\n\nConfigurations that are applied across all collections.\n\n| Name          | Default                                            | Description                                                                                                                                                                                                                                        |\n| ------------- | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| baseName      | `s3db`                                             | Used in the bucketPattern and logging as a namespace.                                                                                                                                                                                              |\n| stage         | `dev`                                              | The logical environment. Used in the bucketPattern.                                                                                                                                                                                                |\n| region        | `us-west-2`                                        | Used in the AWS configuration to target a specific region. Also used in the bucketpattern.                                                                                                                                                         |\n| bucketPattern | `{{stage}}-{{region}}-{{baseName}}-{{bucketName}}` | The name that is used to lookup the bucket for a collection. Must use valid S3 bucket name characters. The replacement values for `{{stage}}`, `{{region}}`, `{{baseName}}` and `{{bucketName}}` are all case sensitive. You can omit any of them. |\n\n## Collection\n\nConfigurations specific to a collection.\n\n| Name | Default | Description |\n| -------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| pageSize              | `100`                | Maximum of 1000. How many documents to return when doing a `.find()` operation.  |\n| serversideEncryption  | `true`               | If S3 server side encryption is enabled (encryption at rest.)   |\n| checkIsMOdified       | `true`               | If enabled, `save()` operations will check if the object provided has been modified before being saved. If it is not modified it returns without attempting to save to S3. |\n| isModified            | `MD5IsModified`      | A function that is used to check if an object is modified. If you override it, implement the `IsModified` interface.|\n| serialization         | `JSONSerialization`  | How objects are serialized to a string before they are perstisted to S3.  |\n| defaultIdGenerator    | `defaultIDGenerator` | Default generation is UUID v4. This is called when no generator is provided on the `@id()` annotation. |\n| validator             | `undefined`          | A function that can be used to check if the object being saved is valid. |\n| noExceptionOnNotFound | `false`              | Changes the behavior to return undefined rather than throw an excpetion, when no document is found. |\n\n# API's\n\nThe available objects, decorators and functions.\n\n## @collection(string? | CollectionConfiguration?)\n\nAnnotation indicates that a specific class corresponds to an S3 Bucket.\n\n## @id(generator?)\n\nAnotation indicates what attribute or field on a class will be the key for the persisted object. If this annotation is not used then 'id' is used, or added.\n\n## S3DB\n\nSingleton containing the 'global' configurations for the S3DB instance.\n\n### S3DB.update({ baseName?: string; stage?: string; bucketPattern?: string; region?: string })\n\nUpdates the default configuration with the values provided.\n\n### S3DB.setLogLevel(level: LogLevel): void\n\nUpdates the logging level of the S3DB logger and will change the default level that each collection instance defines.\n\n## Collection\n\nUsed to do CRUD operations. Need to create an instance to use.\n\n### const collection: Collection\u003cSomeClass\u003e = new Collection(SomeClass);\n\nCreates a new instance that will use the SomeClass definition (which should contain the `@collection` and `@id` decorators) to determine its configuration.\n\n### collection.head(id): Promise\u003cS3Metadata\u003e\n\nReturns the metadata for the corresponding object identified by the id.\n\n### collection.exists(id): Promise\u003cboolean\u003e\n\nReturns true if the object exists within the corresponding S3 bucket.\n\n### collection.save(toSave): Promise\u003cSomeClass\u003e\n\nCreates or updates an object in S3 that corresponds to the id of the toSave object passed in. If there is no id on the object, then one is generated.\n\n### collection.delete(id): Promise\u003cboolean\u003e\n\nRemoves an object from an S3 bucket for the corresponding id.\n\n### collection.find(prefix, pageSize, continuationToken): Promise\u003cReferenceList\u003e\n\nReturns a list of `S3Metadata` objects for all objects in the corresponding S3 bucket htat start with the prefix value provided. If continuationToken is passed in then the list will be a 'continuation' of a previous find operation.\n\n### collection.subCollection(prefix: string, typeOf: SomeClass): Collection\u003cSomeClass\u003e\n\nCreates a new collection where all operations will execute with the prefix in front of the id's used. So if the prefix is `/users/` then when `.load('1234')` is called the request will result in an ID lookup for `/users/1234`. Similarly, all objects saved will have the prefix applied when the ID is generated by the save operation, or, when an ID is provided and it does not `startWith()` the configured prefix.\n\n### collection.setLogLevel(level: LogLevel): void\n\nLets you change the logging level for this specific collection instance. At creation of the collection, the logging level is taken from the S3DB logger, as a child logger is created from it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatt-filion%2Fs3-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatt-filion%2Fs3-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatt-filion%2Fs3-db/lists"}