{"id":13621961,"url":"https://github.com/circles-learning-labs/ecto_adapters_dynamodb","last_synced_at":"2025-10-21T15:16:08.653Z","repository":{"id":21890632,"uuid":"94349953","full_name":"circles-learning-labs/ecto_adapters_dynamodb","owner":"circles-learning-labs","description":"DynamoDB adapter for Elixir's Ecto Database layer.","archived":false,"fork":false,"pushed_at":"2025-03-06T16:11:56.000Z","size":955,"stargazers_count":65,"open_issues_count":19,"forks_count":14,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-10T23:38:48.016Z","etag":null,"topics":["adapter","dynamodb","dynamodb-adapter","ecto","ecto-adapter","ecto-adapters","elixir"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/circles-learning-labs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-14T16:10:15.000Z","updated_at":"2025-02-24T11:26:27.000Z","dependencies_parsed_at":"2024-01-14T07:59:13.917Z","dependency_job_id":"2eb44e0e-b0fe-47d1-a803-a1f7742cc291","html_url":"https://github.com/circles-learning-labs/ecto_adapters_dynamodb","commit_stats":{"total_commits":616,"total_committers":16,"mean_commits":38.5,"dds":0.5568181818181819,"last_synced_commit":"25d9befec7343e055cbecaa955c9661bcbcb9039"},"previous_names":[],"tags_count":52,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/circles-learning-labs%2Fecto_adapters_dynamodb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/circles-learning-labs%2Fecto_adapters_dynamodb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/circles-learning-labs%2Fecto_adapters_dynamodb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/circles-learning-labs%2Fecto_adapters_dynamodb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/circles-learning-labs","download_url":"https://codeload.github.com/circles-learning-labs/ecto_adapters_dynamodb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248352102,"owners_count":21089385,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adapter","dynamodb","dynamodb-adapter","ecto","ecto-adapter","ecto-adapters","elixir"],"created_at":"2024-08-01T21:01:12.512Z","updated_at":"2025-10-21T15:16:03.603Z","avatar_url":"https://github.com/circles-learning-labs.png","language":"Elixir","funding_links":[],"categories":["Elixir"],"sub_categories":[],"readme":"# Ecto.Adapters.DynamoDB\n[![Hex.pm](https://img.shields.io/hexpm/v/ecto_adapters_dynamodb.svg)](https://hex.pm/packages/ecto_adapters_dynamodb)\n[![Build Docs](https://img.shields.io/badge/hexdocs-release-blue.svg)](https://hexdocs.pm/ecto_adapters_dynamodb/Ecto.Adapters.DynamoDB.html)\n\nThis is a partial implementation of an Elixir Ecto adapter for Amazon's DynamoDB. It's very much a work in progress, and has plenty of rough edges. It's complete enough that we're actually using it in other projects, so we're opening it up to the community in hopes that others will find it useful as well :-)\n\nKeep in mind that DynamoDB is a key-value store designed for very high scale, while the Ecto abstractions are primarily designed to work with relational databases. As such, we've had to make significant compromises in the implementation of this adapter to make it work. Please understand that, while we are using it in production, it should be considered **beta** - **do not deploy it without thoroughly testing it for your use cases.**\n\nIf you wish to contribute, please run `$ mix test` and confirm that the test results are error-free before you push your commits. (Bonus points for improving our tests and adding your own tests for your changes. Patches with corresponding tests are more likely to be accepted, especially if they are significant.)\n\n### Special thanks to ExAws project\n\nWe use [ExAws](https://github.com/CargoSense/ex_aws/) to wrap the actual DynamoDB API and requests. This project would not be possible without the extensive work in ExAws.\n\n### Design limitations\n\nThere are a lot of common things you can do in Ecto with a SQL database that you just can't do (or can't do efficiently) with DynamoDB. If you expect to pick up your existing Ecto-based app and just swap in DynamoDB, you're going to be disappointed. You still have to use this adapter the same way you would approach using a key-value store, and avoid the kinds of patterns you'd use with a relational database.\n\n#### Is DynamoDB the right choice for you?\n\nIt may not be. Understand the DynamoDB limitations. It's designed for very high scale, throughput, and reliability. As a result of this design, there are many kinds of operations that are impossible. Other things are technically possible but not advisable, due to high costs in terms of performance and/or money.\n\nA good starting point is Amazon's own documentation:\n[Amazon: Best Practices for DynamoDB](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/BestPractices.html)\n\nOur philosophy when creating this adapter can generally be summed up as:\n\n *Try to do what the end user will expect the adapter to do, **unless** it's likely to ruin DynamoDB's performance.*\n\nAn example of this is our handling of table scans (see below).\n\nLastly, please read and understand how DynamoDB and its queries and indexes work. If you don't, then a lot of the following behaviour is going to seem random, and you'll be frustrated trying to figure out why things don't work the way you expect them to. We've done our best to simplify what we can, but underneath it all, it's still DynamoDB.\n\n#### How we use indexes\n\nIn DynamoDB, we can fetch individual records or batches of records *very* quickly if we know the primary key to look up, or the key of an indexed field. We **can't** easily perform queries which don't have a simple key or ID to look up:\n\nWill work: (note that this will be a *case sensitive* match as well.)\n\n`select * from people where name = 'ALICE'`\n\nWon't Work:\n\n`select * from people where name like 'Ali%'`\n\n(Obviously these are SQL queries, not Ecto queries, but the above examples just provide a general illustration of what sorts of limitations to expect.)\n\nWe will try our best to parse queries and find any relevant DynamoDB indexes that exists (this includes both HASH indexes, and HASH+RANGE indexes). As long as the `FROM` clause contains at least **one** HASH key from a DynamoDB index, a query will be constructed using our best guess at the most specific matching index (this may not be the best index - unlike a SQL server, we don't understand the data in the table, so the adapter may have to guess - although users can specify the index for queries where the index may be ambiguous, more on that below). Any other fields in the `FROM` criteria will be converted to DynamoDB filters as required to ensure you only get back the data you requested. We also support `is_nil` in queries. This will test whether the attribute is either set to `null` *or* whether the attribute is missing from the record altogether. Please note that DynamoDB does not allow for this type of filtering on attributes that are being queried against, whether in the primary key or in a secondary index.\n\nIf we do not find any matching table index for the query (either a HASH key of an index or the HASH part of a composite HASH+RANGE key), the query will fail by default. It is possible to override this behaviour and have the adapter perform a DynamoDB *scan* instead. Since scans do not scale well, they can potentially be very costly with large data sets, and we have configured the adapter not to scan unless scanning is explicitly enabled. This can be done via global configuration options, or inline as an option to `Repo.all` and other query functions. See the section below on **scan** for more info.\n\nThe adapter will query DynamoDB for a list of indexes and indexed fields on the table, and by default it will cache the results to avoid the overhead of repeatedly pulling the same lists of indexes on every query. This does mean that if you update the indexes on a table in DynamoDB, you will need to execute the **Ecto.Adapters.DynamoDB.Cache.update_table_info!** function or restart the adapter.\n\n##### Secondary index selection - default behavior\n\nWhen querying a table (`from f in Foo, where: ...`), the adapter's default behavior is to automatically select the \"best\" index on which to query based on the fields in the query. A few things to note:\n\n- A hash-only index will be preferred over a composite index with the **same** hash key if only the hash key field is represented in the query. For example, if you have a `Person` model with a hash-only key on `first_name` and a composite key on `first_name, last_name`, a query that only includes `first_name` will use the hash-only index; a query with both fields will use the composite. While this probably seems like expected behavior, consider what will happen if the range key from the composite index is not a required field in your application; a query that includes both `first_name` and `last_name` **will not** include records where the range key is `nil` (as that record hasn't been added to that index), but a query that does not include `last_name` **will** return those records.\n\n- In the event that a query should include two fields fields that are represented by a hash-only index and the hash portion of a composite index, the hash-only index will always be chosen. Again, consider a `Person` model with a hash-only `first_name` index and a composite `last_name, email` index; a query that includes `first_name` and `last_name` will use the hash-only `first_name` index.\n\nSo in general, expect the adapter to prefer hash-only indexes where they could be seen as best-representing a query.\n\n**Lesson:** create your data models and queries carefully.\n\n#### Limited support for fetching all records - 'scan' disabled by default\n\nFetching records based on a hash of the primary key allows DynamoDB to distribute its data across many partitions on many servers, resulting in high scalability and reliability. It also means that you can't do arbitrary queries based on unindexed fields.\n\nWell, that's not quite true, but running queries against un-indexed fields is usually a terrible idea. We can translate queries without any matching indexes to a DynamoDB `scan` operation, but this is not recommended as it can easily burn through all your read capacity. By default, attempting to perform these kinds of queries will raise an error. You can allow them to succeed by enabling the `scan` option at the adapter level for all queries, or by specifying the corresponding option on individual queries. See 'scan' options below for more information.\n\nIf you need to do this a lot, you're losing most of the benefits of DynamoDB, so think carefully before you do.\n\n#### No joins\n\nDynamoDB does not support joins. Thus, neither do we. Pretty simple.\n\nWhile it's technically possible for us to decompose the query into multiple individual requests against each table and then perform the join ourselves, this will likely result in very poor performance, and burning through excess read units to do so. It's better to construct these 'joins' manually using key/value lookups against indexes carefully chosen to preserve your predictable key/value store performance.\n\nThis is one of those things that are technically possible, but would result in very unpredictable performance that could drag down your entire app, reducing or eliminating any benefit from DynamoDB. You're probably better off using another DB if this is a requirement.\n\nThat said, for very simple joins that match a limited number of keys where all the relevant fields are indexed, joins could probably be emulated pretty reasonably. We'd entertain the notion of accepting a patch for this, if anyone wants to go to the trouble, and if the code contains some reasonable safeguards to avoid executing big, expensive joins by accident. It would be tricky though, and it's certainly not a priority for us right now.\n\n#### Limited sorting\n\nDynamoDB can ONLY return sorted results if there is a matching HASH+RANGE index where the desired sort key is the RANGE portion of the index. In this case we support the **:scan_index_forward** [option](http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html) as a parameter to Repo queries. However, writing queries like `SELECT * FROM person ORDER BY last_name LIMIT 50` may not be practical; we'd have to retrieve every record from the table to do this. (See also *DynamoDB LIMIT \u0026 Paging* below.)\n\nFrom DynamoDB's [Query API](http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html):\n\u003eQuery results are always sorted by the sort key value. If the data type of the sort key is Number, the results are returned in numeric order; otherwise, the results are returned in order of UTF-8 bytes. By default, the sort order is ascending. To reverse the order, set the `ScanIndexForward` parameter to false.\n\n#### Update support\n\nWe currently support both `update` and `update_all` with some performance caveats. Since DynamoDB currently does not offer a batch update operation, we emulate it in `update_all` (and `update` if the full primary key is not provided). The adapter first fetches the query results to get all the relevant keys, then updates the records one by one (paging as it goes, see *DynamoDB LIMIT \u0026 Paging* below). Consequently, performance might be slower than expected due to the need to execute individual fetches followed by individual inserts. Also please note that this means that update operations are *not atomic*! Multiple concurrent updates to the same record can race with each other, causing some updates to be silently lost.\n\nAll of these caveats can be especially pernicious if you're performing eventually consistent reads, as is the default for DynamoDB: you could write a new version of a record to a key, then attempt to perform an update to the same key, which could read from a zone that hasn't received your write yet. This would cause the update's fetch to return an older version of the record, which will then be modified and written back to DDB, overwriting your changes from the previous write! Thus, even if a single client synchronously updates a key, waits for success, then does another update, you may still experience a complete loss of the first of those two updates! \n\n**Lesson:** be really careful with updates; and, you may want to use consistent reads unless you really know what you're doing (see the `consistent_read` option for more info).\n\n#### DynamoDB BatchWriteItem\n\nWe currently support DynamoDB's **BatchWriteItem** via Ecto's `insert_all` and `delete_all`. DynamoDB enforces a limit of 25 records maximum per batch write, but we allow for an unlimited number of records by chunking large groups.\n\n#### DynamoDB BatchGetItem\n\nWe currently support DynamoDB's **BatchGetItem** via an **:in** clause in `all` queries - for example, `Repo.all(from m in Model, where: m.id in [\"id_1\", \"id_2\"])` or `Repo.all(from m in Model, where: m.id in ^model_ids)`. For tables with a composite primary key, range keys must be supplied in another **:in** clause in matching order - for example, `TestRepo.all(from bp in BookPage, where: bp.id in [\"page:test-1\", \"page:test-2\"] and bp.page_num in [1, 2])`. DynamoDB enforces a limit of 100 records per batch get, but we allow for an unlimited number of records by chunking large groups into separate requests.\n\n#### Concurrent fetches\n\nAn important cororally to the above is that DynamoDB does not support **BatchGetItem** on secondary keys. The default (and, prior to 3.4.2, only) behaviour in this case is to do a series of individual, sequential fetches. This is generally fine for a small number of keys, but can blow out the query time dramatically for larger numbers. Performance in this case can be improved dramatically by running a series of concurrent processes, each fetching a chunk of the keyspace. You can enable this behaviour by setting the `concurrent_fetches` option to `true` in the adapter configuration. There are two further configuration options to tune the behaviour:\n\n* `max_fetch_concurrency` - the maximum number of concurrent processes to run. Defaults to 10.\n* `min_concurrent_fetch_batch` - the minium number of keys to fetch in each process. Defaults to 10.\n\nThe number of processes and keys per process is then determined by trying to maximise the number of processes (up to `max_fetch_concurrency`) while ensuring that each process fetches at least `min_concurrent_fetch_batch` keys. In the case that the number of processes ends up being 1, no additional processes will be spawned and the behaviour will be the same as if `concurrent_fetches` was set to `false`.\n\nThe \"best\" value for these settings will of course depend on your hardware, network conditions, and the size of the keyspace you're fetching. You may need to experiment to find the best settings for your use case, but the defaults should give a good starting point and a considerable speedup for large fetches.\n\nNote that this functionality only applies to queries that end up mapping to an `:in` clause on a secondary index.\n\nIt's important to note that this is really a bit of a hack - the \"right\" solution is to rejig your data so that you can fetch it with the primary key. But for occasional queries that don't fit neatly into your usual use cases, this can be useful.\n\n#### DynamoDB LIMIT \u0026 Paging\n\nBy default, we configure the adapter to fetch all pages recursively for a DynamoDB `query` operation, and to *not* fetch all pages recursively in the case of a DynamoDB `scan` operation. This default can be overridden with the inline **:recursive** and **:page_limit** options (see below). We do not respond to the Ecto `limit` option; rather, we support a **:scan_limit** option, which corresponds with DynamoDB's [limit option](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html#Query.Limit), limiting \"the number of items that it returns in the result.\"\n\n#### Only ONE DynamoDB adapter can be configured\n\nWe only launch one instance of ExAws application (and have not yet investigated running multiple instances). This means we can only point to a single amazon Dynamo instance. It's currently not possible to run against two different amazon AWS accounts concurrently. Hopefully this won't be a problem for most users.\n\n#### Adapter.Migration\n\nWe support Ecto migration tasks via **create_table**, **create_if_not_exists_table**, and **alter_table** only. The functions `add`, `remove` and `modify` work with corresponding *indexes* on the DynamoDB table, rather than columns (as they would in a relational database). The adapter will automatically wait and retry requests when encountering DynamoDB errors that have \"OK to retry? Yes\" listed in the DynamoDB docs, according to an exponential backoff schedule. Since working with DynamoDB indexes and describing tables includes many options outside of Ecto's scope, for our supported syntax, please see details and examples in the **Ecto.Adapters.DynamoDB.Migration** moduledoc, as well as the configuration options, **:migration_initial_wait**, **:migration_wait_exponent**, **:migration_max_wait**, **:migration_table_capacity**.\n\nPlease note: Ecto migration calls `Repo.all()` on the *schema_migrations* table, which corresponds with a DynamoDB scan. To run migrations, add `\"schema_migrations\"` (or the alternate name you've configured for it) in the configuration file to the config variable **:scan_tables**. Additionally, note that the creation of the schema-migration records table takes time - if you have not created it yourself already, we recommend running `mix ecto.migrate --step 0`, then confirming the table is up, which will prevent the adapter from attempting to retrieve records from the schema-migrations table before it's ready.\n\n### Unimplemented Features\n\nWhile the previous section listed limitations that we're unlikely to work around due to philosophical differences between DynamoDB as a key/value store vs an SQL relational database, there are some features that we just haven't implemented yet. Feel free to help out if any of these are important to you!\n\n#### Transactions\n\nDynamoDB and [ExAws.Dynamo](https://github.com/ex-aws/ex_aws_dynamo) support transactions, but they have not yet been implemented in this adapter. We will probably try to implement transactions at some point, but it's not a high priority for us - if you would like to make a valuable contribution to this project, you could help us to resolve [this issue](https://github.com/circles-learning-labs/ecto_adapters_dynamodb/issues/12).\n\n#### Adapter.Storage\n\nIn the current release, we do not support Adapter.Storage callbacks.\n\n#### Adapter.Structure\n\nTo be honest, we don't even know what this is for. So it's not going to work :)\n\n#### Associations \u0026 Embeds\n\nWhile we've not tested these, without joins it's unlikely they work well (if at all).\n\n#### Ecto.Adapter.Migration.lock_for_migrations/4\n\nThis callback was implemented under Ecto 3 - it does not seem to be critical for this adapter's behaviour, and we do not know if it is appropriate to implement for DynamoDB. If you have input on this, or you want to help out with its implementation, see the issue [here](https://github.com/circles-learning-labs/ecto_adapters_dynamodb/issues/34).\n\n#### Ecto.Adapter.checkout/3\n\nThis callback was implemented under Ecto 3 - it does not seem to be critical for this adapter's behaviour, and we do not know if it is appropriate to implement for DynamoDB. If you have input on this, or you want to help out with its implementation, see the issue [here](https://github.com/circles-learning-labs/ecto_adapters_dynamodb/issues/33).\n\n#### Ecto.Adapter.Queryable.stream/5\n\nThis callback was implemented under Ecto 3 - it does not seem to be critical for this adapter's behaviour, and we do not know if it is appropriate to implement for DynamoDB. If you have input on this, or you want to help out with its implementation, see the issue [here](https://github.com/circles-learning-labs/ecto_adapters_dynamodb/issues/32).\n\n### So what DOES work?\n\nWell, basic CRUD really, which is all you should really expect from a key/value store :).\n\nGet, Insert, Delete and Update. As long as it's simple queries against single tables, it's probably going to work. Anything beyond that probably isn't. All of the following Ecto functions should work to some extent, if not necessarily in every scenario.\n\n* all/2\n* delete/2\n* delete!/2\n* delete_all/2\n* get/3\n* get!/3\n* get_by/3\n* get_by!/3\n* insert/2\n* insert!/2\n* insert_all/3\n* one/2\n* one!/2\n* update/2\n* update!/2\n* update_all/3\n\n#### Migrations\n\nIn late 2018, Amazon introduced a new billing option for tables that allows users to specify table billing as *pay per request* (AKA \"on-demand\") as well as *provisioned* - [more info](https://aws.amazon.com/dynamodb/pricing/on-demand/) - our migrations support both of these options. See `lib/ecto_adapters_dynamodb/migration.ex` for examples.\n\n## Installation\n\nInstall the [Hex](https://hex.pm/packages/ecto_adapters_dynamodb) package by adding `ecto_adapters_dynamodb` to your list of dependencies in `mix.exs`:\n\n```elixir\ndefp deps do\n  [\n    {:ecto_adapters_dynamodb, \"~\u003e 3.1\"}\n  ]\nend\n```\n\nOtherwise, to fetch from GitHub:\n\n```elixir\ndefp deps do\n  [\n    {:ecto_adapters_dynamodb, git: \"https://github.com/circles-learning-labs/ecto_adapters_dynamodb\", tag: \"3.1.2\"}\n  ]\nend\n```\n\n### Local DynamoDB version\n\nIn order to make sure your local version of DynamoDB is up to date with the current production features, please use the latest release of DynamoDB local. As of spring 2020, the latest version is `1.11.478`, released on January 16, 2020.\n\n### Configuration\n\nConfiguring a repository to use the DynamoDB ecto adapter is pretty similar to most other Ecto adapters - you'll set the `adapter` option in your app's Repo configuration file to `Ecto.Adapters.DynamoDB`, and then set the required (or optional) values in a keyword list in the appropriate application configuration files for your repo.\n\nThese values are **different** from the normal Ecto options. For example, in DynamoDB you don't have `username`, `password`, or `database` options - you'll need to delete these lines. Instead you'll add Amazon `access_key_id`, `secret_access_key`, `region`, and the optional `dynamodb` `host` and `scheme` options if you're not running against the default live Amazon instances (for example, running the local Amazon dev version of DynamoDB for testing and development).\n\nAll these options are quietly passed through to ExAws. Prior to version 3, existing ExAws configuration would be overwritten by these options. From version 3, the options are only used in calls made by the adapter, and other global ExAws configuration is maintained. See [ExAws Getting Started](https://hexdocs.pm/ex_aws/ExAws.html#module-getting-started) for more information on these options.\n\nA note on `access_key_id` and `secret_access_key`: This can simply be the actual key string, or it can be set to pull these from environment variables or Amazon roles (as per ExAws configuration). Some basic examples follow.\n\nYou may also omit all these ExAws options from the adapter config if you wish to configure ExAws manually (for example if you're using other features from ExAws such as S3, or dynamo_streams).\n\nNote that as of version 3, with the exception of logging configuration, all config options are set *per-repo*.\n\nThe adapter also supports [Confex](https://hexdocs.pm/confex)-style configuration options.\n\n**config/config.exs**\n\nInclude the repo module that's configured for the adapter among the project's Ecto repos:\n\n```elixir\nconfig :my_app, ecto_repos: [MyApp.Repo]\n```\n\n**repo module**\n\nConfigure your app's repo module to use `Ecto.Adapters.DynamoDB` as its `:adapter`:\n\n```elixir\ndefmodule MyApp.Repo do\n  use Ecto.Repo,\n    otp_app: :my_app,\n    adapter: Ecto.Adapters.DynamoDB\nend\n```\n\n**Supervisor**\n\nAdd the repo to your app's **Supervisor** configuration:\n\n```elixir\ndefmodule MyApp do\n  def start(_type, _args) do\n    children = [\n      MyApp.Repo\n    ]\n\n    opts = [strategy: :one_for_one, name: MyApp.Supervisor]\n    Supervisor.start_link(children, opts)\n  end\nend\n```\n\n**mix.exs**\n\nStart your app by adding it to the `:mod` key under `application`:\n\n```elixir\ndef application do\n  [\n    mod: {MyApp, []}\n  ]\nend\n```\n\nDepending on your app's configuration, you may need to add `:ecto_adapters_dynamodb` under `:extra_applications`.\n\n#### Configuring a Development Environment against a local instance of Dynamo\n\nFor development, we use the [local version](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html) of DynamoDB, and some dummy variable assignments. Note that the access key/secret here are hardcoded in to the config, and that we set a `dynamo` key that overrides the connection parameters from the defaults for AWS. We point it to `localhost:8000` - the default for a local DynamoDB test server.\n\n**config/dev.exs**\n\n```elixir\nconfig :my_app, MyApp.Repo,\n  # ExAws configuration\n  access_key_id: \"abcd\",\n  secret_access_key: \"1234\",\n  region: \"us-east-1\",\n  debug_requests: true,\t# ExAws option to enable debug on aws http request.\n  dynamodb: [\n    scheme: \"http://\",\n    host: \"localhost\",\n    port: 8000,\n    region: \"us-east-1\"\n  ]\n```\n\n#### Configuring a Production environment using Dynamo running on AWS\n\nFor a production setup, it's much simpler. No need to specify the `host`/`port` for `dynamodb`, as it will default to the appropriate AWS service in the region selected.\n\nIn this example configuration we do not hard code the secret or access key. These following settings tell ExAws to first attempt to pull the secret and access key from environment variables labelled `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. If it cannot find those environment variables, it will attempt to fall back on AWS role-based authentication (this only applies if this instance is running in an appropriately configured Amazon instance).\n\nTo reiterate, this is just standard ExAws configuration that we're wrapping up in our adapter config. Please consult the ExAws docs for further information.\n\n**config/prod.exs**\n\n```elixir\nconfig :my_app, MyApp.Repo,\n  # ExAws configuration\n  access_key_id: [{:system, \"AWS_ACCESS_KEY_ID\"}, :instance_role],\n  secret_access_key: [{:system, \"AWS_SECRET_ACCESS_KEY\"}, :instance_role],\n  region: \"us-east-1\"\n```\n\n### Other adapter options\n\nThe following are adapter options that apply to the Ecto adapter, and are NOT related to ExAws configuration. They control certain behavioural aspects for the driver, enabling and disabling default behaviours and features on queries.\n\n```elixir\nconfig :my_app, MyApp.Repo\n  dynamodb_local: true,\n  insert_nil_fields: false,\n  remove_nil_fields_on_update: true,\n  cached_tables: [\"colour\"]\n```\n\nThe above snippet will (1) notify the adapter that you are running it against a local dev instance of DynamoDB, (2) set the adapter to ignore fields that are set to `nil` in the changeset, inserting the record without those attributes, (3) set the adapter to remove attributes in a record during an update where those fields are set to `nil` in the changeset, and (4) cache scan results from the \"colour\" table, providing the cached result in subsequent calls. More details for each of those options follow.\n\n#### Production DynamoDB vs. local development DynamoDB\n\n**:dynamodb_local** :: boolean, *default:* `false`\n\nIndicate whether you are running against production (default) or local DynamoDB. The local development version of DynamoDB is not a true replica of the software used in production, and some local behaviours require special handling - in some scenarios for instance, production DynamoDB will raise an error whereas the local version will just hang until it times out. We **strongly** recommend setting this config value to `true` in any environment you will be running against local DynamoDB.\n\nHere's an illustration of a situation where this will be useful:\n\nA developer writes a migration that creates a new table set to the \"provisioned\" billing mode and runs the migration locally and in production. Several months (and several migrations) later, an administrator changes the billing mode for the table from \"provisioned\" to \"on-demand\" via the AWS dashboard. The developer is asked to add a new index to the table, and she writes a migration for the index that does not specify provisioned throughput. Since the production table is now \"on-demand\", the index will run against production without issue; however, her local table is still set to \"provisioned\". In production, attempting to add an index with no specified throughput to a \"provisioned\" table will raise the error\n\n```\n(ExAws.Error) ExAws Request Error! {\"ValidationException\", \"One or more parameter values were invalid: Both ReadCapacityUnits and WriteCapacityUnits must be specified for index: index_name\"}\n```\n\nbut local DynamoDB will hang until it times out and the migration will not be run.\n\nIt is our opinion that the developer in this scenario would want to be able to run the migration locally without necessarily having to change the billing mode of the table, since there is no local equivalent of the AWS dashboard and rewriting/rerunning migrations may be undesirable or impractical. Instead, by setting `dynamodb_local: true`, this adapter will \"ignore\" the discrepancy by quietly adding default provisioned throughput to the index, allowing the developer to continue her work while keeping her local migrations in sync with production.\n\nNot to put too fine a point on it, but attempting to run a migration to add an index with provisioned throughput to an \"on-demand\" table in production will raise the error\n\n```\n(ExAws.Error) ExAws Request Error! {\"ValidationException\", \"One or more parameter values were invalid: Neither ReadCapacityUnits nor WriteCapacityUnits can be specified for index: name when BillingMode is PAY_PER_REQUEST\"}\n```\n\nwhereas local DynamoDB will simply ignore the throughput and proceed to add the index anyway... so setting this value in your configuration will help to ensure uniform behavior when migrating indexes.\n\n#### `nil` value handling options\n\n**:insert_nil_fields** :: boolean, *default:* `true`\n\nDetermines if fields in the changeset with `nil` values will be inserted as DynamoDB `null` values or not set at all. This option is also available inline per query. Please note that DynamoDB does not allow setting indexed attributes to `null` and will respond with an error. It does allow removal of those attributes.\n\n**:remove_nil_fields_on_update** :: boolean, *default:* `false`\n\nDetermines if, during **Repo.update** or **Repo.update_all**, fields in the changeset with `nil` values will be removed from the record/s or set to the DynamoDB `null` value. This option is also available inline per query.\n\n**:empty_mapset_to_nil** :: boolean, *default:* `false`\n\nWhen a field contains a `DynamoDBSet` type and has a MapSet in it, by default the mapset must contain one or more values, otherwise an `insert` on the object will fail. This is due to DynamoDB not supporting empty sets. Setting this value to `true` will convert empty mapset values to `nil` before writing, allowing them to be written to DynamoDB as a null value.\n\n**:nil_to_empty_mapset** :: boolean, *default:* `false`\n\nWhen a field contains a `DynmamoDBSet` type, and the value in DynamoDB is null, setting this option causes the loaded value to be populated with an empty MapSet (the result of `MapSet.new()`) rather than being left as `nil`.\n\n#### Scan-related options\n\n**:scan_tables** :: [string], *default:* `[]`\n\nA list of table names for tables pre-approved for a DynamoDB **scan** command in case an indexed field is not provided in the query *wheres*. By default, scans are completely disabled on all tables. Use this option carefully; you may be better off using the inline query options to make sure you only perform table scans when you explicitly expect to do so.\n\n**:scan_limit** :: integer, *default:* `100`\n\nSets the default limit on the number of records scanned when calling DynamoDB's **scan** command. This can be overridden by the inline **:scan_limit** option. Included as **limit** in the DynamoDB query. (This option does not apply to queries performing recursive fetches.)\n\n**:scan_all** :: boolean, *default:* `false`\n\nPre-approves all tables for a DynamoDB **scan** command in case an indexed field is not provided in the query *wheres*.\n\n**:cached_tables** :: [string], *default:* `[]`\n\nA list of table names for tables assigned for caching of the first page of results (without setting DynamoDB's **limit** parameter in the scan request). For a set table, call `Repo.all(Model)` to cache the first page of results. To override the caching for a table in this list, and perform a regular scan with associated inline options (see below), provide an additional `scan: true` option with the query; for example, `Repo.all(Model, scan: true, recursive: true)`.\n\n#### Migration-related options\n\n**:migration_source** :: string, *default:* `\"schema_migrations\"`\n\nThe name of the table that should be used for tracking schema migrations.\n\n**:migration_initial_wait** :: integer, *default:* `1000`\n\nThe time in milliseconds of the first wait period before retrying a DynamoDB **create_table** or **update_table** request.\n\n**:migration_wait_exponent** :: float, *default:* `1.05`\n\nThe exponent to which the wait time between sequential retries of **create_table** or **update_table** requests is raised.\n\n**:migration_max_wait** :: integer, *default:* `15000`\n\nThe maximum wait time in milliseconds over sequential retries of a particular **create_table** or **update_table** request. Set this to `0` to avoid retries altogether. \n\n**:migration_table_capacity** :: [integer, integer], *default:* `[1,1]`\n\nProvisionedThroughput as `[ReadCapacityUnits, WriteCapacityUnits]`, for the Schema Migrations table only, automatically created by Ecto if it does not exist.\n\n#### Logging Configuration\n\nAs of 3.3.4, setting the `use_logger` configuration option to `true` will to write logs using Elixir's built-in `Logger` system. This setting causes all other log-related settings below to be ignored, and\ninstead has behaviour dictated by `Logger`'s configuration.\n\nIf `use_logger` is not set, the adapter's logging options are configured during compile time, and can be altered in the application's configuration files (`config/config.exs`, `config/dev.exs`, `config/test.exs` and `config/test.exs`). To enable logging in colour, the `MIX_ENV` environment variable must be explicitly exported as `dev` during compilation.\n\nWe provide a few informational log lines, such as which adapter call is being processed, as well as the table, lookup fields, and options detected. Configure an optional log path to have the messages recorded on file.\n\nNote that logging configuration is at the adapter level and is common to all repos.\n\n**:log_levels** :: `[:info, :debug]`, *default:* `[:info]`\n\n**:log_colours** :: %{log-level-atom: IO.ANSI-colour-atom}, *default:* `info: :green, debug: :normal`\n\n**:log_path** :: string, *default:* `\"\"`\n\nExample:\n\n```elixir\nconfig :ecto_adapters_dynamodb,\n  log_levels: [:info, :debug]\n```\n\n## Inline Options\n\nThe adapter supports a mix of Ecto options and custom inline options; if the Ecto option is not listed here, assume the adapter will ignore it. The following options can be passed during runtime in the Ecto calls. For example, consider a DynamoDB table with a composite index (HASH+RANGE):\n\n```elixir\nMyApp.Repo.all(\n  (from MyApp.HikingTrip, where: [location_id: \"grand_canyon\"]),\n  recursive: false,\n  scan_limit: 5\n)\n```\n\nwill retrieve the first five results from the record set for the indexed HASH, \"location_id\" = \"grand_canyon\", disabling the default recursive page fetch for queries. (Please note that without `recursive: false`, the adapter would ignore the scan limit.)\n\n### Supported Ecto Options\n\n**:on_conflict** :: `:raise | :nothing | :replace_all`, *default:* `:raise`\n\nBy default, the adapter will provide the condition expression, `attribute_not_exists(PARTITION_KEY_ATTRIBUTE)` with the DynamoDB query, failing to insert if the record already exists. To perform an unconditional insert, possibly overwriting an existing record, provide the option `on_conflict: :replace_all` in the insert query. If `on_conflict: :nothing` is provided, a struct will be returned, although the record will not be inserted if there is a preexisting record with the same primary key.\n\n### Custom Inline Options\n\n#### **Inline Options:** *Repo.update*, *Repo.delete*\n\n**:range_key** :: `{attribute_name_atom, value}`, *default:* none\n\nIf the DynamoDB table queried has a composite primary key, an update or delete query must supply both the `HASH` and the `RANGE` parts of the key. We assume that your Ecto model schema will correlate its primary id with DynamoDB's `HASH` part of the key. However, since Ecto will normally only supply the adapter with the primary id along with the changeset, we offer the `range_key` option to avoid an extra query to retrieve the complete key. The adapter will attempt to query the table for the complete key if the **:range_key** option is not supplied.\n\n#### **Inline Options:** *Repo.update_all*\n\n**:add / :delete** :: `[{field_atom, MapSet}]`, *default:* none\n\nEcto does not currently support `:push` and `:pull` on fields that are not `:array` type. To perform DynamoDB's **add** and **delete** on sets, pass the action, field, and value as an option.\n\n**:prepend_to_list** :: `[field_atom]`, *default:* none\n\nTo prepend a value during a `:push` action, include the field in this option. For example:\n\n`Repo.update_all((from Country, where: [name: \"New Zealand\"]), [push: [tags: \"adventure\"]], prepend_to_list: [:tags])`\n\n**:pull_indexes** :: `[{field_atom, [integer]}]`, *default:* none\n\nTo remove an element in a DynamoDB list, we must supply the list index of the element/s. Include them in this option. If `:pull_indexes` is not specified, the adapter will attempt to find and remove all the occurrences of the value in the `:pull` keyword in the corresponding list field.\n\nHere's an example including both of the options above:\n\n```elixir\nRepo.update_all(\n  (from Model, where: [id: \"fffx\"]), \n  [set: [name: \"Speedy\"], inc: [int_field: 2], pull: [list_field_1: \"value to remove\"], list_field_2: \"value will be ignored\"],\n  add: [set_field_1: MapSet.new([\"add_this\"])], delete: [set_field_2: MapSet.new([\"remove_this\"])], pull_indexes: [list_field_2: [5]]\n)\n```\n\n#### DynamoDBSet\n\nFor convenience, we have added an Ecto type, **Ecto.Adapters.DynamoDB.DynamoDBSet**, which casts and validates an Elixir MapSet type. Once you've included it in your schema, or extended the Ecto.Type behaviour to MapSet, Ecto Repo insert, update, and get commands; and the adapter's DynamoDb set related options, `:add` and `:delete` (mentioned in, *Inline Options: Repo.update_all*) will apply to the MapSet type.\n\nHere's an example of how to declare the DynamoDBSet type in an Ecto schema:\n\n```elixir\ndefmodule Model do\n  use Ecto.Schema\n  \n  schema \"model\" do\n    ...\n    field :set, Ecto.Adapters.DynamoDB.DynamoDBSet\n    ...\n```\n\n#### **Inline Options:** *Repo.all, Repo.update_all, Repo.delete_all*\n\n**:scan_limit** :: integer, *default:* none, except configuration default applies to the DynamoDB `scan` command\n\nSets the limit on the number of records scanned in the current query. Included as **limit** in the DynamoDB query.\n\n**:scan** :: boolean, *default:* `false` (also depends on scan-related configuration)\n\nApproves a DynamoDB **scan** command for the current query in case an indexed field is not provided in the query *wheres*.\n\n**:exclusive_start_key** :: [key_atom: value], *default:* none\n\nAdds DynamoDB's **ExclusiveStartKey** to the current query, providing a starting offset.\n\n**:scan_index_forward** :: boolean, *default:* none\n\nAdds DynamoDB's **ScanIndexForward** to the current query, specifying ascending (true/default) or descending (false) traversal of the index. (Quoted from DynamoDB's [documentation](http://docs.aws.amazon.com/sdkfornet1/latest/apidocs/html/P_Amazon_DynamoDBv2_Model_QueryRequest_ScanIndexForward.htm).)\n\n**:consistent_read** :: boolean, *default:* none\n\nIf set to `true`, then the operation uses strongly consistent reads; otherwise, eventually consistent reads are used. Strongly consistent reads are not supported on global secondary indexes. If you query a global secondary index with ConsistentRead set to `true`, you will receive an error message. (Quoted from DynamoDB's [documentation](http://docs.aws.amazon.com/sdkfornet1/latest/apidocs/html/P_Amazon_DynamoDBv2_Model_QueryRequest_ConsistentRead.htm).)\n\n**:recursive** :: boolean, *default:* `true`, except for DynamoDB `scan` where default is `false`\n\nFetches all pages recursively and performs the relevant operation on results in the case of *Repo.update_all* and *Repo.delete_all*\n\n**:page_limit** :: integer, *default:* none\n\nSets the maximum number of pages to access. The query will execute recursively until the page limit has been reached or there are no more pages (overrides **:recursive** option).\n\n**:index** :: string | atom, *default:* none\n\nWhen querying on a secondary index, this option allows a user to explicitly set the index on which to query by providing the name of that index. In most query cases, the adapter will be able to select the \"best\" index to use without this option, but there are some scenarios where the choice may be ambiguous, such as a query on `first_name` and `last_name` where the indexes `first_name_last_name` and `last_name_first_name` both exist. Without passing an `:index` option in such cases, the adapter may arbitrarily select one index over the other, and consistent results can not be guaranteed. This is especially important when a query includes operations that are specific to either hash or range key behaviors such as `:in`, which will work when applied to hash keys but will break for range keys.\n\n#### QueryInfo agent\n\n**:query_info_key** :: string, *default:* none\n\nIf you would like the query information provided by DynamoDB (for example, to retrieve the LastEvaluatedKey even when no results are returned from the current page), include the option, **query_info_key:** *key_string*.\n\nAfter the query is completed, retrieve the query info from the adapter's **QueryInfo** agent (the key is automatically deleted from the agent upon retrieval):\n\n`Ecto.Adapters.DynamoDB.QueryInfo.get(key_string)`\n\nThe returned map corresponds with DynamoDB's return values:\n\n`%{\"Count\" =\u003e 10, \"LastEvaluatedKey\" =\u003e %{\"id\" =\u003e %{\"S\" =\u003e \"6814\"}}, \"ScannedCount\" =\u003e 100}`\n\n**Ecto.Adapters.DynamoDB.QueryInfo.get_key** provides a 32-character random string for convenience.\n\n#### **Inline Options:** *Repo.insert, Repo.insert_all*\n\n**:insert_nil_fields** :: boolean, *default:* set in configuration\n\nDetermines if fields in the changeset with `nil` values will be inserted as DynamoDB `null` values or not set at all.\n\n#### **Inline Options:** *Repo.update, Repo.update_all*\n\n**:remove_nil_fields** :: boolean, *default:* set in configuration\n\nDetermines if fields in the changeset with `nil` values will be removed from the record/s or set to the DynamoDB `null` value.\n\n### DynamoDB `between` and Ecto `:fragment`\n\nWe currently only support the Ecto fragments of the form:\n\n`from(m in Model, where: fragment(\"? between ? and ?\", m.attribute, ^range_start, ^range_end))`\n\n`from(m in Model, where: fragment(\"begins_with(?, ?)\", m.attribute, ^prefix))`\n\n## Caching\n\nThe adapter automatically caches its own calls to **describe_table** for retrieval of table information. We also offer the option to configure tables for scan caching. To update the cache after making a change in a table, the cache offers two functions:\n\n**Ecto.Adapters.DynamoDB.Cache.update_table_info!(table_name)**, *table_name* :: string\n\nThis re-fetches and caches the index data for the given table.\n\n**Ecto.Adapters.DynamoDB.Cache.update_cached_table!(table_name)**, *table_name* :: string\n\nThis runs a scan against the given table and updates the in-memory cached copy of it.\n\n## Developer Notes\n\nThe **projection_expression** option is used internally during **delete_all** to select only the key attributes and is recognized during query construction.\n\nDocumentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc) and published on [HexDocs](https://hexdocs.pm). Once published, the docs can be found at [https://hexdocs.pm/ecto_adapters_dynamodb](https://hexdocs.pm/ecto_adapters_dynamodb).\n\n## Upgrade guides\n\n### 0.X.X -\u003e 1.X.X\n\nPlease see the instructions [here](upgrade_guides/version_1_upgrade_guide.md)\n\n### 1.X.X -\u003e 2.X.X\n\nPlease see the instructions [here](upgrade_guides/version_2_upgrade_guide.md)\n\n### 2.X.X -\u003e 3.X.X\n\nPlease see the instructions [here](upgrade_guides/version_3_upgrade_guide.md)\n\n# License\nCopyright Circles Learning Labs\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this project except in compliance with the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcircles-learning-labs%2Fecto_adapters_dynamodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcircles-learning-labs%2Fecto_adapters_dynamodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcircles-learning-labs%2Fecto_adapters_dynamodb/lists"}