{"id":16875311,"url":"https://github.com/alexcastano/fluid","last_synced_at":"2025-03-22T07:31:46.650Z","repository":{"id":62429694,"uuid":"148039425","full_name":"alexcastano/fluid","owner":"alexcastano","description":"Fluid is a library to create meaningful IDs easily in Elixir","archived":false,"fork":false,"pushed_at":"2018-09-10T16:12:11.000Z","size":17,"stargazers_count":27,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-01T14:07:44.497Z","etag":null,"topics":["elixir","id"],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexcastano.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-09T15:34:52.000Z","updated_at":"2023-04-06T21:32:19.000Z","dependencies_parsed_at":"2022-11-01T20:09:54.549Z","dependency_job_id":null,"html_url":"https://github.com/alexcastano/fluid","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexcastano%2Ffluid","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexcastano%2Ffluid/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexcastano%2Ffluid/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexcastano%2Ffluid/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexcastano","download_url":"https://codeload.github.com/alexcastano/fluid/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244189790,"owners_count":20412991,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","id"],"created_at":"2024-10-13T15:35:38.418Z","updated_at":"2025-03-22T07:31:46.274Z","avatar_url":"https://github.com/alexcastano.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Fluid\n\nFluid is a library to create meaningful IDs.\n\nIt may be useful in context where data space it is important, ie: sending package over the network.\n\n## Installation\n\nThe package can be installed by adding `fluid` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:fluid, \"~\u003e 0.0.1-dev\"}\n  ]\nend\n```\n\n[Documentation](https://hexdocs.pm/fluid).\n\n## Example\n\nLet's see how we can implement the [Instagram ID](https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c) using fluid.\nThis format allows us to create ID independently, each Elixir node (or PostgreSQL instance) can create them\nwithout collisions and without communication.\nIt is sortable by time, independently where the ID was generated.\nLast, but not least, it is small. It is only 64 bits. This is perfect to index in SQL or to cache in Redis or in a ETS table.\n\n### Definition\n\n```elixir\ndefmodule MyApp.ID do\n  use Fluid,\n    fields: [\n      inserted_at: %Fluid.Field.NaiveDateTime{\n        size: 41,\n        epoch: ~N[2018-01-01 00:00:00],\n        time_unit: :millisecond\n      },\n      node_id: %Fluid.Field.Integer{size: 13},\n      local_id: %Fluid.Field.Integer{size: 10, unsigned: false}\n    ],\n    formats: [\n      hex: %Fluid.Format.Hexadecimal{\n        separator: ?-,\n        groups: 4\n      }\n    ],\n    ecto: [type: :integer, format: %Fluid.Formatter.Hexadecimal{}]\nend\n```\n\n### Fields\n\nWe defined 3 fields:\n* `inserted_at`: when the id was generated\n* `node_id`: in which node was generated\n* `local_id`: just a consecutive counter to avoid collision in the same millisecond\n\nThe type of `inserted_at` is `Fluid.Field.NaiveDateTime` which means it returns `NaiveDateTime` values.\nIts `time_unit` is `:millisecond`. We could use `:second` but the precision would not be enough for\nmany applications, or `:microsecond`, but the maximum date would be much closer.\n\nWe don't use Unix epoch (1970-01-01 00:00) to save the date,\nwe changed to a more modern date to optimize space.\nThe size of `inserted_at` is `41` bits.\nThis allows to create IDs until the following date:\n\n```elixir\niex(3)\u003e MyApp.ID.__fluid__(:max, :inserted_at)\n~N[2087-09-07 15:47:35]\niex(4)\u003e MyApp.ID.__fluid__(:min, :inserted_at)\n~N[2018-01-01 00:00:00]\n```\n\n80 years! Not bad at all!\nThe ID saves the number of millisecond since the epoch date.\nLet's see an example:\n\n```elixir\niex(7)\u003e MyApp.ID.__fluid__(:load, :inserted_at, \u003c\u003c1000::41\u003e\u003e)\n{:ok, ~N[2018-01-01 00:00:01]}\n```\n\nWhen we saved a `1000`, it is a second from the epoch date.\nThe inverse operation it is also available:\n\n```elixir\niex(9)\u003e MyApp.ID.__fluid__(:dump, :inserted_at, ~N[2018-01-01 00:00:01])\n{:ok, \u003c\u003c0, 0, 0, 1, 244, 0::size(1)\u003e\u003e}\niex(10)\u003e MyApp.ID.__fluid__(:dump, :inserted_at, ~N[2000-01-01 00:00:01])\n:error\n```\n\nIf we try to encode (or decode) invalid values it returns `:error`\n\nWe could access to the size of the field with the following call:\n\n```elixir\niex(11)\u003e MyApp.ID.__fluid__(:bit_size, :inserted_at)\n41\n```\n\nFor the `node_id` we have similar behaviour:\n\n```elixir\niex(13)\u003e MyApp.ID.__fluid__(:bit_size, :node_id)\n13\niex(14)\u003e MyApp.ID.__fluid__(:min, :node_id)\n0\niex(15)\u003e MyApp.ID.__fluid__(:max, :node_id)\n8191\n```\n\nSo we can have 8191 different nodes generating ids.\n\nAnd, of course, we can `load` and `dump` with a bitstring:\n\n```elixir\niex(17)\u003e MyApp.ID.__fluid__(:load, :node_id, \u003c\u003c255, 10::5\u003e\u003e)\n{:ok, 8170}\niex(18)\u003e MyApp.ID.__fluid__(:dump, :node_id, 8000)\n{:ok, \u003c\u003c250, 0::size(5)\u003e\u003e}\niex(19)\u003e MyApp.ID.__fluid__(:dump, :node_id, 9999)\n```\n\nFor the `local_id` field we gave the option of `unsigned: false` for demonstration purposes only:\n\n```elixir\niex(20)\u003e MyApp.ID.__fluid__(:bit_size, :local_id)\n10\niex(21)\u003e MyApp.ID.__fluid__(:min, :local_id)\n-512\niex(22)\u003e MyApp.ID.__fluid__(:max, :local_id)\n511\n```\n\nFor each node, we can generate 1024 ids per millisecond. Enough for the majority of apps.\n\n### Formats\n\nWe chose `Fluid.Format.Hexadecimal` because it is easier to read\nand it keeps the order correctly.\n\nLet's create a full ID:\n\n```elixir\niex(3)\u003e MyApp.ID.new(inserted_at: ~N[2018-01-01 00:00:00], node_id: 0, local_id: 0)\n{:ok, \"0000-0000-0000-0000\"}\niex(4)\u003e MyApp.ID.new(inserted_at: ~N[2043-07-31 12:34:56.654], node_id: 1976, local_id: 432)\n{:ok, \"5df8-423a-071e-e1b0\"}\n```\n\nSo, we can see it is simple to create IDs.\nThe format is using `groups: 4` hexadecimal characters and the separator is `-`\njust because it easier to read for the human eye.\n\nIn addition, the format respect the `inserted_at` order:\n\n```elixir\niex(5)\u003e \"5df8-423a-071e-e1b0\" \u003e \"0000-0000-0000-0000\"\ntrue\n```\n\nSo this way we can use those binary strings in our code.\nThis method is similar to the one used by Ecto with the UUID,\nit works with strings and not with bitstrings.\n\nHowever, we can decode an id to its bits representation if it is needed:\n\n```elixir\niex(6)\u003e MyApp.ID.decode(\"5df8-423a-071e-e1b0\")\n{:ok, \u003c\u003c93, 248, 66, 58, 7, 30, 225, 176\u003e\u003e}\niex(7)\u003e MyApp.ID.decode(\"0000-0000-0000-0000\")\n{:ok, \u003c\u003c0, 0, 0, 0, 0, 0, 0, 0\u003e\u003e}\n```\n\nIf we want to access to relevant data coded inside the ID we just:\n\n```elixir\niex(8)\u003e MyApp.ID.get(\"5df8-423a-071e-e1b0\", :inserted_at)\n{:ok, ~N[2043-07-31 12:34:56]}\niex(9)\u003e MyApp.ID.get(\"5df8-423a-071e-e1b0\", :node_id)\n{:ok, 1976}\niex(10)\u003e MyApp.ID.get(\"5df8-423a-071e-e1b0\", :local_id)\n{:ok, 432}\n```\n\n### More Introspection\n\n```elixir\niex(2)\u003e MyApp.ID.__fluid__(:bit_size)\n64\niex(3)\u003e MyApp.ID.__fluid__(:fields)\n[:inserted_at, :node_id, :local_id]\niex(4)\u003e MyApp.ID.__fluid__(:field, :inserted_at)\n%Fluid.Field.NaiveDateTime{\n  epoch: ~N[2018-01-01 00:00:00],\n  size: 41,\n  time_unit: :millisecond\n}\niex(5)\u003e MyApp.ID.__fluid__(:field, :node_id)\n%Fluid.Field.Integer{size: 13, unsigned: true}\n```\n\n### Ecto\n\nThe last part of the definition of the ID is the `:ecto` part.\nThis creates the functions needed by Ecto to store the ID in the database.\nIn this case the `:type` is `:integer`.\nTo save as an integer we have to set the format to `Fluid.Format.Integer`.\nThat's all. Now we can use it:\n\n```elixir\ndefmodule MyApp.Repo.Migrations.CreatePost do\n  use Ecto.Migration\n\n  def change() do\n    create table(:post, primary_key: false) do\n      add(:id, :bigint, primary_key: true)\n      add(:user_id, references(:users, type: :bigint), null: false)\n      add(:body, :text)\n    end\n  end\nend\n\ndefmodule MyApp.Post do\n  use Ecto.Schema\n\n  @primary_key {:id, MyApp.ID, autogenerate: true, read_after_writes: true}\n  @foreign_key MyApp.ID\n  @timestamps_opts [inserted_at: false, type: :utc_datetime, usec: false]\n\n  schema \"posts\" do\n    belongs_to :user, MyApp.User\n    field :body, :string\n  end\n\n  def inserted_at(%__MODULE__{id: id}), do: MyApp.ID.get(id, :inserted_at)\nend\n```\n\nWe are still working with hexadecimal strings that are easy to read.\nEcto, internally, will use 64 bits integer to improve indexing and to save space in the database.\n\n```elixir\niex\u003e Repo.insert!(%Post{id: \"0000-1111-2222-3333\", user_id: \"ffff-0000-ffff-0000\", body: \"text\"})\n%Post{...}\niex\u003e Repo.get!(Post, \"0000-1111-2222-3333\")\n%Post{...}\n```\n\nLike we have the `inserted_at` field in the ID, we don't need the same timestamp field.\nWe defined a function to get it easily given the struct.\nWe can also paginate or search by `inserted_at` with just the ID:\n\n```elixir\niex\u003e init_date = MyApp.ID.new(inserted_at: ~N[2018-03-01 00:00:00], local_id: 0, node_id: 0)\n{:ok, \"0097-eb9a-0000-0000\"}\niex\u003e final_date = MyApp.ID.new(inserted_at: ~N[2018-04-01 00:00:00], local_id: 0, node_id: 0)\n{:ok, \"00e7-be2c-0000-0000\"}\niex\u003e from p in Post, where: p.id \u003e ^init_date and p.id \u003c ^final_date\n```\n\nAnd ordering by ID means ordering by date as well.\n\n\n## Optimized in compilation\n\nThe generated modules are optimized in compilation stage, avoiding unnecessary operation in runtime.\nMost of the functions use pattern matching with bitstrings which are very fast in Elixir.\nThis is needed because the functions are used often:\n\n* casting parameters in queries\n* to load any model\n* to insert any model\n* to update any model\n* to delete any model\n\n## Work in progress\n\nThis is a proof of concept. I use this kind of ID with very good results.\nThis library is a try to make it more generic, so everyone can create its own ID.\nThere are more options I like to add, better errors, etc.\n\nMore formats:\n\n  * Fluid.Format.Bitstring\n  * Fluid.Format.Base32\n  * Fluid.Format.Base64\n  * Fluid.Format.UrlBase64\n  * Fluid.Format.OrderedUrlBase64\n  * Fluid.Format.Map\n  * Fluid.Format.Struct\n\nAnd more field types:\n\n  * Fluid.Field.Binary\n  * Fluid.Field.Boolean\n  * Fluid.Field.Enum\n\nIf you are interested, just let me know.\n\n[Alex Castaño](https://alexcastano.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexcastano%2Ffluid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexcastano%2Ffluid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexcastano%2Ffluid/lists"}