{"id":24569368,"url":"https://github.com/pzingg/sandbox","last_synced_at":"2025-08-07T01:25:55.437Z","repository":{"id":269607291,"uuid":"907127158","full_name":"pzingg/sandbox","owner":"pzingg","description":"Example Phoenix LiveView application for Bluesky timelines and firehose","archived":false,"fork":false,"pushed_at":"2025-01-05T18:47:12.000Z","size":1755,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-05T11:00:53.155Z","etag":null,"topics":["atproto","bluesky-api","bluesky-social","cid","dids","elixir","elixir-phoenix","merkle-tree","oauth2","phoenix-liveview","websocket-client"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pzingg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-22T22:05:44.000Z","updated_at":"2025-01-05T18:47:16.000Z","dependencies_parsed_at":"2025-01-23T15:25:16.519Z","dependency_job_id":"5d597936-f015-4fb3-a125-5b95c3ddabab","html_url":"https://github.com/pzingg/sandbox","commit_stats":null,"previous_names":["pzingg/sandbox"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pzingg/sandbox","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pzingg%2Fsandbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pzingg%2Fsandbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pzingg%2Fsandbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pzingg%2Fsandbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pzingg","download_url":"https://codeload.github.com/pzingg/sandbox/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pzingg%2Fsandbox/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269185042,"owners_count":24374606,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-06T02:00:09.910Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atproto","bluesky-api","bluesky-social","cid","dids","elixir","elixir-phoenix","merkle-tree","oauth2","phoenix-liveview","websocket-client"],"created_at":"2025-01-23T15:22:54.473Z","updated_at":"2025-08-07T01:25:55.385Z","avatar_url":"https://github.com/pzingg.png","language":"Elixir","readme":"# Sandbox\n\nA Phoenix 1.7 / LiveView 1.0 testbed for Bluesky.\n\nFeatures: \n\n- Elixir client for Bluesky OAuth2, app password authentication, and xrpc requests\n- Bluesky timeline view in Phoenix\n- Bluesky firehose client and parser in Elixir\n- Python script to turn a firehost commit into a Graphviz diagram of the commit tree\n\nTo start your Phoenix server:\n\n  * Run `mix setup` to install and setup dependencies\n  * Start Phoenix endpoint with `mix phx.server` or inside IEx with `iex -S mix phx.server`\n\nNow you can visit [`localhost:4000`](http://localhost:4000) from your browser.\n\nReady to run in production? Please [check our deployment guides](https://hexdocs.pm/phoenix/deployment.html).\n\n## Configuration\n\nEnable `ngrok` (must be installed on your development machine) support in config.exs:\n\n- `:ngrok_envs` - mix environments that will use ngrok\n\n```Elixir\nconfig :sandbox, ngrok_envs: [:dev, :test]\n```\n\nSet up various Bluesky settings in config.exs:\n\n- `:timezone` - to show timeline in local time instead of UTC\n- `:client_type` (default `:public`) - set to :confidential to use client assertions in OAuth client\n- `:client_scope` (default `\"atproto transition:generic\"`)\n- `:app_password_file` - file path to a JSON file for app password login\n\n```Elixir\nconfig :sandbox, Sandbox.Bluesky,\n  timezone: \"America/Los_Angeles\",\n  client_type: :public,\n  client_scope: \"atproto transition:generic\"\n  app_password_file: \"/path-to-file.json\"\n```\n\nThe JSON app password file (used for tests only) must contain these keys:\n\n- `\"did\"` - your did\n- `\"handle\"` - your \"bsky.social\" handle\n- `\"app_password\"` - your registered app password\n- `\"pds_url\"` - normally set to \"https://bsky.social\"\n\n## Learn more\n\n  * Official website: https://www.phoenixframework.org/\n  * Guides: https://hexdocs.pm/phoenix/overview.html\n  * Docs: https://hexdocs.pm/phoenix\n  * Forum: https://elixirforum.com/c/phoenix-forum\n  * Source: https://github.com/phoenixframework/phoenix\n\n## Bluesky firehose\n\nThe project has code to start a WebSocket client connection to the Bluesky\nfirehose in the `Bluesky.Firehose` modules, using native Elixir decoders\nfor CIDs and CAR \"files\".\n\nTo subscribe to the Bluesky Firehose, call this somewhere in your code:\n\n```ELixir\nSandbox.Bluesky.WebsocketClient.start(stream: :repos)\n```\n\n## Bluesky OAuth client authentication\n\nThe project has code to authenticate to a Bluesky Authorization Server\naccording to the Bluesky documentation, [here](https://atproto.com/specs/oauth) and \n[here](https://docs.bsky.app/docs/advanced-guides/oauth-client),\nusing details ported from the \n[Bluesky Python OAuth demonstration web app]( https://github.com/bluesky-social/cookbook/tree/main/python-oauth-web-app).\n\nThe code uses the `dpop` branch in a \n[fork of the OAuth2 Elixir library](https://github.com/pzingg/oauth2).\nThis fork adds new fields to the `OAuth2.Client` struct to handle the\n[Pushed Authorization Requests](https://docs.bsky.app/docs/advanced-guides/oauth-client#par) \nand [DPoP](https://docs.bsky.app/docs/advanced-guides/oauth-client#dpop) \nrequest and response headers used in Bluesky's OAuth specification.\n\nThe SandboxWeb Phoenix application provides the necessary callback, client \nmetadata and JWKS endpoints.\n\n## Bluesky app password client authentication\n\nBluesky [app password](https://lifehacker.com/tech/why-you-should-be-using-bluesky-app-passwords) \nauthentication is used in tests. See the Configuration section above for \nhow to create and configure an app password file once you have obtained \nan app password.\n\n## Bluesky timeline\n\nThe Sandbox.Feed module contains decoders for the `getTimeline` XRPC\ncall, and there is a minimal Phoenix Live View UI that displays the \ndecoded timeline.\n\n## Bluesky XRPC requests\n\nThree different modes are used for XRPC requests: unauthenticated,\nauthenticated with a Bearer access token retrieved with the \n`\"com.atproto.server.createSession\"` request, and authenticated with \na DPoP token and nonce after OAuth2 authorization.\n\n### Rate limiting errors\n\nThe code checks for rate limit errors in XRPC responses, where:\n\n- `response.status_code == 429`\n- `response.body[\"error\"] == \"RateLimitExceeded\"`\n- `response.body[\"message\"] == \"Rate Limit Exceeded\"`, and\n- `response.headers` will have a member `{\"ratelimit-reset\", [timestamp]}`\n  where timestamp is the UNIX epoch time at which the request can be resumed\n\n### Token expiration errors\n\nIf the access token for XRPC calls has expired, the response will have: \n\n- `response.status_code == 401`\n- `response.body[\"error\"] == \"invalid_token\"` and\n- `response.body[\"message\"] == \"\\\"exp\\\" claim timestamp check failed\"`\n\nCurrently, the code does not automatically attempt to refresh \nan expired access token. The \"Refresh token\" button on the \"/account\" page \ncan be used to update the acces token. (TODO!)\n\n## Bluesky Merkle Trees\n\nFrom the Bluesky documentation:\n\nAt a high level, the repository MST is a key/value mapping where the keys are \nnon-empty byte arrays, and the values are CID links to records. The MST data \nstructure should be fully reproducible from such a mapping of \nbytestrings-to-CIDs, with exactly reproducible root CID hash (aka, the \n`\"data\"` field in commit object).\n\nEvery node in the tree structure contains a set of key/CID mappings, as well \nas links to other sub-tree nodes. The entries and links are in key-sorted \norder, with all of the keys of a linked sub-tree (recursively) falling in the \nrange corresponding to the link location. The sort order is from **left** \n(lexically first) to **right** (lexically latter). Each key has a **depth** \nderived from the key itself, which determines which sub-tree it ends up in. \nThe top node in the tree contains all of the keys with the highest depth \nvalue (which for a small tree may be all depth zero, so a single node). Links\nto the left or right of the entire node, or between any two keys in the node, \npoint to a sub-tree node containing keys that fall in the corresponding key \nrange.\n\nAn empty repository with no records is represented as a single MST node with \nan empty array of entries. This is the only situation in which a tree may \ncontain an empty leaf node which does not either contain keys (\"entries\") or \npoint to a sub-tree containing entries. The top of the tree must not be a an\nempty node which only points to a sub-tree. Empty intermediate nodes are \nallowed, as long as they point to a sub-tree which does contain entries. \nIn other words, empty nodes must be pruned from the top and bottom of the \ntree, but empty intermediate nodes must be kept, such that sub-tree links \ndo not skip a level of depth. The overall structure and shape of the MST is \ndeterministic based on the current key/value content, regardless of the \nhistory of insertions and deletions that lead to the current contents.\n\nFor the atproto MST implementation, the hash algorithm used is SHA-256 \n(binary output), counting \"prefix zeros\" in 2-bit chunks, giving a fanout \nof 4. To compute the depth of a key:\n\n- hash the key (a byte array) with SHA-256, with binary output\n- count the number of leading binary zeros in the hash, and divide by two, \n  rounding down\n- the resulting positive integer is the depth of the key\n\nSome examples, with the given ASCII strings mapping to byte arrays:\n\n- `\"2653ae71\"`: depth \"0\"\n- `\"blue\"`: depth \"1\"\n- `\"app.bsky.feed.post/454397e440ec\"`: depth \"4\"\n- `\"app.bsky.feed.post/9adeb165882c\"`: depth \"8\"\n\nThere are many MST nodes in repositories, so it is important that they have\na compact binary representation, for storage efficiency. Within every node, \nkeys (byte arrays) are compressed by eliding common prefixes, with each \nentry indicating how many bytes it shares with the previous key in the array.\nThe first entry in the array for a given node must contain the full key,\nand a common prefix length of 0. This key compaction is internal to nodes, \nit does not extend across multiple nodes in the tree. The compaction scheme\nis mandatory, to ensure that the MST structure is deterministic across \nimplementations.\n\nThe `Node` IPLD schema fields are:\n\n  - `\"l\"` (\"left\", CID link, optional): link to sub-tree Node on a lower level \n    and with all keys sorting before keys at this node\n  - `\"e\"` (\"entries\", array of objects, required): ordered list of `TreeEntry` \n    objects\n\nThe `TreeEntry` schema fields are:\n\n  - `\"p\"` (\"prefixlen\", integer, required): count of bytes shared with previous \n    TreeEntry in this Node (if any)\n  - `\"k\"` (\"keysuffix\", byte array, required): remainder of key for this \n    TreeEntry, after \"prefixlen\" have been removed\n  - `\"v\"` (\"value\", CID Link, required): link to the record data (CBOR) for \n    this entry\n  - `\"t\"` (\"tree\", CID Link, optional): link to a sub-tree `Node` at a lower \n    level which has keys sorting after this `TreeEntry`'s key (to the \"right\"),\n    but before the next `TreeEntry`'s key in this `Node` (if any)\n\nWhen parsing MST data structures, the depth and sort order of keys should be \nverified. This is particularly true for untrusted inputs, but is simplest to \njust verify every time. Additional checks on node size and other parameters \nof the tree structure also need to be limited.\n  \n```elixir\ndefmodule Node do\n  @moduledoc \"\"\"\n    - `:l` - left (CID | nil)\n    - `:e` - entries (list(TreeEntry))\n  \"\"\"\n\n  defstruct [:l, :e]\nend\n\ndefmodule TreeEntry do \n  @moduledoc \"\"\"\n    - `:p` - prefixlen (u64)\n    - `:k` - keysuffix (binary)\n    - `:v` - value (CID)\n    - `:t` - tree (CID | nil)\n\n  Examples of `:k`\n\n  `\"app.bsky.feed.post/3laf7splhud26\"`\n  `\"b25ndt3qc2m\"`\n  \"\"\"\n  \n  defstruct [:p, :k, :v, :t]\nend\n\n\n```","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpzingg%2Fsandbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpzingg%2Fsandbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpzingg%2Fsandbox/lists"}