{"id":16299542,"url":"https://github.com/doorgan/sourceror","last_synced_at":"2025-05-14T14:08:24.870Z","repository":{"id":37894273,"uuid":"370864432","full_name":"doorgan/sourceror","owner":"doorgan","description":"Utilities to manipulate Elixir source code","archived":false,"fork":false,"pushed_at":"2025-05-08T21:23:23.000Z","size":633,"stargazers_count":344,"open_issues_count":11,"forks_count":23,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-05-08T22:19:26.886Z","etag":null,"topics":["elixir-ast","elixir-formatter","source-code"],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/doorgan.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":"dorgan","tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2021-05-26T00:42:09.000Z","updated_at":"2025-05-08T21:22:53.000Z","dependencies_parsed_at":"2024-01-05T01:06:18.149Z","dependency_job_id":"8f22c589-f4c5-4ff0-9139-1a75ac5405ce","html_url":"https://github.com/doorgan/sourceror","commit_stats":{"total_commits":275,"total_committers":17,"mean_commits":"16.176470588235293","dds":"0.21090909090909093","last_synced_commit":"95ba89d541aa09dc970a309409023fcce81e0841"},"previous_names":[],"tags_count":49,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doorgan%2Fsourceror","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doorgan%2Fsourceror/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doorgan%2Fsourceror/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doorgan%2Fsourceror/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/doorgan","download_url":"https://codeload.github.com/doorgan/sourceror/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254159769,"owners_count":22024564,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir-ast","elixir-formatter","source-code"],"created_at":"2024-10-10T20:48:26.885Z","updated_at":"2025-05-14T14:08:24.864Z","avatar_url":"https://github.com/doorgan.png","language":"Erlang","funding_links":["https://ko-fi.com/dorgan"],"categories":[],"sub_categories":[],"readme":"# Sourceror 🧙\n\n![Github\nActions](https://github.com/doorgan/sourceror/actions/workflows/main.yml/badge.svg?branch=main)\n[![Coverage\nStatus](https://coveralls.io/repos/github/doorgan/sourceror/badge.svg?branch=main)](https://coveralls.io/github/doorgan/sourceror?branch=main)\n[![Module\nVersion](https://img.shields.io/hexpm/v/sourceror.svg)](https://hex.pm/packages/sourceror)\n[![Hex\nDocs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/sourceror/)\n[![Total\nDownload](https://img.shields.io/hexpm/dt/sourceror.svg)](https://hex.pm/packages/sourceror)\n[![License](https://img.shields.io/hexpm/l/sourceror.svg)](https://github.com/doorgan/sourceror/blob/master/LICENSE)\n[![Last\nUpdated](https://img.shields.io/github/last-commit/doorgan/sourceror.svg)](https://github.com/doorgan/sourceror/commits/master)\n\n\u003c!-- MDOC !--\u003e\n\nUtilities to work with Elixir source code.\n\n## Installation\n\nAdd `:sourceror` as a dependency to your project's `mix.exs`:\n\n```elixir\ndefp deps do\n  [\n    {:sourceror, \"~\u003e 1.10\"}\n  ]\nend\n```\n\n### A note on compatibility\n\nSourceror is compatible with Elixir versions down to 1.10 and OTP 21. For Elixir\nversions prior to 1.13 it uses a vendored version of the Elixir parser and\nformatter modules. This means that for Elixir versions prior to 1.12 it will\nsuccessfully parse the new syntax for stepped ranges instead of raising a\n`SyntaxError`, but everything else should work as expected.\n\n## Goals of the library\n\n- Be as close as possible to the standard Elixir AST.\n- Make working with comments as simple as possible.\n- No dev/prod dependencies, to simplify integration with other tools.\n\n## Sourceror's AST\n\nHaving the AST and comments as separate entities allows Elixir to expose the\ncode formatting utilities without making any changes to it's AST, but also\ndelegates the task of figuring out what's the most appropriate way to work with\nthem to us.\n\nSourceror's take is to use the node metadata to store the comments. This allows\nus to work with an AST that is as close to regular elixir AST as possible. It\nalso allows you to move nodes around without worrying about leaving a comment\nbehind and ending up with misplaced comments.\n\nTwo metadata fields are added to the regular Elixir AST:\n\n- `:leading_comments` - holds the comments directly above the node or are in the\n  same line as it. For example:\n\n  ```elixir\n  test \"parses leading comments\" do\n    quoted = \"\"\"\n    # Comment for :a\n    :a # Also a comment for :a\n    \"\"\" |\u003e Sourceror.parse_string!()\n\n    assert {:__block__, meta, [:a]} = quoted\n    assert meta[:leading_comments] == [\n      %{line: 1, column: 1, previous_eol_count: 1, next_eol_count: 1, text: \"# Comment for :a\"},\n      %{line: 2, column: 4, previous_eol_count: 0, next_eol_count: 1, text: \"# Also a comment for :a\"},\n    ]\n  end\n  ```\n\n- `:trailing_comments` - holds the comments that are inside of the node, but\n  aren't leading any children, for example:\n\n  ```elixir\n  test \"parses trailing comments\" do\n    quoted = \"\"\"\n    def foo() do\n    :ok\n    # A trailing comment\n    end # Not a trailing comment for :foo\n    \"\"\" |\u003e Sourceror.parse_string!()\n\n    assert {:__block__, block_meta, [{:def, meta, _}]} = quoted\n    assert [%{line: 3, text: \"# A trailing comment\"}] = meta[:trailing_comments]\n    assert [%{line: 4, text: \"# Not a trailing comment for :foo\"}] = block_meta[:trailing_comments]\n  end\n  ```\n\nNote that Sourceror considers leading comments to the ones that are found in the\nsame line as a node, and trailing comments to the ones that are found before the\nending line of a node, based on the `end`, `closing` or `end_of_expression`\nline. This also makes the Sourceror AST consistent with the way the Elixir\nformatter works, making it easier to reason about how a given AST would be\nformatted.\n\n## Traversing the AST\n\nElixir provides the `Macro.prewalk`, `Macro.postwalk` and `Macro.traverse`\nfunctions to traverse the AST. You can use the same functions to traverse the\nSourceror AST as well, since it has the same shape as the standard Elixir AST.\n\nSourceror also provides the `Sourceror.prewalk`, `Sourceror.postwalk` and\n`Sourceror.traverse` variants. At the time of writing they are mostly wrappers\naround the standard Elixir functions for AST traversal, but they may be enhanced\nin the future if more AST formats are introduced.\n\nIn addition to these, Sourceror also provides a Zipper implementation for the\nElixir AST. You can learn more about it in the [Zippers\nnotebook](https://hexdocs.pm/sourceror/zippers.html).\n\n## Patching the source code\n\nYou can use Sourceror to manipulate the AST and turn it back into human readable\nElixir code, this is commonly known as writing a \"codemod\". For example, you can\nwrite a codemod to replace calls to `String.to_atom` to\n`String.to_existing_atom`:\n\n```elixir\ntest \"updates the source code\" do\n  source =\n    \"\"\"\n    String.to_atom(foo)\\\n    \"\"\"\n\n  new_source =\n    source\n    |\u003e Sourceror.parse_string!()\n    |\u003e Macro.postwalk(fn\n      {{:., dot_meta, [{:__aliases__, alias_meta, [:String]}, :to_atom]}, call_meta, args} -\u003e\n        {{:., dot_meta, [{:__aliases__, alias_meta, [:String]}, :to_existing_atom]}, call_meta, args}\n\n      quoted -\u003e\n        quoted\n    end)\n    |\u003e Sourceror.to_string()\n\n  assert new_source ==\n    \"\"\"\n    String.to_existing_atom(foo)\\\n    \"\"\"\nend\n```\n\nHowever, this will affect the whole source code, as we are working on the full\nsource AST. Sourceror relies on the Elixir formatter to produce human readable\ncode, so the original code formatting will be lost by using\n`Sourceror.to_string`. If your code is already using the Elixir formatter then\nthis won't be an issue, but it will be an undesirable effect if you're not using\nit.\n\nAn alternative to this is to use Patches instead. A patch is a data structure\nthat specifies the text range that should be replaced, and either a replacement\nstring or a function that takes the text in that range and produces a string\nreplacement.\n\nUsing patches, we could do the same as above, but produce a patch instead of\nmodifying the AST. As a result, only the parts that need to be changed will be\naffected, and the rest of the code keeps the original formatting:\n\n```elixir\ntest \"patches the source code\" do\n  source =\n    \"\"\"\n    case foo do\n      nil -\u003e         :bar\n      _ -\u003e\n\n          String.to_atom(foo)\n\n          end\\\n    \"\"\"\n\n  {_quoted, patches} =\n    source\n    |\u003e Sourceror.parse_string!()\n    |\u003e Macro.postwalk([], fn\n      {{:., dot_meta, [{:__aliases__, alias_meta, [:String]}, :to_atom]}, call_meta, args} = quoted, patches -\u003e\n        range = Sourceror.get_range(quoted)\n        replacement =\n          {{:., dot_meta, [{:__aliases__, alias_meta, [:String]}, :to_existing_atom]}, call_meta, args}\n          |\u003e Sourceror.to_string()\n\n        patch = %{range: range, change: replacement}\n        {quoted, [patch | patches]}\n\n      quoted, patches -\u003e\n        {quoted, patches}\n    end)\n\n  assert Sourceror.patch_string(source, patches) ==\n    \"\"\"\n    case foo do\n      nil -\u003e         :bar\n      _ -\u003e\n\n          String.to_existing_atom(foo)\n\n          end\\\n    \"\"\"\nend\n```\n\nYou have to keep in mind that:\n\n1. If you patch a node that has inner code, like replacing a full `case`, then\n   the contents of the node will be reformatted as well.\n\n2. At the moment, Sourceror won't check for conflicts in the patches ranges, so\n   care needs to be taken to not produce conflicting patches. You may need to do\n   a number of `parse -\u003e patch -\u003e reparse` if you find yourself generating\n   conflicting patches.\n\nSome of the most common patching operations are available in the\n[Sourceror.Patch](https://hexdocs.pm/sourceror/Sourceror.Patch.html) module\n\n## Background\n\nThere have been several attempts at source code manipulation in the Elixir\ncommunity. Thanks to its metaprogramming features, Elixir provides builtin tools\nthat let us get the AST of any Elixir code, but when it comes to turning the AST\nback to code as text, we had limited options. `Macro.to_string/2` is a thing,\nbut the produced code is generally ugly, mostly because of the extra parenthesis\nor because it turns string interpolations into calls to erlang modules, to name\nsome examples. This meant that, even if we could use `Macro.to_string/2` to get\na string and then give that to the Elixir formatter `Code.format_string!/2`, the\noutput would still be suboptimal, as the formatter is not designed to change the\nsemantics of the code, only to pretty print it. For example, call to erlang\nmodules would be kept as is instead of being turned back to interpolations.\n\nWe also had the additional problem of comments being discarded by the tokenizer,\nand literals not having information like line numbers or delimiter characters.\nThis makes the regular AST too lossy to be useful if what we want is to\nmanipulate the source code, because we need as much information as possible to\nbe able to stay as close to the source as possible. There have been several\nproposal in the past to bring all this information to the Elixir AST, but they\nall meant a change that would either break macros due to the addition of new\ntypes of AST nodes, or making a compromise in core Elixir itself by storing\ncomments in the nods metadata. [This\ndiscussion](https://groups.google.com/u/1/g/elixir-lang-core/c/GM0yM5Su1Zc/m/poIKsiEVDQAJ)\nin the Elixir mailing list highlights the various issues faced when deciding if\nand how the comments would be preserved. Arjan Scherpenisse also did a\n[talk](https://www.youtube.com/watch?v=aM0BLWgr0g4\u0026t=117s) where he discusses\nabout the problems of using the standard Elixir AST to build refactoring tools.\n\nDespite of all these issues, the Elixir formatter is still capable of\nmanipulating the source code to pretty print it. Under the hood it does some\nneat tricks to have all this information available: on one hand, it tells the\ntokenizer to extract the comments from the source code and keep it at hand(not\nin the AST itself, but as a separate data structure), and on the other hand it\ntells the parser to wrap literals in block nodes so metadata can be preserved.\nOnce it has all it needs, it can start converting the AST and comments into an\nalgebra document, and ultimately convert that to a string. This functionality\nwas private, and if we wanted to do it ourselves we would have to replicate or\nvendor the Elixir formatter with all its more than 2000 lines of code. This\napproach was explored by Wojtek Mach in\n[wojtekmach/fix](https://github.com/wojtekmach/fix), but it involved vendoring\nthe elixir Formatter code, was tightly coupled to the formatting process, and\nany change in Elixir would break the code.\n\nSince Elixir 1.13 this functionality from the formatter was finally exposed via\nthe `Code.string_to_quoted_with_comments/2` and `Code.quoted_to_algebra/2`\nfunctions. The former gives us access to the list of comments in a shape the\nElixir formatter is able to use, and the latter lets us turn _any arbitrary\nElixir AST_ into an algebra document. If we also give it the list of comments,\nit will merge them together, allowing us to format AST _and_ preserve the\ncomments. Now all we need to care about is of manipulating the AST, and let the\nformatter do the rest.\n\n\u003c!-- MDOC !--\u003e\n\n## Documentation\n\nYou can find Sourceror documentation on [Hex\nDocs](https://hexdocs.pm/sourceror/readme.html).\n\n## Examples\n\nYou can find usage examples in the `examples` folder. You can run them by\ncloning the repo and running `elixir examples/\u003cexample_file\u003e.exs`.\n\nYou can also find documented examples you can run with\n[Livebook](https://github.com/elixir-nx/livebook) in the `notebooks` folder.\n\n## Contributing\n\nIf you want to contribute to Sourceror, please check our\n[Contributing](https://github.com/doorgan/sourceror/blob/master/CONTRIBUTING.md)\nsection for pointers.\n\n## Getting assistance\n\nIf you have any questions about Sourceror or need assistance, please open a\nthread in the [Discussions](https://github.com/doorgan/sourceror/discussions)\nsection.\n\n## Copyright and License\n\nCopyright (c) 2021 dorgandash@gmail.com\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use\nthis file except in compliance with the License. You may obtain a copy of the\nLicense at\n[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)\n\nUnless required by applicable law or agreed to in writing, software distributed\nunder the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR\nCONDITIONS OF ANY KIND, either express or implied. See the License for the\nspecific language governing permissions and limitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoorgan%2Fsourceror","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoorgan%2Fsourceror","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoorgan%2Fsourceror/lists"}