{"id":13706903,"url":"https://github.com/MicahElliott/dbdoc","last_synced_at":"2025-05-05T23:31:53.410Z","repository":{"id":138841577,"uuid":"336058576","full_name":"MicahElliott/dbdoc","owner":"MicahElliott","description":"Document your database schema, because your team will thank you, and a single text file makes it easy. Works well with PostgreSQL and others.","archived":false,"fork":false,"pushed_at":"2024-12-31T20:59:26.000Z","size":833,"stargazers_count":44,"open_issues_count":4,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-26T14:14:01.486Z","etag":null,"topics":["cockroachdb","datagrip","db2-database","dbeaver","dbms","documentation","migrations","oracle-database","pgadmin","postgresql","schema-design","snowflakedb","sql"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MicahElliott.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-02-04T19:19:01.000Z","updated_at":"2025-01-18T10:07:15.000Z","dependencies_parsed_at":"2024-01-14T20:29:58.298Z","dependency_job_id":null,"html_url":"https://github.com/MicahElliott/dbdoc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MicahElliott%2Fdbdoc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MicahElliott%2Fdbdoc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MicahElliott%2Fdbdoc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MicahElliott%2Fdbdoc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MicahElliott","download_url":"https://codeload.github.com/MicahElliott/dbdoc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252593178,"owners_count":21773422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cockroachdb","datagrip","db2-database","dbeaver","dbms","documentation","migrations","oracle-database","pgadmin","postgresql","schema-design","snowflakedb","sql"],"created_at":"2024-08-02T22:01:11.868Z","updated_at":"2025-05-05T23:31:53.400Z","avatar_url":"https://github.com/MicahElliott.png","language":"Clojure","readme":"#+Title: DBDoc (Database Schema Documenter)\n\nDocument your database schema (tables and columns), because your team will\nthank you, and this makes it easy. You really need a [[https://en.wikipedia.org/wiki/Data_dictionary][Data Dictionary]] to\ndescribe your database. Your team will be living in chaos until you have one.\nDBDoc gives you a simple one that everyone can easily work with.\n\nDBDoc enables you to describe (via generated =COMMENT ON= statements) your\nrelational database schema(s) in a simple text file, which is easy for\ndevelopers/DBAs to edit and search in the repo. The docs (each a snippet of a\nsentence or a few, all in a single, versatile =dbdoc.org= file) are then:\n\n- *viewable in an SQL client* like DBeaver or Datagrip (the main use\n  case) as tooltips and in other views\n\n- *greppable* in your code base\n\n- *web-publishable as docs*, enabling other\n  stakeholders (eg, Product people) to view DB documentation (in\n  Confluence or wherever)\n\n- *presentable* one table at a time for brainstorming, explaining, etc,\n  with an org presenter, like [[https://github.com/eschulte/epresent][epresent]] or [[https://github.com/takaxp/org-tree-slide][org-tree-slide]] (screenshot\n  at bottom)\n\nIt works by running a simple =dbdoc= script to convert a very minimal\nORG-valid and -prescribed syntax (top-level bullets, single paragraphs, and\ndefinition lists) into (long) [[https://www.postgresql.org/docs/current/sql-comment.html][SQL =COMMENT ON= statements]], which can be run\nautomatically on your database (via migration or sourced or however you like).\n\n#+html: \u003cp align=\"center\"\u003e\u003cimg src=\"film-hover.png\" title=\"DBeaver showing a hover on column/heading\" /\u003e\u003c/p\u003e\n#+caption: DBeaver showing a hover on column/heading\n\n_(See far below for more screenshots.)_\n\n** Changelog\n\n*** *2023-09-12* (all postgres-only)\n- Support non-=public= schemas (use =someschema.sometable= for top-level bullet)\n- Round-tripper: see data in DB not in ORG file\n- Round-tripper: see conflicts between DB and ORG file\n- Error checking for duplicate tables and fields in ORG file (common when\n  pasting from round-tripping)\n\n** Example dbdoc.org file\n\nThe following shows an example =dbdoc.org= file describing a movie store\nrental database and a few of its tables: =film= (containing =title=\nand =description= columns), =movie= (a deprecated table with no\ncolumns documented), and =actor=. Notice: the hyphens instead of\nunderscores, newlines before definitions, other indentation.\n\nAn example translation then is from:\n\n#+begin_src org\n#+Title: Pagila Movie Store Rental Database\n\nThis is the \"dbdoc\" description file for the Pagila database. See the\n[[https://github.com/micahelliott/dbdoc][dbdoc README]]\nfor more detailed instructions on its purpose and expanding it. This file\ncontains short documentation for any tables and columns that could use\neven the slightest bit of explanation.\n\nEdit this file whenever you make schema changes. And be a good citizen\nby helping to grow this file any time you're touching a table!\nThe remainder of this file will be used processed into comment\ndescriptions that will be visible in your SQL client, and can also be\nexported as HTML.\n\n* FILM\n\nA film, aka movie, is released initially in theaters, and then\navailable to movie /stores/, at which point they become available to\nthe DB.\n\n- title ::\n  The full name of the film, including things like sub-title and part\n  in a series; does not include language\n\n- description ::\n  A brief synopsis (catchy prose) about the plot\n\n* MOVIE\n\nDEPRECATED: replaced by =film=\n\n* ACTOR\n\nAn actor is very simple and non-comprehensive table to record the main\nheadlining /stars/ of the film. All fields are obvious. Note that\nthere may be duplicate actors that use slightly different names on occasion.\n#+end_src\n\nto an SQL migration file containing:\n\n#+begin_src sql\nCOMMENT ON TABLE film IS 'A film, aka movie …';\nCOMMENT ON COLUMN film.title IS 'The full name …';\n…\nCOMMENT ON TABLE movie IS 'DEPRECATED: replaced …';\n…\n#+end_src\n\nCompared to the ORG version, that SQL is pretty ugly – editing\n(quoting, line-length/newlines, indentation, formatting) becomes quite\ndifficult. That’s why this tiny tool exists.\n\nThere is a testable =docs/dbdoc.org= example (and its generated SQL\nmigration file =resources/migrations/20210804162056-dbdoc.up.sql=) in\nthis repo that was written to minimally describe the [[https://github.com/devrimgunduz/pagila][pagila toy\ndatabase]]. Just run =dbdoc= in the root of this repo to try it out!\n\n** Installation\n\n- Install [[https://github.com/babashka/babashka#installation][Babashka]] (any OS, tiny, fast, no dependencies).\n- Clone this repo and put its root on your =PATH=.\n\nOR\n\n- Use [[https://github.com/babashka/bbin][bbin]] to install `dbdoc` as an\n  executable into =~/.local/bin=:\n\n  #+begin_src shell\n  bbin install io.github.micahelliott/dbdoc\n  #+end_src\n\nNow you're ready to run =dbdoc= from anywhere, and that's all\nthere is to it! Not even any CLI options. :)\n\n** Documentation Process (Usage)\n\n*** One time only\n\n- Create a single living .org file in your repo, eg, =docs/dbdoc.org=\n  for growing docs for your tables.\n\n- Assuming you haven't already somehow written a =COMMENT= for your\n  DB, turn a SME analyst type or long-time developer or DBA in your\n  company loose to write up a bunch of notes in the org file. Then\n  edit a bit to ensure it's valid ORG that DBDoc can handle..\n\n- Set up env vars to change default file locations (optional, not well\n  tested):\n\n  #+begin_src shell\n  export DBDOC_ORG=docs/dbdoc.org\n  export DBDOC_SQL=resouces/migrations/\u003ctimestamp\u003e-dbdoc.up.sql\n  export DBDOC_HTML=docs/dbdoc.html\n  #+end_src\n\n*** Continually (this is the only real process)\n\n1. Keep describing as many tables and columns as you see fit in your\n   =docs/dbdoc.org= file. Every time a developer changes or adds a\n   field or table, they also should put a sentence or two describing its\n   purpose in the org file.\n\n2. Run =dbdoc= to generate a time-stamped file like\n   =resources/migrations/20201027000000-dbdoc.up.sql=. IMPORTANT!!\n   Don't forget this step! (You don't need all the developers on the\n   teams do this, so long as /someone/ does the generation/migrating\n   once in a while.)\n\n3. Commit both the org and migration files.\n\n*** Optional\n\n- Generate HTML (from command line [[https://pandoc.org/][with Pandoc]] or [[https://stackoverflow.com/a/22091045/326516][Emacs]]) and publish\n  the new version to some site your company views (optional, see\n  =org2conflu.zsh= script).\n\n- If your migrations aren't automatic as part of your CI, run your\n  migration (or just load the new SQL file if you don't do\n  migrations).\n\n** Table Documentation Best Practices\n\n- Don’t need to be comprehensive and document every field when names\n  make them obvious\n- Add an example datum for a column\n- Used-by references: other tables (probably not FKs) and code areas\n- Gotchas/quirks\n- Add characteristic tags: deprecated/defunct, xl, hot, new, static,\n  performance, donttouch, dragons\n\n** Showing Comments in Clients\n\n- psql: =\\d+=\n- [[https://dataedo.com/kb/tools/dbeaver/how-to-view-and-edit-table-and-column-comments][dbeaver]] (HIGHLY RECOMMENDED!! the docs pop up everywhere)\n- [[https://eggerapps.at/postico/][postico]] (see the _Structure_ tab, as shown is screenshot)\n- [[https://postgrest.org/en/v7.0.0/api.html#openapi-support][postgrest/swagger]]\n- [[https://dataedo.com/kb/tools/pgadmin/how-to-view-and-edit-table-and-column-comments][pgadmin]]\n- [[https://dataedo.com/kb/tools/datagrip/how-to-view-and-edit-table-and-column-comments][datagrip]] ([[https://stackoverflow.com/questions/66129447/how-to-show-column-and-table-comment-in-jetbrains-datagrip][how to enable]])\n\n** Read on if you want more details...\n\n*** Transformations\n\nThe parser is limited and rigid and wants to see a _table_\ndescription paragraph for every table you wish to document. So, if you\nwant to document some column in a table, you must also provide at\nleast a tidbit sentence for the table too. It's not a robust parser so\njust be careful. Alignment/indentation is important too, so follow the\nexample format precisely – this is a tiny subset of actual org.\n\nOrg uses underscores for italic, and it’s tedious enough to have to\nwrap every DB entity in equals (+=+) in org to escape them, so they\nshould instead be documented with hyphens ( =-= ) (though this isn’t\nrequired). IOW, all ORG hyphenated variables (eg, =my-var-name=)\nbecome underscores in SQL (=my_var_name=). So prefer to use\n=my-var-name= in the ORG description.\n\nIt you use “straight” apostrophes ('), they’ll be converted to\ncurlies so as not to need SQL string escaping (and be prettier).\n\n*** Git Diffs\n\nThe first version of your migration file is a direct mapping from =dbdoc.org=:\nit contains a =COMMENT ON= for each description. Then each time you run\n=dbdoc=, that migration file is maintained but renamed and always has a 1-to-1\nmapping of org descriptions to =COMMENT ON=.\n\nThe =dbdoc= script looks for an old migration file called\n=\u003ctimestamp\u003e-dbdoc.up.sql= and renames it (via =git-move=) to a\npresent timestamp. This enables Git to see the the new migration as\nsimply a change from the last run, and so you can easily see the\nbefore/after diff. This also saves on a clutter of generating a bunch\nof extra migrations.\n\n*** Doc Coverage\n\nYou can track progress of your documenting by noting how many tables\nhave or have not been covered. Use the =coverage.zsh= script to offer\na simple coverage report.\n\n*** Seeding an ORG doc file for first-time use\n\nYou can create a listing of all existing tables as a starter\nORG file: see =schema2org.zsh=. Once created, you can just start\ndocumenting! This is probably totally buggy; it's a tiny sed script\nworking off a pg-dump.\n\nThis may be improved to populate with existing comment descriptions\nto enable “round-tripping”.\n\n*** Round-Tripping (postgres only, for now)\n\nIf you already have comments on your tables, you can pull them into your ORG\ndoc (semi-manually) to still get the benefits of shared editing/viewing. So if\nsome of your team happens to add comments (inside a client, or with =COMMENT\nON= statements) to your production DB (instead of the using dbdoc process),\n/round-tripping/ ensures you never lose data, keeping your =dbcoc.org= as the\nSPOT and synced with the DB. But encourage your teammates not to be writing\n=COMMENT ON= statements and use dbdoc instead!\n\nTo run the round-tripper, dbdoc needs access to an up-to-date, running DB\ninstance. Export the =PGDATABASE= env var to specify that DB. It will query\nfor all the descriptions and send them into a TSV =indb.tsv=. Then it converts\nthe existing =dbdoc.org= texts (as inorg.tsv) to be able to diff and determine\nwhat's new. Run =roundtrip.zsh= to see it.\n\n#+begin_src shell\nPGDATABASE=mydb roundtrip.zsh \u003e\u003edocs/dbdoc.org # careful here with the append!\n#+end_src\n\nThat output contains org formatted text. Rather than dbdoc trying to inject\nthe new text into your hand-crafted =dbdoc.org= doc, it simply prints the new\ndata in org-format to /stdout/, so that you can paste it into the appropriate\nplaces in your =dbdoc.org= file (or just append it as per that example). It is\nalphabetized, so simply appending may not be wanted if you're trying to keep\nyour =dbdoc.org= file sorted by table name.\n\nIf there are conflicts (same field described in both ORG and DB), those are\nWARNINGs printed to /stderr/, and you're expected to resolve and paste them into\nyour =dbdoc.org= file with the description you feel is most up-to-date.\n\n*** FAQs\n\n*Why use org instead of the more popular/common markdown?*\n\nORG has definition lists which work great for column docs. For the\nlimited syntax that is DBDoc, org and md are effectively the same\n(just use =*= for heading instead of =#=).\n\nBut [[https://github.com/MicahElliott/dbdoc/issues/2][I will implement Markdown]] if anyone feels they need it.\n\n*Do I need Emacs to work with Org files?*\n\nNo! Emacs is not required to for any part of DBDoc. Most common\neditors have some proper way to work with Org. Even if yours doesn't,\njust edit in plain text mode.\n\n*How far should I go with documenting my tables?*\n\nNot super far. See recommendations above. I like to limit column docs\nto not more than a few sentences. A table doc can be a legthy\nparagraph (only one!). Your source code docstrings are probably a\nbetter place to get into the nitty gritty.\n\n*Why not just write the doc strings in SQL?*\n\nThen your editor would think you’re in SQL mode and wouldn’t do things\nlike spell-checking or nice formatting. Plus, using ORG gives you a\npublishable HTML version of your docs.\n\n*Does this work for all databases?*\n\nIt does work for many! It's been tested with PostgreSQL, and should\nwork with others too, such as:\n\n- [[https://www.cockroachlabs.com/docs/stable/comment-on.html][CockroachDB]]\n- [[https://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_4009.htm][Oracle]]\n- [[https://www.ibm.com/support/producthub/db2/docs/content/SSEPGG_11.5.0/com.ibm.db2.luw.sql.ref.doc/doc/r0000901.html][IBM DB2]]\n- [[https://docs.snowflake.com/en/sql-reference/sql/comment.html][Snowflake]]\n- [[https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Statements/COMMENT/COMMENTONTABLE.htm?tocpath=SQL%20Reference%20Manual%7CSQL%20Statements%7CCOMMENT%C2%A0ON%C2%A0Statements%7C_____9][Vertica]]\n\n[[https://issues.apache.org/jira/browse/DERBY-7008][Apache Derby may get support]].\n\n[[https://stackoverflow.com/questions/7426205/sqlite-adding-comments-to-tables-and-columns][I don't think SQLite supports =COMMENT=.]] And [[https://stackoverflow.com/questions/2162420/alter-mysql-table-to-add-comments-on-columns][MySQL makes it very\ndifficult]] (and [[https://stackoverflow.com/questions/58665398/modifing-comment-into-spark-table-on-databricks][Spark]]) to the point that DBDoc won't attempt to make it\nwork. [[https://feedback.azure.com/forums/307516-sql-data-warehouse/suggestions/16317988-table-extended-properties][SQL Server/Azure is a fail too]]. And [[https://community.cloudera.com/t5/Support-Questions/Is-there-way-to-add-comment-to-a-phoenix-table/td-p/165405][Ignite]].\n\n*How do I get this into Confluence without API access?*\n\nYour Confluence setup might only support creating a page from markdown\n(not org or html). So you can use pandoc to convert from org to md\nwith: =pandoc -s docs/dbdoc.org -o temp.md= and then paste it into\nConfluence from its \"plus\" menu while editing a page:\n/Markup \u003e Markdown \u003e Paste \u003e Insert/\n\n*Why can't I use just my SQL client to add descriptive comments?*\n\nBecause it seems wrong. Which copy of your DB are you wanting to\nmodify? Are you connecting your client to a production DB and making\nedits to prod data? This doesn't make sense to me and I don't\nunderstand why SQL clients support =COMMENT= editing. Developers, DBAs,\nQA, and others may not have prod access, and probably all need\ndifferent non-prod DBs to have up-to-date documentation at their\nfingertips, and DBDoc enables putting that documentation into every\ninstance.\n\n** Similar Tools Comparison\n\n*** dbdocs (same name but plural!)\n\n[[https://dbdocs.io/][dbdocs]] (plural) is decsribed as: \"A free \u0026 simple tool to create\nweb-based database documentation using DSL code. Designed for\ndevelopers. Integrate seamlessly with your development workflow.\" As a\nfull DDL DSL, it is a much heavier commitment to incorporate. It also\ncreates a rich website for your tables, whereas /DBDoc/ just creates a\nsingle webpage that can be synced with Confluence or published\nwherever you choose. dbdocs creates ERDs, but /DBDoc/ lets capable\nclients like DBeaver handle that for you.\n\n*** Rails ActiveRecord\n\nThe [[https://github.com/rails/rails/pull/22911][ActiveRecord ORM]] has the ability to support comments as part of a\nschema definition and migration syntax. You may not need DBDoc if\nyou're using AR. But if you want to publish your schema documentation,\nyou should still use DBDoc!\n\n*** Commercial Tools\n\nThere are many DB documentation tools in this realm. For any use cases\nI've encountered, they are overkill. But if you're interested in much\nmore sophisticated kitchen sink tools that may work with other types\nof DBMSs, look into [[https://www.apexsql.com/sql-tools-doc.aspx][ApexSQL]], [[https://www.red-gate.com/products/sql-development/sql-doc/][Redgate]], and [[https://dataedo.com/][Dataedo]].\n\n** Future Enhancements\n\n- Support =COMMENT ON DATABASE= as top-level paragraph (but ignore\n  myriad other types). *Actually, this can't be done flexibly since it\n  requires knowing the DB name.*\n\n- Identify fields/tables that are missing comments\n\nPlease submit an issue if you think of any enhancements or find bugs.\nI'm eager to improve this, but need your ideas!\n\n** More Screenshots\n\nHover to see captions, just like in DB clients! There, you've been trained.\n\n#+html: \u003cp align=\"center\"\u003e\u003cimg src=\"film-table2.png\" title=\"DBeaver properties view\" /\u003e\u003c/p\u003e\n#+caption: DBeaver properties view\n\n#+html: \u003cp align=\"center\"\u003e\u003cimg src=\"postico.png\" title=\"Postico structure view with doc snippets in red\" /\u003e\u003c/p\u003e\n#+caption: Postico \"structure\" view with doc snippets in red\n\n#+html: \u003cp align=\"center\"\u003e\u003cimg src=\"datagrip1.png\" title=\"Datagrip tree table hover\" /\u003e\u003c/p\u003e\n#+caption: Datagrip tree table hover\n\n#+html: \u003cp align=\"center\"\u003e\u003cimg src=\"datagrip2.png\" title=\"Datagrip column comments\" /\u003e\u003c/p\u003e\n#+caption: Datagrip column hover\n\n#+html: \u003cp align=\"center\"\u003e\u003cimg src=\"datagrip3.png\" title=\"Datagrip tree view comments\" /\u003e\u003c/p\u003e\n#+caption: Datagrip tree view comments\n\n#+html: \u003cp align=\"center\"\u003e\u003cimg src=\"org-preso.png\" title=\"Emacs Org slide presentation view\" /\u003e\u003c/p\u003e\n#+caption: Emacs Org slide presentation view\n","funding_links":[],"categories":["Clojure"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMicahElliott%2Fdbdoc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMicahElliott%2Fdbdoc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMicahElliott%2Fdbdoc/lists"}