{"id":20529378,"url":"https://github.com/jlevy/frontmatter-format","last_synced_at":"2025-04-14T05:08:30.346Z","repository":{"id":259505472,"uuid":"877700412","full_name":"jlevy/frontmatter-format","owner":"jlevy","description":"A micro-format for YAML metadata on any file","archived":false,"fork":false,"pushed_at":"2025-03-29T20:34:19.000Z","size":534,"stargazers_count":16,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-14T05:08:10.744Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jlevy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-24T05:07:16.000Z","updated_at":"2025-03-29T20:33:29.000Z","dependencies_parsed_at":"2025-04-11T22:52:53.224Z","dependency_job_id":"b9e449c5-7be3-481f-a910-60c359064d66","html_url":"https://github.com/jlevy/frontmatter-format","commit_stats":null,"previous_names":["jlevy/frontmatter-format"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlevy%2Ffrontmatter-format","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlevy%2Ffrontmatter-format/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlevy%2Ffrontmatter-format/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlevy%2Ffrontmatter-format/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jlevy","download_url":"https://codeload.github.com/jlevy/frontmatter-format/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248824682,"owners_count":21167345,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T23:31:26.829Z","updated_at":"2025-04-14T05:08:30.339Z","avatar_url":"https://github.com/jlevy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Frontmatter Format\n\n## Motivation\n\nSimple, readable metadata attached to files can be useful in numerous situations, such\nas recording title, author, source, copyright, or the provenance of a file.\n\nUnfortunately, it's often unclear how to format such metadata consistently across\ndifferent file types while preserving valid syntax, making parsing easy, and not breaking\ninteroperability with existing tools.\n\n**Frontmatter format** is a way to add metadata as frontmatter on any file.\nIt is basically a micro-format: a simple set of conventions to put structured metadata\nas YAML at the top of a file in a syntax that is broadly compatible with programming\nlanguages, browsers, editors, and other tools.\n\nFrontmatter format specifies a syntax for the metadata as a comment block at the top of\na file. This approach works while ensuring the file remains valid Markdown, HTML, CSS,\nPython, C/C++, Rust, SQL, or most other text formats.\n\nFrontmatter format is a generalization of the common format for frontmatter used by\nJekyll and other CMSs for Markdown files.\nIn that format, frontmatter is enclosed in lines containing `---` delimiters.\n\nIn this generalized format, we allow several styles of frontmatter demarcation, with the\nfirst line of the file indicating the format and style.\n\nThis repository is a **description of the format** and an easy-to-use **reference\nimplementation**. The implementation is in Python but the format is very simple and easy\nto implement in any language.\n\nThis readme aims to explain the format so anyone can use it and encourage the adoption\nof the format, especially for workflows around text documents that are becoming\nincreasingly common in AI tools and pipelines.\n\n## Examples\n\nSome simple examples:\n\n```markdown\n---\ntitle: Sample Markdown File\nstate: draft\ncreated_at: 2022-08-07 00:00:00\ntags:\n  - yaml\n  - examples\n# This is a YAML comment, so ignored.\n---\nHello, *World*!\n```\n\n```html\n\u003c!---\ntitle: Sample HTML File\n---\u003e\nHello, \u003ci\u003eWorld\u003c/i\u003e!\n```\n\n```python\n#---\n# author: Jane Doe\n# description: A sample Python script\n#---\nprint(\"Hello, World!\")\n```\n\n```css\n/*---\nfilename: styles.css\n---*/\n.hello {\n  color: green;\n}\n```\n\n```sql\n----\n-- title: Sample SQL Script\n----\nSELECT * FROM world;\n```\n\nNote that a few scripts like \"shebang\"-style shell scripts or Python scripts with inline\ndependencies require a first line in a different format.\nThis is allowed as long as these `#`-commented lines precede the initial delimiter\n`#---`:\n\n```python\n# /// script\n# requires-python = \"\u003e=3.12\"\n# dependencies = []\n# ///\n#---\n# title: An Example Python Script\n# description: This Script uses PEP 723 style inline dependencies.\n#---\n\ntype Point = tuple[float, float]\nprint(Point)\n```\n\nHere's an example of a richer metadata in use, from a tool that does video\ntranscription. You can see how it's useful having a simple and clear format for title,\ndescription, history, source of the content, etc.\n\n![Credit for video to @KBoges on YouTube](images/example.png)\n\n## Advantages of this Approach\n\n- **Compatible with existing syntax:** By choosing a style for the metadata consistent\n  with any given file, it generally doesn't break existing tools.\n  Almost every language has a style for which frontmatter works as a comment.\n\n- **Auto-detectable format:** Frontmatter and its format can be recognized by the first\n  few bytes of the file.\n  That means it's possible to detect metadata and parse it automatically.\n\n- **Metadata is optional:** Files with or without metadata can be read with the same\n  tools. So it's easy to roll out metadata into files gracefully, as needed file by file.\n\n- **YAML syntax:** JSON, YAML, XML, and TOML are all used for metadata in some\n  situations. YAML is the best choice here because it is already in widespread use with\n  Markdown, is a superset of JSON (in case an application wishes to use pure JSON), and\n  is easy to read and edit manually.\n\n## Format Definition\n\nFrontmatter is read as a text file, one line at a time, using standard text line reading\nand UTF8 encoding.\n\nA file is in frontmatter format if the first characters are one of the following:\n\n- `---`\n\n- `\u003c!---`\n\n- `#---`\n\n- `//---`\n\n- `/*---`\n\n- `-----`\n\nand these characters are followed by a newline (`\\n`).\n\nThis line is called the *initial delimiter*.\n\nThe initial delimiter is always at the start of the file, except for a special case:\nLines at the beginning of a file are ignored if they are consecutive and begin with `#`\nbefore an initial delimiter of `#---` and a newline (`-n`). In this case, the initial\ndelimiter is the first line of the file that is `#---`.\n\nThe initial delimiter determines the *style* of the frontmatter.\nThe style specifies the matching *terminating delimiter* for the end of the frontmatter\nas well as an optional prefix (which is typically a comment character in some language).\n\nThe allowed frontmatter styles are:\n\n1. *YAML style*: delimiters `---` and `---` with no prefix on each line.\n   Useful for **text** or **Markdown** content.\n\n2. *HTML style*: delimiters `\u003c!---` and `---\u003e` with no prefix on each line.\n   Useful for **HTML** or **XML** or similar content.\n\n3. *Hash style*: delimiters `#---` and `#---` with `# ` prefix on each line.\n   Useful for **Python** or similar code content.\n   Also works for **CSV** files for some (but sadly not all) tools.\n\n4. *Slash style*: delimiters `//---` and `//---` with `// ` prefix on each line.\n   Useful for **Rust** or **C++** or similar code content.\n\n5. *Slash star style*: delimiters `/*---` and `---*/` with no prefix on each line.\n   Useful for **JavaScript**, **TypeScript**, **CSS** or **C** or similar code content.\n\n6. *Dash style*: delimiters `----` and `----` with `-- ` prefix on each line.\n   Useful for **SQL** or similar code content.\n\nRules:\n\n- The delimiters must be alone on their own lines, terminated with a newline.\n\n- Any frontmatter style is acceptable on any file.\n  When writing a file, you can choose any style, typically the one suitable to the file\n  syntax.\n\n- For all frontmatter styles, the content between the delimiters is YAML text in UTF-8\n  encoding, with an optional prefix on each line that depends on the style.\n\n- For *hash style*, *slash style*, and *dash style*, each frontmatter line begins with\n  with a *prefix* (`# `, `// `, and `-- `, respectively) to make sure the entire file\n  remains valid in a given syntax (Python, Rust, SQL, etc.). This prefix is stripped\n  during parsing. It is *recommended* to use a prefix with a trailing space (`# `, `// `,\n  or `-- `) but bare prefixes without the trailing space (`#`, `//`, or `--`) are\n  allowed.\n\n- As a special case, *hash style* files may have an arbitrary number of additional lines\n  starting with `#` before the initial `#---` delimiter.\n  This allows for \"shebang\" lines like `#!/usr/bin/bash` at the top of a file, or for\n  Python\n  [inline script metadata](https://packaging.python.org/en/latest/specifications/inline-script-metadata/#inline-script-metadata)\n  to work.\n\n- Other than stripping prefixes, all whitespace in the frontmatter is preserved before\n  it is parsed as YAML.\n\n- Note that YAML comments, which are lines beginning with `#` in the metadata, are\n  allowed. For example, for hash style, this means there must be two hashes (`# #` or\n  `##`) at the start of a comment line, within the delimiters.\n\n- There is no restriction on the content of the file after the frontmatter.\n  It may even contain other content in frontmatter format, but this will not be parsed\n  as frontmatter. Typically, it is text, but it could be binary as well.\n\n- Frontmatter is optional.\n  This means almost any text file can be read as frontmatter format.\n\n## Reference Implementation\n\nThis is a simple Python reference implementation.\nIt auto-detects all the frontmatter styles above.\nIt supports reading small files easily into memory, but also allows extracting or\nchanging frontmatter without reading an entire file.\n\nBoth raw (string) parsed YAML frontmatter (using ruamel.yaml) are supported.\nFor readability, there is also support for preferred sorting of YAML keys.\n\n## Installation\n\nUse pip, poetry, or uv to add `frontmatter-format`.\n\n## Usage\n\nBasic use:\n\n```python\nfrom frontmatter_format import fmf_read, fmf_read_raw, fmf_write, FmStyle, custom_key_sort\n\n# Write some content:\ncontent = \"Hello, World!\\n\"\nmetadata = {\"author\": \"Test Author\", \"title\": \"Test Title\"}\nfmf_write(\"example.md\", content, metadata, style=FmStyle.md)\n\n# Read it back. Style is auto-detected:\ncontent, metadata = fmf_read(\"example.md\")\nprint(content)  # Hello, World!\nprint(metadata)  # {'author': 'Test Author', 'title': 'Test Title'}\n```\n\nThe file then contains:\n\n```md\n---\nauthor: Test Author\ntitle: Test Title\n---\nHello, World!\n```\n\nBy default, writes are atomic.\nKey sort is preserved, but you can provide a sorting function if you prefer metadata\nkeys to be in a specific order (often more readable than being alphabetical).\nThere is also an option for making custom sorts, so certain keys come first and the rest\nare sorted in natural order.\n\nExamples with more formats:\n\n```python\n# Write in any other desired style:\nhtml_content = \"\u003cp\u003eHello, World!\u003c/p\u003e\"\ntitle_first_sort = custom_key_sort([\"title\", \"author\"])\nfmf_write(\"example.html\", content, metadata, style=FmStyle.html, key_sort=title_first_sort)\n```\n\nThe file then contains:\n\n```html\n\u003c!---\ntitle: Test Title\nauthor: Test Author\n---\u003e\nHello, World!\n```\n\nYAML parsing is optional:\n\n```\n# Read metadata without parsing:\ncontent, raw_metadata = fmf_read_raw(\"example.md\")\nprint(repr(raw_metadata))  # 'author: Test Author\\ntitle: Test Title\\n'\n```\n\nThe above is easiest for small files, but you can also operate more efficiently directly\non files, without reading the file contents into memory.\n\n```python\nfrom frontmatter_format import fmf_strip_frontmatter, fmf_insert_frontmatter, fmf_read_frontmatter, fmf_read_frontmatter_raw\n\n# Strip and discard the metadata from a file:\nfmf_strip_frontmatter(\"example.md\")\n\n# Insert the metadata at the top of an existing file:\nnew_metadata = {\"title\": \"New Title\", \"author\": \"New Author\"}\nfmf_insert_frontmatter(\"example.md\", new_metadata, fm_style=FmStyle.yaml)\n\n# Read the raw frontmatter metadata and get the offset for the rest of the content:\nmetadata, offset = fmf_read_frontmatter(\"example.md\")\nprint(metadata)  # {'title': 'Test Title', 'author': 'Test Author'}\nprint(offset)  # The byte offset where the content starts\nraw_metadata, offset = fmf_read_frontmatter_raw(\"example.md\")\nprint(raw_metadata)  # 'title: Test Title\\nauthor: Test Author\\n'\n```\n\n## FAQ\n\n- **Hasn't this been done before?** Possibly, but as far as I can tell, not in a\n  systematic way for multiple file formats.\n  I needed this myself, and think we'd all be better off if more tools used YAML\n  metadata consistently, so I've released the format and implementation here.\n\n- **Is this mature?** This is the first draft of this format.\n  But I've been using this on my own projects for a couple months.\n  The flexibity of just having metadata on all your text files has been great for\n  workflows, pipelines, etc.\n\n- **When should we use it?** All the time if you can!\n  It's especially important for command-line tools, AI agents, LLM workflows, since you\n  often want to store extra metadata is a consistent way on text inputs of various\n  formats like Markdown, HTML, CSS, and Python.\n\n- **Does this specify the format of the YAML itself?** No.\n  This is simply a format for attaching metadata.\n  What metadata you attach is up to your use case.\n  Standardizing headings like title, author, description, let alone other more\n  application-specific information is beyond the scope of this frontmatter format.\n\n- **Can this work with Pydantic?** Yes, definitely.\n  In fact, I think it's probably a good practice to define self-identifiable Pydantic\n  (or Zod) schemas for all your metadata, and then just serialize and deserialize them\n  to frontmatter everywhere.\n\n- **Isn't this the same as what some CMSs use, Markdown files and YAML at the top?**\n  Yes! But this generalizes that format, and removes the direct tie-in to Markdown or any\n  CMS. This can work with any tool.\n  For HTML and code, it works basically with no changes at all since the frontmatter is\n  considered a comment.\n\n- **Can this work with binary files?** No reason why not, if it makes sense for you!\n  You can use `fmf_insert_frontmatter()` to add metadata of any style to any file.\n  Whether this works for your application depends on the file format.\n\n- **Does this work for CSV files?** Sort of.\n  Some tools do properly honor hash style comments when parsing CSV files.\n  A few do not. Our recommendation is go ahead and use it, and find ways to strip the\n  metadata at the last minute if you really can't get a tool to work with the metadata.\n\n\n- **Does this also work for YAML files?** Yes!\n  It's fine to have YAML metadata on YAML metadata.\n  There are just two nuances.\n\n  Firstly, watch out for duplicate `---` separators, if you insert frontmatter in front\n  of a file that already has it.\n\n  Secondly, it's up to you to use the YAML itself to distinguish whether a file has\n  frontmatter or is just a plain YAML file.\n  Both of these can be avoided if you use plain YAML with `---` separators only when\n  using frontmatter format.\n\n* * *\n\n*This project was built from\n[simple-modern-uv](https://github.com/jlevy/simple-modern-uv).*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjlevy%2Ffrontmatter-format","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjlevy%2Ffrontmatter-format","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjlevy%2Ffrontmatter-format/lists"}