{"id":19211338,"url":"https://github.com/truebrain/truewiki","last_synced_at":"2025-05-12T20:07:49.245Z","repository":{"id":36972542,"uuid":"307321756","full_name":"TrueBrain/TrueWiki","owner":"TrueBrain","description":"A wikiserver alternative to mediawiki and gollum","archived":false,"fork":false,"pushed_at":"2025-04-01T05:40:50.000Z","size":720,"stargazers_count":41,"open_issues_count":10,"forks_count":5,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-20T17:38:41.759Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TrueBrain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-10-26T09:31:34.000Z","updated_at":"2025-03-07T18:01:46.000Z","dependencies_parsed_at":"2024-01-04T11:28:23.664Z","dependency_job_id":"e75fc28e-1c9b-42db-963e-e3cdc4fe9e2f","html_url":"https://github.com/TrueBrain/TrueWiki","commit_stats":null,"previous_names":[],"tags_count":48,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBrain%2FTrueWiki","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBrain%2FTrueWiki/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBrain%2FTrueWiki/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBrain%2FTrueWiki/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TrueBrain","download_url":"https://codeload.github.com/TrueBrain/TrueWiki/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253814975,"owners_count":21968560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T13:42:11.994Z","updated_at":"2025-05-12T20:07:49.223Z","avatar_url":"https://github.com/TrueBrain.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TrueWiki\n\n[![GitHub License](https://img.shields.io/github/license/TrueBrain/TrueWiki)](https://github.com/TrueBrain/TrueWiki/blob/main/LICENSE)\n[![GitHub Tag](https://img.shields.io/github/v/tag/TrueBrain/TrueWiki?include_prereleases\u0026label=stable)](https://github.com/TrueBrain/TrueWiki/releases)\n[![GitHub commits since latest release](https://img.shields.io/github/commits-since/TrueBrain/TrueWiki/latest/main)](https://github.com/TrueBrain/TrueWiki/commits/main)\n\n[![GitHub Workflow Status (Testing)](https://img.shields.io/github/actions/workflow/status/TrueBrain/TrueWiki/testing.yml?branch=main\u0026label=main)](https://github.com/TrueBrain/TrueWiki/actions/workflows/testing.yml)\n\nTrueWiki is an opinionated wikitext server similar to [MediaWiki](https://github.com/wikimedia/mediawiki) and/or [gollum](https://github.com/gollum/gollum).\nAs default storage it uses GitHub (instead of a database).\n\n- [Opinionated](#opinionated)\n- [Configuration](#configuration)\n- [Configuration / GitHub](#using-github)\n- [Configuration / GitLab](#using-gitlab)\n- [Development](#development)\n- [Why Yet-Another-Wiki-Server](#why-yet-another-wiki-server)\n\n## Opinionated\n\nTrueWiki iterates over what MediaWiki offers to bring a more modern approach to a wiki.\nThese opinions are described below.\n\n### Wikitext\n\nAlthough most of the wikitext is compatible with the one implemented by MediaWiki, TrueWiki is a lot more strict and accepts less mistakes.\n\nFor example, `[[ File:en/MyFile.png ]]` is an error, as there should not be a space between `[[` and `File`.\nOther errors, like an unknown variable, links to pages that do not exist, etc, are also reported to the user.\nThis in the hope to motivate people to both write better wikitext, but also to fix mistakes that otherwise go unnoticed.\n\nIf there is one or more error, a badge is shown over \"View Source\", to inform anyone using the wiki that the page contains errors.\n\n### Translations\n\nWhile MediaWiki uses separate Wikis for each language, TrueWiki includes them in a single wiki.\nThis leaves it open to the wiki owner which language is the primary, and TrueWiki doesn't care about it.\nAdditionally, with the help of the `Translation` namespace, TrueWiki helps to make navigating between different translations of the same page.\nIt also informs if a link is of a different language, by postfixing the link with the language.\n\nThis is all done to make it easier for translators to translate the wiki, and easier for readers to know what to expect.\n\n### Folders\n\nTrueWiki uses a folder-tree structure to organise pages (similar to Confluence or other commercial Wikis).\nThere are no uncategorised / unlinked / free-floating pages as in MediaWiki.\n\nFolders can easily be navigated, and complement Categories and Namespaces.\n\n### Namespaces\n\nTrueWiki implements namespaces similar to MediaWiki, with the exception users of the wiki cannot create their own namespaces.\nCustom namespaces are replaced by a folder-tree structure.\n\nNamespaces have a specific function, and can render heavily altered pages depending on the namespace.\nFor example, the `File` namespace allows uploading of files, where no other namespace allows this.\n\n#### Category\n\n`[[Category:en/MyCategory]]` marks a Category / File / Page / Template as being part of that category.\n\n`[[:Category:en/MyCategory]]` links to the Category.\n\n#### File\n\nThis is mostly identical to what MediaWiki implements.\n\n`[[File:en/MyFile.png]]` embeds the image in your page.\nParameters like `thumb`, `frame`, `left`, `link=` etc are all supported.\n\n`[[:File:en/MyFile.png]]` adds a link to the file.\n\nFiles have to be either PNG (with `.png` extension), GIF (with `.gif` extension), or JPEG (with `.jpeg` extension).\nNo other formats are supported, and you have to upload the correct type for the given extension.\n\n#### Folder\n\nThis is to navigate the folder structure of the wiki.\nIt is meant as alternative of custom namespaces as used by MediaWiki, while giving clear overview of what pages exist.\nAll pages have in their bottom right a link to the folder their page is located in.\n\n#### Page\n\nSimilar to MediaWiki's `Main` namespace; this contains all the pages.\nThis is the default namespace, meaning that `[[en/Page]]` implies the `Page` namespace.\n\n#### Template\n\nSimilar to MediaWiki's `Template` namespace; this contains all the templates which can be transcluded on Pages.\nThere is no real difference between `Template` and `Page`, except that `{{MyTemplate}}` uses `Template` namespace by default.\nOne can use `{{Page:MyPage}}` to transclude a Page.\n\n#### Translation\n\nThe translation system used by TrueWiki.\n\n`[[Translation:en/MyPage]]` marks the page as a translation (or the English version) or a Page.\nAll translations of this page automatically pick up on the new translation and add them to their header.\n\n## Configuration\n\nTrueWiki has two methods of configuration: runtime options and wiki configuration file.\n\nThe runtime options defines how TrueWiki (the server) should run.\nThe wiki configuration file defines how the wiki should look and feel.\n\n### Runtime options\n\nRunning `python -m truewiki --help` will give a detailed overview of all the settings.\n\n```\nUsage: python -m truewiki [OPTIONS]\n\nOptions:\n  --sentry-dsn TEXT               Sentry DSN.\n  --sentry-environment TEXT       Environment we are running in.\n  --bind TEXT                     The IP to bind the server to  [default: ::1,\n                                  127.0.0.1]\n  --port INTEGER                  Port of the web server  [default: 80]\n  --storage [github|gitlab|git|local]\n                                  [required]\n  --frontend-url TEXT             URL of the frontend, used for creating\n                                  absolute links in the sitemap.xml\n  --cache-time INTEGER            Cache time of uploaded images (in seconds)\n                                  [default: 300]\n  --reload-secret TEXT            Secret to allow an index reload. Always use\n                                  this via an environment variable!\n  --cache-metadata-file TEXT      File used to cache metadata.  [default:\n                                  .cache_metadata.json]\n  --storage-folder DIRECTORY      Folder to use for storage.  [default:\n                                  ./data]\n  --storage-git-username TEXT     Username to use when creating commits.\n                                  [default: Librarian]\n  --storage-git-email TEXT        Email to use when creating commits.\n                                  [default: wiki@localhost]\n  --storage-github-url URL        Repository URL on GitHub.  [default:\n                                  https://github.com/TrueBrain/wiki-example]\n  --storage-github-history-url URL\n                                  Repository URL on GitHub to visit history\n                                  (defaults to --storage-github-url).\n  --storage-github-deploy-key, --storage-github-private-key TEXT\n                                  Base64-encoded GitHub Deploy key to access\n                                  the repository. Use either this or a GitHub\n                                  App. Always use this via an environment\n                                  variable!\n  --storage-github-app-id TEXT    GitHub App ID that has write access to the\n                                  repository. Use either this or a GitHub\n                                  Deploy Key.\n  --storage-github-app-key TEXT   Base64-encoded GitHub App Private Key. Use\n                                  either this or a GitHub Deploy Key. Always\n                                  use this via an environment variable!\n  --storage-github-api-url URL    GitHub API URL to use with GitHub App.\n                                  [default: https://api.github.com]\n  --storage-github-branch branch  Branch of the GitHub repository to use.\n                                  [default: main]\n  --storage-gitlab-url URL        Repository URL on Gitlab.  [default:\n                                  https://gitlab.com/TrueBrain/wiki-\n                                  example.git/]\n  --storage-gitlab-history-url URL\n                                  Repository URL on Gitlab to visit history\n                                  (defaults to --storage-gitlab-url).\n  --storage-gitlab-private-key TEXT\n                                  Base64-encoded private key to access Gitlab.\n                                  Always use this via an environment variable!\n  --storage-gitlab-branch branch  Branch of the Gitlab repository to use.\n                                  [default: main]\n  --user [developer|github|gitlab|microsoft]\n                                  User backend to use (can have multiple).\n                                  [required]\n  --user-session-expire SECONDS   Time for a session to expire (measured from\n                                  the moment of login).  [default: 50400]\n  --user-login-expire SECONDS     Time for a login attempt to expire.\n                                  [default: 600]\n  --user-session-expire-schedule SECONDS\n                                  The interval between check if a user session\n                                  is expired.  [default: 900]\n  --user-github-client-id TEXT    GitHub client ID. (user=github only)\n  --user-github-client-secret TEXT\n                                  GitHub client secret. Always use this via an\n                                  environment variable! (user=github only)\n  --user-gitlab-client-id TEXT    Gitlab client ID. (user=gitlab only)\n  --user-gitlab-client-secret TEXT\n                                  Gitlab client secret. Always use this via an\n                                  environment variable! (user=gitlab only)\n  --user-microsoft-client-id TEXT\n                                  Microsoft client ID. (user=microsoft only)\n  --user-microsoft-client-secret TEXT\n                                  Microsoft client secret. Always use this via\n                                  an environment variable! (user=microsoft\n                                  only)\n  --cache-page-folder TEXT        Folder used to cache rendered pages.\n  --validate-all                  Validate all mediawiki files and report all\n                                  errors\n  --validate-output-json          Report validation result as JSON\n  -h, --help                      Show this message and exit.\n```\n\n#### Using GitHub\n\nSetting `--storage` and `--user` to `github` will run TrueWiki on a [GitHub](https://github.com) git repository as backend.\n\nBy default, it will checkout the [wiki-example](https://github.com/TrueBrain/wiki-example) project, and edits will not persist (a restart will remove any edits).\n\nWith `--storage-github-url` you can change this to your own repository; but in order for TrueWiki to push changes, you will have to setup an SSH key with write permissions.\n\nThe easiest way to do this is via a [Deploy Key](https://docs.github.com/en/developers/overview/managing-deploy-keys#deploy-keys) on GitHub.\n`TRUEWIKI_STORAGE_GITHUB_DEPLOY_KEY` should be set to the base64-encoded Deploy key.\n`--storage-github-url` should be set to the SSH url of your GitHub repository (for example: `git@github.com:TrueBrain/wiki-example.git`).\nWith this, TrueWiki will automatically push changes to the GitHub repository, making your wiki persistent.\n\nIf using SSH isn't possible, you have two other options:\n- Install a git credential helper on the host which has push permissions.\n- Use a GitHub Apps.\n\nFor the latter, you need to create a GitHub Apps, give it permission to read/write content, and install it for the repository.\nAfter this, you can use for example `https://github/TrueBrain/wiki-example` as `--storage-github-url`.\nMake sure to set `--storage-github-app-id` and `TRUEWIKI_STORAGE_GITHUB_APP_KEY`, where the latter is a base64-encoded Private key of the GitHub App.\nIn case you use GitHub enterprise, `--storage-github-api-url` needs to point to the API endpoint of your instance.\n\n#### Using GitLab\n\nSetting `--storage` and `--user` to `gitlab` will run TrueWiki on a [GitLab](https://gitlab.com) git repository as backend.\n\nBy default, it will checkout the [wiki-example](https://gitlab.com/TrueBrain/wiki-example) project, and edits will not persist (a restart will remove any edits).\n\nWith `--storage-gitlab-url` you can change this to your own repository; but in order for TrueWiki to push changes, you will have to setup an SSH key with write permissions.\n\nThe easiest way to do this is via a [SSH Key](https://docs.gitlab.com/ee/ssh/) on GitLab.\n`TRUEWIKI_STORAGE_GITLAB_PRIVATE_KEY` should be set to the base64-encoded private key matching the deployment key.\n`--storage-gitlab-url` should be set to the SSH url of your GitLab repository (for example: `git@gitlab.com:TrueBrain/wiki-example.git`).\nWith this, TrueWiki will automatically push changes to the GitLab repository, making your wiki persistent.\n\n#### Using Microsoft (user-backend only)\n\nSetting `--user` to `microsoft` will use Microsoft as user-backend.\nYou will still need to select either `github` or `gitlab` for `--storage`.\n\nIn your Azure Portal you can create an Application Registration.\nAfter creation you need to create a Client Secret.\nThis secret combined with the application (client) ID is all that is needed to use the Microsoft user-backend.\n\nDepending on the Application Registration it can allow either only accounts in your subscription, or any Microsoft account.\nThe Application Registration wizard helps you make this decision.\n\n### Wiki configuration file\n\nThis tells TrueWiki how to render the wiki pages.\nIt allows for configuration of CSS, Javascript, header, footer and more.\nFor detailed information about the options and what they do, please see the example project:\n\nhttps://github.com/TrueBrain/wiki-example/blob/main/.truewiki.yml\n\n## Development\n\nThis server is written in Python 3.11 with aiohttp.\n\n### Running a local server\n\n#### Dependencies\n\n- Python3.11 or higher.\n\n#### Preparing your venv\n\nTo start it, you are advised to first create a virtualenv:\n\n```bash\npython3 -m venv .env\n.env/bin/pip install -r requirements.txt\n```\n\n#### Preparing a data folder\n\nTrueWiki needs to read its data from somewhere, and this normally is a git repository in the `data` folder.\nIt should contain the following folders (TrueWiki will create required folders if needed):\n\n- `Category` - For the files in the Category namespace; should contain `.mediawiki` files.\n- `File` - For the files in the File namespace; can contain any file.\n- `Page` - For the files in the default namespace; should contain `.mediawiki` files.\n- `Template` - For the files in the Template namespace; should contain `.mediawiki` files.\n\n#### Starting a local server\n\nYou can start the HTTP server by running:\n\n```bash\n.env/bin/python -m truewiki\n```\n\nThis will start the server on port 8000 for you to work with locally.\n\n### Running via docker\n\n```bash\ndocker build -t truebrain/truewiki:local .\ndocker run --rm -p 127.0.0.1:8000:80 -v \"`pwd`/data:/data\" -v \"`pwd`/cache:/cache\" truebrain/truewiki:local\n```\n\n### End-to-end tests\n\nIn order to know all functionality is still working as expected, TrueWiki uses [Playwright](https://playwright.dev) to do end-to-end tests.\nThis is located in the `e2e` folder, and requires some additional tooling to be installed.\n\n```bash\n.env/bin/pip install -r e2e/requirements.txt\n.env/bin/playwright install\nCOVERAGE_RCFILE=\"$(pwd)/.coveragerc\" COVERAGE_FILE=\"$(pwd)/.coverage\" pytest e2e -v --storage local\ncoverage combine\ncoverage report -m\n```\n\n## Why Yet-Another-Wiki-Server\n\nI have been responsible for the [MediaWiki](https://github.com/wikimedia/mediawiki) installation on https://wiki.openttd.org/ from ~2005 till 2020.\nOne thing became clear: it is very difficult to keep it up-to-date and to find quality extensions that live for more than a few years.\nFor example, finding a good quality OAuth2 extension is not only difficult, their workflows are also often weird and errors are difficult to trace.\n\nIn 2019 I started with migrating all OpenTTD services to AWS.\nThe main issue with running a MediaWiki installation in AWS, that is is non-trivial.\nYou need either an expensive database via RDS, or you need to run an EC2 instance yourself.\nIf you go for an RDS, scaling up MediaWiki is also non-trivial, as it needs persistent storage for uploads.\nIn the end, it was concluded that running MediaWiki on AWS would be both expensive and requires a lot of custom maintenance.\nIn contrast, all other services run on AWS ECS, where a tag on GitHub causes a new version to be rolled out.\n\nSo, we set out to find an alternative.\nWith the experience we have had with for example [BaNaNaS](https://github.com/OpenTTD/BaNaNaS) we ideally would like to store all the data in git.\nThis is mostly as data in git is easier for more people to maintain, then a database where very few people have access to.\nAlso, the wiki of OpenTTD was not big enough (~5000 pages) to really need a database to support it.\nAs extra bonus, we would favour any system that could easily be cached.\n\nThe software closest to this is [gollum](https://github.com/gollum/gollum).\nAlthough gollum appears to do exactly what we want, especially as wikitext is supported via [WikiCloth](https://github.com/usemodj/WikiCloth), reality turned out to be something else.\nAlthough it does support wikitext, or a subset thereof, it does not support templates.\nThis is a huge issue for any real wiki, as templates make a wiki of any decent size possible.\nInitially we did hack in support for templates, but as it is written in Ruby, not a language any of the people involved knew sufficiently to make any decent contribution, it only added more issues than it resolved.\nIn the end, it was decided this was not a road to go.\n\nThis made me interested in how difficult it would be to write this ourself.\nA quick search showed the existence of [wikitextparser](https://github.com/5j9/wikitextparser), written in Python, the programming languages all OpenTTD services are written in.\nThis library seems to be able to parse wikitext just fine, the repository exists for many years, the author is still active, and has a 100% coverage.\nIn short this library is absolutely amazing.\nIt took ~4 hours to make a proof of concept, to show we can render HTML via this library from wikitext.\nSo, [wikitexthtml](https://github.com/TrueBrain/wikitexthtml) was born.\n\nThe next week or so was spend in finding bugs in [wikitextparser](https://github.com/5j9/wikitextparser) and [wikitexthtml](https://github.com/TrueBrain/wikitexthtml), while we were running the current content of the OpenTTD wiki through it.\nNothing groundbreaking, just the normal edge-cases you get when parsing a wiki that has been around for 15+ years with many different people thinking they can write wikitext.\nIt was also where we found out that wikitext has not really a specification; it is more of a: what-ever the PHP implementation does.\nA [rust library](https://docs.rs/parse_wiki_text/0.1.5/parse_wiki_text/) for parsing wikitext has an excellent rant about this, expressing exactly how I feel.\n\nIn the end, this repository was born.\nIt renders pages faster than MediaWiki serves pages from its cache (TrueWiki: ~30ms, MediaWiki: ~120ms).\n\nThe wikitext supported is far from completely, and it is easily to find many edge-cases where it fails.\nBut mostly, this can be solved by informing the user, while creating the page, something is going wrong.\nOften the fixes are trivial, and good user-feedback avoids this.\nThis means that with only supporting a subset of wikitext, all of the 5000+ pages on the OpenTTD wiki as it was in 2020 could be rendered to HTML in nearly the same way as MediaWiki can.\n\nAnd that is the history of TrueWiki.\n\nOwh, and the name? Well, frosch123 helped out a lot with exporting the OpenTTD wiki from MediaWiki to our on-disk format.\nAnd he joked it would be a good name. TrueBrain .. TrueWiki .. get it?\nI agree, it is an excellent name.\nThank you so much frosch123 for the help with this repository!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftruebrain%2Ftruewiki","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftruebrain%2Ftruewiki","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftruebrain%2Ftruewiki/lists"}