{"id":20773828,"url":"https://github.com/neurocracy/drupal-omnipedia-changes","last_synced_at":"2026-04-21T17:06:14.568Z","repository":{"id":106914756,"uuid":"499690646","full_name":"neurocracy/drupal-omnipedia-changes","owner":"neurocracy","description":"Provides the wiki page changes functionality for Omnipedia.","archived":false,"fork":false,"pushed_at":"2024-12-27T17:16:17.000Z","size":225,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"7.x","last_synced_at":"2025-01-18T07:42:17.912Z","etag":null,"topics":["drupal","drupal-module","php"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neurocracy.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-04T00:58:44.000Z","updated_at":"2024-12-27T17:16:20.000Z","dependencies_parsed_at":"2023-12-23T04:25:37.730Z","dependency_job_id":"4b49a7fe-d14f-4e31-974c-53dbc45ac15b","html_url":"https://github.com/neurocracy/drupal-omnipedia-changes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurocracy%2Fdrupal-omnipedia-changes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurocracy%2Fdrupal-omnipedia-changes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurocracy%2Fdrupal-omnipedia-changes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurocracy%2Fdrupal-omnipedia-changes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neurocracy","download_url":"https://codeload.github.com/neurocracy/drupal-omnipedia-changes/tar.gz/refs/heads/7.x","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243112187,"owners_count":20238181,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["drupal","drupal-module","php"],"created_at":"2024-11-17T12:27:25.757Z","updated_at":"2026-04-21T17:06:14.562Z","avatar_url":"https://github.com/neurocracy.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"This contains the source files for the \"*Omnipedia - Changes*\" Drupal module,\nwhich provides the wiki page changes functionality for\n[Omnipedia](https://omnipedia.app/).\n\n⚠️ ***[Why open source? / Spoiler warning](https://omnipedia.app/open-source)***\n\n*Please note that [all development and issue tracking is done on \u003cimg src=\"https://gitlab.com/neurocracy/omnipedia/omnipedia/-/raw/main/docs/assets/gitlab/gitlab-logo.svg\" alt=\"The GitLab logo\" width=\"16\" height=\"16\"\u003e GitLab](https://gitlab.com/neurocracy/omnipedia/modules/omnipedia-changes).*\n\n----\n\n# Description\n\nThis contains our infrastructure for generating the wiki page changes between\ntwo in-universe dates. We realized early on in development that the sheer amount\nof content that editors would have to deal with would become a nightmare to\nmanage and keep track of if they also had to manually mark up the changes, and\nthis problem would only become exponentially worse the more content and dates\nwere added. A completely automated solution to generate these changes was a\nmust.\n\n## Under the hood\n\nOne major problem when dealing with trying to generate a diff between two\nstrings that are HTML is that a lot of libraries out there don't actually\nunderstand HTML elements and would mangle the HTML structure in expected and\nunexpected ways. We could render the HTML as plain text, without the HTML\nelements, but that would require somehow reconstructing the HTML on top of a\ndiff, which did not seem remotely practical. What we needed was a library that\nunderstands HTML and where elements start and end. After a bit of research, we\nlooked into what [the Diff module](https://www.drupal.org/project/diff) uses,\nand found our solution: the [`caxy/php-htmldiff`\nlibrary](https://github.com/caxy/php-htmldiff).\n\nAfter solving that initial problem, we needed to customize the output of the\nlibrary, but it didn't offer any useful way to do this before it rendered its\ndiffs. The solution we settled on, like many other things on Omnipedia, was to\nparse the rendered diffed HTML into a\n[DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model)\nand manipulate it using [the Symfony DomCrawler\ncomponent](https://symfony.com/doc/current/components/dom_crawler.html) and\n[PHP's DOM](https://www.php.net/manual/en/book.dom.php) classes that it wraps.\n\nThe core code that orchestrates all of this is [the wiki changes builder\nservice](/src/Service/WikiNodeChangesBuilder.php), which configures the\n`caxy/php-htmldiff` instance, validates that a wiki page can have a diff (i.e.\nhas a previous in-universe date to diff against), returns a cached copy if one\nis found, or renders the actual diff and dispatches events both before the diff\nis generated and after. Once that core system was in place, we wrote [several\nevent subscribers](/src/EventSubscriber/Omnipedia/Changes) to alter the output\nto our requirements.\n\n### Asynchronicity\n\nIt's at this point that we ran into a serious problem: while uncached changes\nfor most wiki pages would take a second or two to generate and be sent to the\nbrowser, a few outliers would consistently take far longer, up to 30 seconds or\nmore. We were hitting the limits of what the library and PHP could handle, even\nafter [some excellent work by the library maintainer to improve\nperformance](https://github.com/caxy/php-htmldiff/issues/101).\n\nThe solution to this required significantly more engineering. The server was\nfully capable of generating the wiki page changes, so what we had to do was to\nrender those changes independently of when they were requested; they would have\nto be rendered ahead of time, asynchronously, in a separate process. [Drupal\ncore has a queue\nsystem](https://api.drupal.org/api/drupal/core!core.api.php/group/queue) that\nallows for batch processing, which [the Warmer\nmodule](https://www.drupal.org/project/warmer) builds on top of to allow for\nperforming cache warming tasks.\n\n[We wrote our own custom Warmer\nplug-in](/src/Plugin/warmer/WikiNodeChangesWarmer.php) which is invoked via [a\ncron job](https://en.wikipedia.org/wiki/Cron) that runs multiple times an hour.\nThe plug-in determines all possible variations a set of changes would need to be\nrendered in, specifically different sets of user permissions, and then renders\nthem one by one. These are then cached to a [Permanent Cache\nBin](https://www.drupal.org/project/pcb) so that they survive any Drupal cache\nclear that may be required when deploying updated code (though we try to\nminimize cache clears).\n\n### All together now\n\nWhile all of this is happening in the background process, [the changes route\ncontroller](/src/Controller/OmnipediaWikiNodeChangesController.php) was\nrewritten to handle three possible states so that it always returns a fast\nresponse to the browser:\n\n1. If no changes have been built between the requested couple of dates, it will show a placeholder message telling the user to check back in a few minutes; it doesn't risk trying to build the changes and potentially make the user wait a long time to see them.\n\n2. If changes have been built, but one of the two wiki pages was updated and thus the changes [cache item was invalidated](https://api.drupal.org/api/drupal/core!core.api.php/group/cache#delete) and the cron job hasn't run yet, it will show the old (invalidated) cache item.\n\n3. If changes have been built and the cache item is valid, it will show that item.\n\n----\n\n# Requirements\n\n* [Drupal 10 or 11](https://www.drupal.org/download)\n\n* PHP 8.1\n\n* [Composer](https://getcomposer.org/)\n\n## Drupal dependencies\n\nBefore attempting to install this, you must add the Composer repositories as\ndescribed in the installation instructions for these dependencies:\n\n* The [`ambientimpact_core` module](https://github.com/Ambient-Impact/drupal-ambientimpact-core).\n\n* The following Omnipedia modules:\n\n  * [`omnipedia_core`](https://gitlab.com/neurocracy/omnipedia/modules/omnipedia-core)\n\n  * [`omnipedia_date`](https://gitlab.com/neurocracy/omnipedia/modules/omnipedia-date)\n\n  * [`omnipedia_main_page`](https://gitlab.com/neurocracy/omnipedia/modules/omnipedia-main-page)\n\n  * [`omnipedia_user`](https://gitlab.com/neurocracy/omnipedia/modules/omnipedia-user)\n\n## Front-end dependencies\n\nTo build front-end assets for this project, [Node.js](https://nodejs.org/) and\n[Yarn](https://yarnpkg.com/) are required.\n\n----\n\n# Installation\n\n## Composer\n\n### Set up\n\nEnsure that you have your Drupal installation set up with the correct Composer\ninstaller types such as those provided by [the `drupal/recommended-project`\ntemplate](https://www.drupal.org/docs/develop/using-composer/starting-a-site-using-drupal-composer-project-templates#s-drupalrecommended-project).\nIf you're starting from scratch, simply requiring that template and following\n[the Drupal.org Composer\ndocumentation](https://www.drupal.org/docs/develop/using-composer/starting-a-site-using-drupal-composer-project-templates)\nshould get you up and running.\n\n### Repository\n\nIn your root `composer.json`, add the following to the `\"repositories\"` section:\n\n```json\n{\n  \"type\": \"vcs\",\n  \"url\": \"https://gitlab.com/neurocracy/omnipedia/modules/omnipedia-changes.git\",\n  \"only\": [\"drupal/omnipedia_changes\"]\n}\n```\n\n### Installing\n\nOnce you've completed all of the above, run `composer require\n\"drupal/omnipedia_changes:^7.0@dev\"` in the root of your project to have\nComposer install this and its required dependencies for you.\n\n## Front-end assets\n\nTo build front-end assets for this project, you'll need to install\n[Node.js](https://nodejs.org/) and [Yarn](https://yarnpkg.com/).\n\nThis package makes use of [Yarn\nWorkspaces](https://yarnpkg.com/features/workspaces) and references other local\nworkspace dependencies. In the `package.json` in the root of your Drupal\nproject, you'll need to add the following:\n\n```json\n\"workspaces\": [\n  \"\u003cweb directory\u003e/modules/custom/*\"\n],\n```\n\nwhere `\u003cweb directory\u003e` is your public Drupal directory name, `web` by default.\nOnce those are defined, add the following to the `\"dependencies\"` section of\nyour top-level `package.json`:\n\n```json\n\"drupal-omnipedia-changes\": \"workspace:^7\"\n```\n\nThen run `yarn install` and let Yarn do the rest.\n\n### Optional: install yarn.BUILD\n\nWhile not required, [yarn.BUILD](https://yarn.build/) is recommended to make\nbuilding all of the front-end assets even easier.\n\n----\n\n# Building front-end assets\n\nThis uses [Webpack](https://webpack.js.org/) and [Symfony Webpack\nEncore](https://symfony.com/doc/current/frontend.html) to automate most of the\nbuild process. These will have been installed for you if you followed the Yarn\ninstallation instructions above.\n\nIf you have [yarn.BUILD](https://yarn.build/) installed, you can run:\n\n```\nyarn build\n```\n\nfrom the root of your Drupal site. If you want to build just this package, run:\n\n```\nyarn workspace drupal-omnipedia-changes run build\n```\n\n----\n\n# Major breaking changes\n\nThe following major version bumps indicate breaking changes:\n\n* 4.x - Front-end package manager is now [Yarn](https://yarnpkg.com/); front-end build process ported to [Webpack](https://webpack.js.org/).\n\n* 5.x - Moved and refactored the `omnipedia.wiki_node_changes_user` service to multiple services in the [`omnipedia_user` module](https://gitlab.com/neurocracy/omnipedia/modules/omnipedia-user).\n\n* 6.x - Requires Drupal 9.5; includes backward compatible [Drupal 10](https://www.drupal.org/project/drupal/releases/10.0.0) deprecation fixes but is still not fully compatible.\n\n* 7.x:\n\n  * Requires [Drupal 10](https://www.drupal.org/project/drupal/releases/10.0.0) due to non-backwards compatible change to [`\\Drupal\\Component\\EventDispatcher\\ContainerAwareEventDispatcher::dispatch()`](https://git.drupalcode.org/project/drupal/-/commit/7b324dd8f18919fc4d728bdb0afbcf27c8c02cb2#6e9d627c11801448b7a793c204471d8f951ae2fb).\n\n  * Requires [`ambientimpact_core`](https://github.com/Ambient-Impact/drupal-ambientimpact-core) 2.x due to the event dispatcher change.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneurocracy%2Fdrupal-omnipedia-changes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneurocracy%2Fdrupal-omnipedia-changes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneurocracy%2Fdrupal-omnipedia-changes/lists"}