{"id":17422097,"url":"https://github.com/bevansm/migration-station","last_synced_at":"2026-01-19T08:01:41.768Z","repository":{"id":40763829,"uuid":"274381846","full_name":"bevansm/migration-station","owner":"bevansm","description":"POC PHPBB3-\u003ePHPBB3 Migration Tool w/ Webscraper -\u003eMySql","archived":false,"fork":false,"pushed_at":"2023-01-06T09:44:08.000Z","size":1582,"stargazers_count":0,"open_issues_count":14,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-06T15:51:02.156Z","etag":null,"topics":["phpbb","phpbb3","webscraper"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bevansm.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-23T10:54:49.000Z","updated_at":"2020-07-20T00:03:18.000Z","dependencies_parsed_at":"2023-02-05T20:01:39.078Z","dependency_job_id":null,"html_url":"https://github.com/bevansm/migration-station","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bevansm/migration-station","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bevansm%2Fmigration-station","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bevansm%2Fmigration-station/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bevansm%2Fmigration-station/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bevansm%2Fmigration-station/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bevansm","download_url":"https://codeload.github.com/bevansm/migration-station/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bevansm%2Fmigration-station/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28563215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-19T03:31:16.861Z","status":"ssl_error","status_checked_at":"2026-01-19T03:31:15.069Z","response_time":67,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["phpbb","phpbb3","webscraper"],"created_at":"2024-10-17T03:05:35.937Z","updated_at":"2026-01-19T08:01:41.753Z","avatar_url":"https://github.com/bevansm.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PHPBB3 to PHPBB3 Migration\n\nA PHPBB3 to PHPBB3 migration tool heavily inspired by the python crawlers at https://www.phpbb.com/community/viewtopic.php?f=65\u0026t=1761395. This only supports a subset of features, but does provide avenues to navigate around issues like gcaptcha logins \u0026 custom nested forums.\n\nThis will recursively migrate everything from the given base URL to the given destination forum/subforum (to ease the migration of dependant communities), and will only map over users who have posted in the given forum.\n\nOutput results can be found in the `out` folder. They will be in the form of: - paginated \"Insert\" queries for post - a user creation query, with ghost accounts and non-functional passwords - a forum structure query to create the new forum \u0026 topics - a dump json object, with no posts - a dump posts object\n\nNote that this can produce a large amount of output data, and will generate a high volume of calls to an outbound forum.\n\nPlease treat this as a POC. It performed the migration in a way that we needed, and it provides a decent basis for a TypeScript wrapper of the phpbb api (look in `clients`).\n\n## Notes\n\n### Options\n\nThe Migrator class has a handful of options, many of which were for quick and easy debugging purposes. Note that forcing early execution (users, posts, topics, ect.) may result in malformed or incomplete sets of sql. Use with caution.\n\n### BBCode Parsing\n\nBBCode parsing is hit and miss. You can play around with the values in [Parser.ts](src/package/parsers/Parser.ts) to generate better bitfields for \"legacy\" (pbpbb 3.0.x and 3.1.x) codes (the ids should map to the values inside of the `phpbb_bbcodes` table), but this doesn't support the \"new\" parsing algorithms from phpbb 3.3.x. We're also using an out-of-the box html2bbcode parser to handle pages from locked threads; while this handles standard bbcode fine, it breaks on most other things, and currently doesn't map `[list]` to a `[list]` element (it maps to `[ul]` and `[ol]` instead).\n\nWe do, however, manage to pull out all the codes \u0026 append identifiers to tags, so re-parsing posts from the output json object shouldn't be too horrible. The best way to do an export is always, always to ask the dba/admin of your homesite if they'd run a simple sql export for you:\n\n        Hi, [x].\n\n        We are looking to our community to a different forum. Would you be willing to run a query for us so we can export the posts on the fourm?\n\n        SELECT * FROM phpbb_posts WHERE forum_id IN ([AN ARRAY OF YOUR FORUM IDS HERE])\n\n        SELECT * FROM phpbb_bbcode\n\n        We appreciate your time,\n\n        [z]\n\nIt will always be quicker, and less resource intensive, for them to run that query and dump the results in a bucket someplace instead of running the script. You're hitting the DB indirectly thousands of times, rather than one direct query. From there, you can rehash the bitfields \u0026 update your bbcode as needed.\n\nIf not, then a webscraper is inevitable. Hopefully this one helps, or at least provides a starting point for your use case.\n\n### migrate_user.sql\n\nStrictly intended to map over ghost users generated by this sql. It doesn't do anything fancy. You might have to look around the phpbb/area51 boards if you have a different use case for migration.\n\n## Use\n\nThis code is somewhat specialized for our usecase, which was exporting a single subforum (with no dependents) from a phpbb forum. You can find that forum [here](https://vldrhub.com/search.php?author_id=338\u0026sr=posts) (exported posts in the \"Game Archive\"). Hence, there are no guarentees that this code is bug-free. Treat it as a POC.\n\n1. Fill out the env file as appropriate\n2. Run this script to generate users, forum data, ect.\n3. Jam the resulting SQL commands through phpadmin\n4. Navigate to the Admin Control Panel (ACP) and sync forum stats.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbevansm%2Fmigration-station","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbevansm%2Fmigration-station","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbevansm%2Fmigration-station/lists"}