{"id":15283944,"url":"https://github.com/eigenmagic/fediblockhole","last_synced_at":"2025-04-05T03:11:06.375Z","repository":{"id":65251550,"uuid":"580140727","full_name":"eigenmagic/fediblockhole","owner":"eigenmagic","description":"A tool for automatically syncing Mastodon admin domain blocks.","archived":false,"fork":false,"pushed_at":"2024-11-01T22:23:08.000Z","size":332,"stargazers_count":80,"open_issues_count":7,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-29T02:05:43.703Z","etag":null,"topics":["fediverse","fediverse-blocklist","fediverse-tool"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eigenmagic.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-19T20:25:33.000Z","updated_at":"2025-02-10T15:43:42.000Z","dependencies_parsed_at":"2024-10-30T11:03:50.741Z","dependency_job_id":"b36f0e43-a5e5-4431-8c06-0709caec3239","html_url":"https://github.com/eigenmagic/fediblockhole","commit_stats":{"total_commits":231,"total_committers":4,"mean_commits":57.75,"dds":0.4588744588744589,"last_synced_commit":"ba40084772a565f36af1290070e6c9bba14fb9e7"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eigenmagic%2Ffediblockhole","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eigenmagic%2Ffediblockhole/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eigenmagic%2Ffediblockhole/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eigenmagic%2Ffediblockhole/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eigenmagic","download_url":"https://codeload.github.com/eigenmagic/fediblockhole/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247280272,"owners_count":20912967,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fediverse","fediverse-blocklist","fediverse-tool"],"created_at":"2024-09-30T14:48:19.315Z","updated_at":"2025-04-05T03:11:06.356Z","avatar_url":"https://github.com/eigenmagic.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FediBlockHole\n\nA tool for keeping a Mastodon instance blocklist synchronised with remote lists.\n\nThe broad design goal for FediBlockHole is to support pulling in a list of\nblocklists from a set of trusted sources, merge them into a combined blocklist,\nand then push that merged list to a set of managed instances.\n\nMastodon admins can choose who they think maintain quality lists and subscribe\nto them, helping to distribute the load for maintaining blocklists among a\ncommunity of people. Control ultimately rests with the admins themselves so they\ncan outsource as much, or as little, of the effort to others as they deem\nappropriate.\n\nInspired by the way PiHole works for maintaining a set of blocklists of adtech\ndomains. Builds on the work of\n[@CaribenxMarciaX@scholar.social](https://scholar.social/@CaribenxMarciaX) and\n[@gingerrroot@kitty.town](https://kitty.town/@gingerrroot) who started the\n#Fediblock hashtag and did a lot of advocacy around it, often at great personal\ncost.\n\n## Features\n\n### Blocklist Sources\n\n - Read domain block lists from other instances via the Mastodon API.\n - Supports both public lists (no auth required) and 'admin' lists requiring\n   authentication to an instance.\n - Read domain block lists from arbitrary URLs, including local files.\n - Supports CSV and JSON format blocklists\n - Supports RapidBlock CSV and JSON format blocklists\n\n### Blocklist Export/Push\n\n - Push a merged blocklist to a set of Mastodon instances.\n - Export per-source, unmerged block lists to local files, in CSV format.\n - Export merged blocklists to local files, in CSV format.\n - Read block lists from multiple remote instances\n - Read block lists from multiple URLs, including local files\n - Write a unified block list to a local CSV file\n - Push unified blocklist updates to multiple remote instances\n - Control import and export fields\n\n### Flexible Configuration\n\n - Provides (hopefully) sensible defaults to minimise first-time setup.\n - Global and fine-grained configuration options available for those complex situations that crop up sometimes.\n - Allowlists to override blocks in blocklists to ensure you never block instances you want to keep.\n - Blocklist thresholds if you want to only block when an instance shows up in multiple blocklists.\n\n## Installing\n\nInstallable using `pip`.\n\n```\npython3 -m pip install fediblockhole\n```\n\nInstall from source by cloning the repo, `cd fediblockhole` and run:\n\n```\npython3 -m pip install .\n```\n\nInstallation adds a commandline tool: `fediblock-sync`\n\nInstance admins who want to use this tool for their instance will need to add an\nApplication at `https://\u003cinstance-domain\u003e/settings/applications/` so they can\nauthorize the tool to create and update domain blocks with an OAuth token.\n\nMore on authorization by token below.\n\n### Reading remote instance blocklists\n\nIf a remote instance makes its domain blocks public, you don't need\na token to read them.\n\nIf a remote instance only shows its domain blocks to local accounts\nyou'll need to have a token with `read:blocks` authorization set up.\nIf you have an account on that instance, you can get a token by setting up a new\nApplication at `https://\u003cinstance-domain\u003e/settings/applications/`.\n\nTo read admin blocks from a remote instance, you'll need to ask the instance\nadmin to add a new Application at\n`https://\u003cinstance-domain\u003e/settings/applications/` and then tell you the access\ntoken.\n\nThe application needs the `admin:read:domain_blocks` OAuth scope. You can allow\nfull `admin:read` access, but be aware that this authorizes someone to read all\nthe data in the instance. That's asking a lot of a remote instance admin who\njust wants to share domain_blocks with you.\n\nThe `admin:read:domain_blocks` scope is available as of Mastodon v4.1.0, but for\nearlier versions admins will need to use the manual method described below.\n\nYou can update the scope for your application in the database directly like\nthis:\n\n```\nUPDATE oauth_applications as app\n  SET scopes = 'admin:read:domain_blocks'\n  FROM oauth_access_tokens as tok\n  WHERE app.id = tok.application_id\n  AND app.name = '\u003cthe_app_name\u003e'\n;\n```\n\nWhen that's done, regenerate the token (so it has the new scopes) in the\napplication screen in the instance GUI. FediBlockHole should then able to use\nthe app token to read domain blocks via the API, but nothing else.\n\nAlternately, you could ask the remote instance admin to set up FediBlockHole and\nuse it to dump out a CSV blocklist from their instance and then put it somewhere\ntrusted parties can read it. Then you can define the blocklist as a URL source,\nas explained below.\n\n### Writing instance blocklists\n\nTo write domain blocks into an instance requires both the `admin:read` and\n`admin:write:domain_blocks` OAuth scopes.\n\nThe tool needs `admin:read:domain_blocks` scope to read the current list of\ndomain blocks so we update ones that already exist, rather than trying to add\nall new ones and clutter up the instance.\n\n`admin:read` access is needed to check if the instance has any accounts that\nfollow accounts on a domain that is about to get `suspend`ed and automatically\ndrop the block severity to `silence` level so people have time to migrate\naccounts before a full defederation takes effect. Unfortunately, the statistics\nmeasure used to learn this information requires `admin:read` scope.\n\nYou can add `admin:read` scope in the application admin screen. Please be aware\nthat this grants full read access to all information in the instance to the\napplication token, so make sure you keep it a secret. At least remove\nworld-readable permission to any config file you put it in, e.g.:\n\n```\nchmod o-r \u003cconfigfile\u003e\n```\n\nYou can also grant full `admin:write` scope to the application, but if you'd\nprefer to keep things more tightly secured, limit the scope to\n`admin:read:domain_blocks`.\n\nAgain, this scope is only available in the application config screen as of\nMastodon v4.1.0. If your instance is on an earlier version, you'll need to use\nSQL to set the scopes in the database and then regenerate the token:\n\n```\nUPDATE oauth_applications as app\n  SET scopes = 'admin:read admin:write:domain_blocks'\n  FROM oauth_access_tokens as tok\n  WHERE app.id = tok.application_id\n  AND app.name = '\u003cthe_app_name\u003e'\n;\n```\n\nWhen that's done, FediBlockHole should be able to use its token to authorise\nadding or updating domain blocks via the API.\n\n## Using the tool\n\nRun the tool like this:\n\n```\nfediblock-sync -c \u003cconfigfile_path\u003e\n```\n\nIf you put the config file in `/etc/default/fediblockhole.conf.toml` you don't\nneed to pass in the config file path.\n\nFor a list of possible configuration options, check the `--help`.\n\nYou can also read the heavily commented sample configuration file in the repo at\n[etc/sample.fediblockhole.conf.toml](https://github.com/eigenmagic/fediblockhole/blob/main/etc/sample.fediblockhole.conf.toml).\n\n## Configuring\n\nOnce you have your applications and tokens and scopes set up, create a\nconfiguration file for FediBlockHole to use. You can put it anywhere and use the\n`-c \u003cconfigfile\u003e` commandline parameter to tell FediBlockHole where it is.\n\nOr you can use the default location of `/etc/default/fediblockhole.conf.toml`.\n\nAs the filename suggests, FediBlockHole uses TOML syntax.\n\nThere are 4 key sections:\n \n 1. `blocklist_urls_sources`: A list of URLs to read blocklists from\n 1. `blocklist_instance_sources`: A list of Mastodon instances to read blocklists from via API\n 1. `blocklist_instance_destinations`: A list of Mastodon instances to write blocklists to via API\n 1. `allowlist_url_sources`: A list of URLs to read allowlists from\n\nMore detail on configuring the tool is provided below.\n\n### URL sources\n\nThe URL sources is a list of URLs to fetch blocklists from.\n\nSupported formats are currently:\n\n - Comma-Separated Values (CSV)\n - JSON\n - Mastodon v4.1 flavoured CSV\n - RapidBlock CSV\n - RapidBlock JSON\n\nBlocklists must provide a `domain` field, and should provide a `severity` field.\n\n`domain` is the domain name of the instance to be blocked/limited.\n\n`severity` is the severity level of the block/limit. Supported values are: `noop`, `silence`, and `suspend`.\n\nOptional fields that the tool understands are `public_comment`, `private_comment`, `reject_media`, `reject_reports`, and `obfuscate`.\n\n#### CSV format\n\nA CSV format blocklist must contain a header row with at least a `domain` and `severity` field.\n\nOptional fields, as listed about, may also be included.\n\n#### Mastodon v4.1 CSV format\n\nAs of v4.1.0, Mastodon can export domain blocks as a CSV file. However, in their\ninfinite wisdom, the Mastodon devs decided that field names should begin with a\n`#` character in the header, unlike the field names in the JSON output via the\nAPI… or in pretty much any other CSV file anywhere else.\n\nSetting the format to `mastodon_csv` will strip off the `#` character when\nparsing and FediBlockHole can then use Mastodon v4.1 CSV blocklists like any\nother CSV formatted blocklist.\n\n#### JSON format\n\nJSON is also supported. It uses the same format as the JSON returned from the Mastodon API.\n\nThis is a list of dictionaries, with at minimum a `domain` field, and preferably\na `severity` field. The other optional fields are, well, optional.\n\n#### RapidBlock CSV format\n\nThe RapidBlock CSV format has no header and a single field, so it's not\n_strictly_ a CSV file as there are no commas separating values. It is basically\njust a list of domains to block, separated by '\\r\\n'.\n\nWhen using this format, the tool assumes the `severity` level is `suspend`.\n\n#### RapidBlock JSON format\n\nThe RapidBlock JSON format provides more detailed information about domain\nblocks, but is still somewhat limited.\n\nIt has a single `isBlocked` flag indicating if a domain should be blocked or\nnot. There is no support for the 'silence' block level.\n\nThere is no support for 'reject_media' or 'reject_reports' or 'obfuscate'.\n\nAll comments are public, by virtue of the public nature of RapidBlock.\n\n### Instance sources\n\nThe tool can also read domain_blocks from instances directly.\n\nThe configuration is a list of dictionaries of the form:\n```\n{ domain = '\u003cdomain_name\u003e', token = '\u003cBearerToken\u003e', admin = false }\n```\n\nThe `domain` is the fully-qualified domain name of the API host for an instance\nyou want to read domain blocks from. \n\nThe `token` is an optional OAuth token for the application that's configured in\nthe instance to allow you to read domain blocks, as discussed above.\n\nThe `token` can also be specified using environment variables. This provides\nimproved security compared to storing the OAuth token in a configuration file,\nbut it will require the environment variable to be set so that FediBlockHole can\naccess it. See below in [Instance destinations](#instance-destinations) for more\ndetail on how to use environment variables to provide authentication tokens.\n\n`admin` is an optional field that tells the tool to use the more detailed admin\nAPI endpoint for domain_blocks, rather than the more public API endpoint that\ndoesn't provide as much detail. You will need a `token` that's been configured to\npermit access to the admin domain_blocks scope, as detailed above.\n\n### Instance destinations\n\nThe tool supports pushing a unified blocklist to multiple instances.\n\nConfigure the list of instances you want to push your blocklist to in the\n`blocklist_instance_destinations` list. Each entry is of the form:\n\n```\n{ domain = '\u003cdomain_name\u003e', import_fields = ['public_comment'], max_severity = 'suspend', max_followed_severity = 'suspend' }\n```\n\nThe field `domain` is required. It is the fully-qualified domain name of the\ninstance you want to push to. \n\nA BearerToken is also required, for authenticating with the instance. It can be provided in two ways:\n\n1. A token can be provided directly in the entry as a `token` field, like this:\n    ```\n    { domain = '\u003cdomain_name\u003e', token = '\u003cBearerToken\u003e', import_fields = ['public_comment'], max_severity = 'suspend', max_followed_severity = 'suspend' }\n    ```\n    This was the only mechanism available up to version 0.4.5 of Fediblockhole.\n\n1. A token can be provided from the environment.\n\n    If a token is not directly provided with the `token` field, Fediblockhole will\n    look for an environment variable that contains the token.\n\n    By default, the name of the environment variable will be the domain name\n    converted to upper case and with dot/period characters converted to\n    underscores, and the suffix `_TOKEN`. For example, the token variable for the\n    domain `eigenmagic.net` would be `EIGENMAGIC_NET_TOKEN`.\n\n    You can also specify the environment variable to look for, using the\n    `token_env_var` field, like this:\n    ```\n    { domain = '\u003cdomain_name\u003e', token_env_var = 'MY_CUSTOM_DOMAIN_TOKEN', import_fields = ['public_comment'], max_severity = 'suspend', max_followed_severity = 'suspend' }\n    ```\n\n    Fediblockhole will then look for a token in the `MY_CUSTOM_DOMAIN_TOKEN` environment variable.\n\n    If a specific `token_env_var` is provided, the default variable name will\n    not be used. If both the `token` and `token_env_var` fields are provided,\n    the token provided in the `token` field will be used, and a warning will be\n    issued to notify you that you might have misconfigured things.\n\n\nThe BearerToken is\nan application token with both `admin:read:domain_blocks` and\n`admin:write:domain_blocks` authorization.\n\nThe fields `max_followed_severity` and `import_fields` are optional.\n\nThe optional `import_fields` setting allows you to restrict which fields are\nimported from each instance. If you want to import the `reject_reports` settings\nfrom one instance, but no others, you can use the `import_fields` setting to do\nit. **Note:** The `domain` and `severity` fields are always imported.\n\nThe optional `max_severity` setting limits the maximum severity you will allow a\nremote blocklist to set. This helps you import a list from a remote instance but\nonly at the `silence` level, even if that remote instance has a block at\n`suspend` level. If not set, defaults to `suspend`.\n\nThe optional `max_followed_severity` setting sets a per-instance limit on the\nseverity of a domain_block if there are accounts on the instance that follow\naccounts on the domain to be blocked. If `max_followed_severity` isn't set, it\ndefaults to `silence`.\n\nThis setting exists to give people time to move off an instance that is about to\nbe defederated and bring their followers from your instance with them. Without\nit, if a new `suspend` block appears in any of the blocklists you subscribe to (or\na block level increases from `silence` to `suspend`) and you're using the default\n`max` mergeplan, the tool would immediately suspend the instance, cutting\neveryone on the blocked instance off from their existing followers on your\ninstance, even if they move to a new instance. If you actually want that\noutcome, you can set `max_followed_severity = 'suspend'` and use the `max`\nmergeplan.\n\nOnce the follow count drops to 0 on your instance, the tool will automatically\nuse the highest severity it finds again (if you're using the `max` mergeplan).\n\n### Allowlists\n\nSometimes you might want to completely ignore the blocklist definitions for\ncertain domains. That's what allowlists are for.\n\nAllowlists remove any domain in the list from the merged list of blocks before\nthe merged list is saved out to a file or pushed to any instance.\n\nAllowlists can be in any format supported by `blocklist_urls_sources` but ignore\nall fields that aren't `domain`.\n\nYou can also allow domains on the commandline by using the `-A` or `--allow`\nflag and providing the domain name to allow. You can use the flag multiple\ntimes to allow multiple domains.\n\nIt is probably wise to include your own instance domain in an allowlist so you\ndon't accidentally defederate from yourself.\n\n## More advanced configuration\n\nFor a list of possible configuration options, check the `--help` and read the\nsample configuration file in `etc/sample.fediblockhole.conf.toml`.\n\n### save_intermediate\n\nThis option tells the tool to save the unmerged blocklists it fetches from\nremote instances and URLs into separate files. This is handy for debugging, or\njust to have a non-unified set of blocklist files.\n\nWorks with the `savedir` setting to control where to save the files.\n\nThese are parsed blocklists, not the raw data, and so will be affected by `import_fields`.\n\nThe filename is based on the URL or domain used so you can tell where each list came from.\n\n### savedir\n\nSets where to save intermediate blocklist files. Defaults to `/tmp`.\n\n### blocklist_auditfile\n\nIf provided, will save an audit file of counts and percentages by domain. Useful for debugging \nthresholds. Defaults to None.\n\n### no_push_instance\n\nDefaults to False.\n\nWhen set, the tool won't actually try to push the unified blocklist to any\nconfigured instances.\n\nIf you want to see what the tool would try to do, but not actually apply any\nupdates, use `--dryrun`.\n\n### no_fetch_url\n\nSkip the fetching of blocklists from any URLs that are configured.\n\n### no_fetch_instance\n\nSkip the fetching of blocklists from any remote instances that are configured.\n\n### override_private_comment\n\nDefaults to None.\n\nStamp all *new* blocks pushed to a remote server with this comment or code. \nHelps to identify blocks you've created on a server via Fediblockhole versus ones that\nalready existed.\n\n### mergeplan\n\nIf two (or more) blocklists define blocks for the same domain, but they're\ndifferent, `mergeplan` tells the tool how to resolve the conflict.\n\n`max` is the default. It uses the _highest_ severity block it finds as the one\nthat should be used in the unified blocklist.\n\n`min` does the opposite. It uses the _lowest_ severity block it finds as the one\nto use in the unified blocklist.\n\nA full discussion of severities is beyond the scope of this README, but here is\na quick overview of how it works for this tool.\n\nThe severities are:\n\n - **noop**, level 0: This is essentially an 'unblock' but you can include a\n   comment.\n - **silence**, level 1: A silence adds friction to federation with an instance.\n - **suspend**, level 2: A full defederation with the instance.\n\nWith `mergeplan` set to `max`, _silence_ would take precedence over _noop_, and\n_suspend_ would take precedence over both.\n\nWith `mergeplan` set to `min`, _silence_ would take precedence over _suspend_,\nand _noop_ would take precedence over both.\n\nYou would want to use `max` to ensure that you always block with whichever your\nharshest fellow admin thinks should happen.\n\nYou would want to use `min` to ensure that your blocks do what your most lenient\nfellow admin thinks should happen.\n\n### import_fields\n\n`import_fields` controls which fields will be imported from remote\ninstances and URL blocklists, and which fields are pushed to instances from the\nunified blocklist.\n\nThe fields `domain` and `severity` are always included, so only define extra\nfields, if you want them.\n\nYou can't export fields you haven't imported, so `export_fields` should be a\nsubset of `import_fields`, but you can run the tool multiple times. You could,\nfor example, include lots of fields for an initial import to build up a\ncomprehensive list for export, combined with the `--no-push-instances` option so\nyou don't actually apply the full list to anywhere.\n\nThen you could use a different set of options when importing so you have all the\ndetail in a file, but only push `public_comment` to instances.\n\n### export_fields\n\n`export_fields` controls which fields will get saved to the unified blocklist\nfile, if you export one.\n\nThe fields `domain` and `severity` are always included, so only define extra\nfields, if you want them.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feigenmagic%2Ffediblockhole","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feigenmagic%2Ffediblockhole","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feigenmagic%2Ffediblockhole/lists"}