{"id":21290015,"url":"https://github.com/mtingers/opacify","last_synced_at":"2026-05-18T18:40:45.045Z","repository":{"id":57448932,"uuid":"167108579","full_name":"mtingers/opacify","owner":"mtingers","description":"Opacify reads a file and builds a manifest of external sources to rebuild said file.","archived":false,"fork":false,"pushed_at":"2020-01-02T18:21:25.000Z","size":56,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-26T09:15:53.448Z","etag":null,"topics":["backup","data","obfuscation","python"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mtingers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-23T03:12:44.000Z","updated_at":"2020-01-02T18:21:28.000Z","dependencies_parsed_at":"2022-09-26T17:30:24.899Z","dependency_job_id":null,"html_url":"https://github.com/mtingers/opacify","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtingers%2Fopacify","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtingers%2Fopacify/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtingers%2Fopacify/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtingers%2Fopacify/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mtingers","download_url":"https://codeload.github.com/mtingers/opacify/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243754094,"owners_count":20342542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","data","obfuscation","python"],"created_at":"2024-11-21T12:44:21.382Z","updated_at":"2025-12-30T18:14:00.778Z","avatar_url":"https://github.com/mtingers.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Opacify\n\nOpacify reads a file and builds a manifest of external URLs to rebuild said file.\n\n[![asciicast](https://asciinema.org/a/AubzHtwn5qSRTFuFL1lV72w5h.png)](https://asciinema.org/a/AubzHtwn5qSRTFuFL1lV72w5h)\n\n# Install\n```\npip install opacify\n```\n\n# Must Knows\n\n1. Opacify is slow (and probably always will be)!\n2. A cache is built locally to speedup both pacify and satisfy. It is removed on completed unless you specify ```--keep```.\n3. *The cache is built from downloading the data from the urls list.* TODO: Add cache limit flag.\n4. ```--threads N``` option will help speedup the pacify command.\n\n# Examples\n\nPlease note that the example output may not be accurate at this time as it is a work\nin progress.\n\n## Pacify A File\n```\n$ opacify pacify --input test.txt --manifest test.manifest --cache cache/ --urls urls.txt --keep --threads 4 --force\nProgress: |████████████████████████████████████████████████████| * 100.0% thread-2 0.00m remaining\n\nWrote manifest to: test.manifest\n   Avg chunk size: 3.40\n     Total chunks: 2107\n    Manifest size: 164291\n    Original size: 7173\n     Input sha256: 44060449ed92a19e59231d48ab634cbe89d7328f1c24ac7b48b4992b1256657f\n         Duration: 7.170s\n```\n\n## Satisfy A File\n```\n$ opacify satisfy --out test.txt.out --manifest test.manifest --cache dcache/ --force\nProgress: |████████████████████████████████████████████████████| . 100.0%  0.00m remaining\n\n    Manifest size: 164291\n    Output sha256: 44060449ed92a19e59231d48ab634cbe89d7328f1c24ac7b48b4992b1256657f\n      Output size: 7173\n         Duration: 15.079s\n$ shasum test.txt.out test.txt\n85c7bd6f40ba36326f9acd695779db7847434db4  test.txt.out\n85c7bd6f40ba36326f9acd695779db7847434db4  test.txt\n```\n\n## Build Url List from Reddit\nPlease note that Reddit data is volatile and often disappears.\n```\n$ opacify reddit --out reddit-urls.txt --count 20\nGenerating urls from reddit data...\nWrote urls data to: reddit-urls.txt\n\n$ wc -l reddit-urls.txt\n      20 reddit-urls.txt\n```\n\n## Validate Manifest\nAs time goes by, external sources may disappear or content may change. The following will check that the source\nexists (has a valid HTTP response) and check that the source provides enough data of offset+length:\n```\n$ opacify verify --manifest test.opacify\nValidating external sources listed in manifest ...\nStatus: 100% ... Complete!\n```\n\n# Usage\n```\nusage: opacify [-h] [-V] {pacify,satisfy,verify,reddit} ...\n\nOpacify : v0.3.0\nProject : http://github.com/mtingers/opacify\nAuthor  : Matth Ingersoll \u003cmatth@mtingers.com\u003e\n\npositional arguments:\n  {pacify,satisfy,verify,reddit}\n    pacify              Run in pacify mode (builds manifest from input file)\n    satisfy             Run in satisfy mode (extracts file using manifest)\n    verify              Validate manifest URLs and response length\n    reddit              Auto-generate a urls file from reddit links\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -V, --version         Display Opacify version info\n\nExamples:\n    $ opacify pacify --input test.txt --urls urls.txt --manifest test.opm --cache /tmp/cache/\n    $ opacify satisfy --out test.txt.out --urls urls.txt --manifest test.opm --cache /tmp/dcache/\n```\n\n```\nusage: opacify pacify [-h] -i INPUT -u URLS -m MANIFEST -c CACHE [-k] [-f]\n                      [-d] [-t THREADS] [-s CHUNKSIZE]\n\nRun in pacify mode (builds manifest from input file)\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -i INPUT, --input INPUT\n                        Path to input file\n  -u URLS, --urls URLS  Path to urls file\n  -m MANIFEST, --manifest MANIFEST\n                        Output path of manifest file\n  -c CACHE, --cache CACHE\n                        Path to cache directory\n  -k, --keep            Do not remove cache after completed. Useful for\n                        testing\n  -f, --force           Overwrite manifest if it exists\n  -d, --debug           Turn on debug output\n  -t THREADS, --threads THREADS\n                        Run processing multiple threads\n  -s CHUNKSIZE, --chunksize CHUNKSIZE\n                        Specify a different chunk size (default is 1 byte)\n```\n\n```\nusage: opacify satisfy [-h] -m MANIFEST -o OUT -c CACHE [-k] [-f] [-d]\n\nRun in satisfy mode (rebuilds file using manifest)\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -m MANIFEST, --manifest MANIFEST\n                        Path of manifest file\n  -o OUT, --out OUT     Path to write output file to\n  -c CACHE, --cache CACHE\n                        Path to cache directory\n  -k, --keep            Do not remove cache after completed. Useful for\n                        testing\n  -f, --force           Overwrite output file if it exists\n  -d, --debug           Turn on debug output\n```\n\n```\nusage: opacify verify [-h] -m MANIFEST [-d]\n\nValidate manifest URLs and response length\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -m MANIFEST, --manifest MANIFEST\n                        Path of manifest file\n  -d, --debug           Turn on debug output\n```\n\n```\nusage: opacify reddit [-h] -o OUT -c COUNT\n\nAuto-generate a urls file from reddit links\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -o OUT, --out OUT     Path to write urls to\n  -c COUNT, --count COUNT\n                        How many links to get\n```\n\n# Errors\nSee [Error Codes](/ERRORS.md) for a list of errors and meanings.\n\n# Manifest Format\n\nThe manifest consists of a header and body.\n\n## Header\nThe header is one line with a ':' delimiter.  It contains the following in order as of this writing:\n    version:source-file-sha256:source-file-length\n\n* version: The version of Opacify that the manifest was built with.\n* source-file-sha256: The sha256 of the input file. This is used to validate on satisfy.\n* source-file-length: The length of the input file. This is also used to validate on satisfy.\n\n## Body\n\nEach line represents an item and has a space as a delimiter.  The lines are in order of the input\nfile data.  Example:\n```\nhttp://foo/bar.png 23 55\nhttp://bar/foo.png 100 32\n```\n\nThe body items (each line) consist of the following parts:\n1. encoded url\n2. external source data offset\n3. external source data length\n\n\nThis example describes the following process to rebuild the input file from the above example:\n1. Read 55 bytes from http://foo/bar.png starting at an offset of 23 bytes.\n2. Append this data to the output file.\n3. Read 32 bytes from http://bar/foo.png starting at an offset of 100 bytes.\n4. Append this data to the output file.\n\n\n# TODO\n\n## Backup\n\nAdd ```--backup-level N``` option to create multiple manifest items for a buffer.\nThis is like having replication/a backup for part of a file. If one URL source fails, a backup\nURL can be used.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtingers%2Fopacify","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmtingers%2Fopacify","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtingers%2Fopacify/lists"}