{"id":24079682,"url":"https://github.com/QuanosSolutions/FastPack","last_synced_at":"2025-09-16T13:30:45.062Z","repository":{"id":41400300,"uuid":"509381243","full_name":"QuanosSolutions/FastPack","owner":"QuanosSolutions","description":null,"archived":false,"fork":false,"pushed_at":"2024-05-13T20:17:56.000Z","size":1852,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2024-05-13T21:26:39.300Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/QuanosSolutions.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-01T08:39:08.000Z","updated_at":"2024-05-13T21:26:56.537Z","dependencies_parsed_at":"2024-05-13T21:26:45.675Z","dependency_job_id":"e137fcc6-b0b1-4ef1-bd3a-7dcbefdb5784","html_url":"https://github.com/QuanosSolutions/FastPack","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QuanosSolutions%2FFastPack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QuanosSolutions%2FFastPack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QuanosSolutions%2FFastPack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QuanosSolutions%2FFastPack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/QuanosSolutions","download_url":"https://codeload.github.com/QuanosSolutions/FastPack/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233265462,"owners_count":18650037,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-09T22:01:19.240Z","updated_at":"2025-09-16T13:30:45.044Z","avatar_url":"https://github.com/QuanosSolutions.png","language":"C#","funding_links":[],"categories":["DevOps"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"#\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003cimg src=\"docs/images/FastPackLogo.svg\" alt=\"FastPack Logo\" width=\"100%\"\u003e\n  \u003c/a\u003e\n  \u003cbr/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/platform-x64-blue.svg?longCache=true\" alt=\"platform\"/\u003e\n  \u003ca href=\"https://github.com/quanossolutions/FastPack/blob/main/LICENSE\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" alt=\"license\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/QuanosSolutions/FastPack/actions/workflows/ci.yml\"\u003e\n    \u003cimg src=\"https://github.com/QuanosSolutions/FastPack/actions/workflows/ci.yml/badge.svg?branch=main\" alt=\"FastPack CI\" /\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n# General\n\nFastPack is a lightning fast deduplication (de)compressor that is particularly useful for compressing and decompressing build artifacts containing a small to high degree of duplicate files. By default file/directory timestamps as well as meta data are preserved and restored. During decompression the restoration of file/directory timestamps and meta data can be skipped.\n\nThis tool was created as an internal tool by our DevOps-Crew to compress the **~27GB** build artifacts of [Quanos](https://www.quanos.com) as fast as possible (**20s**) into a file with reasonable size (**500 MB**), so it can be attached as a build artifact to each build run and later be extracted in a fast time (**40s**).\n\nBy default FastPack uses **Deflate** for compression with a compression level of \"**Optimal**\". The compression algorithm and level can be specified with the pack-action. Currently the only supported compression algorithms are **Deflate**\" and \"**NoCompression**\". **We're looking for contributors to add reasonable compression algorithms**.\n\n# Why should I use FastPack?\n\n* **Deduplication**: Duplicate files are only compressed once\n* **Size**: Because of the deduplication, the size is mostly much smaller\n* **Speed**: FastPack uses all your cores for packing and unpacking\n* **Filtering**: FastPack allows include/exclude filters during pack and unpack\n* **Comparison**: FastPack allows comparison of 2 archives without the need to unpack the content\n* **Cross-Platform**: Runs on Windows, Linux and OSX\n\n# Sponsors\n\n## Quanos Solutions GmbH\n\n[Quanos](http://quanos.com) gave us the time to create/maintain/publish this great tool.\n\n[![Quanos Solutions GmbH](docs/images/sponsors/quanos.gif)](http://quanos.com)\n\n[Quanos](http://quanos.com) - that is the association of software experts who develop unique software products and solutions for after-sales, service and technical documentation. More than 1000 customers worldwide trust in the innovative, successful and sustainable technology of Quanos.\n\nOur 270 employees have more than 20 years of market experience and contribute to the uniqueness of Quanos: We work closely together, complement each other and benefit from each other's know-how.\n\nTrue to our motto: **\"Passion for smart information\"**, we live our mission every day: we empower human to machine communication.\n\nFollow your passion for CODE at Quanos and join our great team. Check out our current vacancies [here](https://karriere.quanos.com/de?id=2f581a).\n\n# Installation (.NET Global Tool)\n\n```\nCMD\u003e dotnet tool install -g FastPack\n```\n\n# Installation (Nuget - Library)\n\n```\nPM\u003e Install-Package FastPack.Lib\n```\n\n# Installation (Download)\n\nDownload is available [here](https://github.com/quanossolutions/FastPack/releases).\n\n## Supported platforms\n\n* **Windows** (x64)\n* **Linux** (x64)\n* **OSX** (x64)\n\n## Versions\n\nFor all platforms there are 2 versions:\n\n* **RuntimeIncluded**: This version of the executable is larger than the other one, because the runtime (.NET) is included in the executable\n* **RuntimeExcluded**: This version of the executable is smaller than the other one, because the runtime (.NET) is \u003cins\u003enot\u003c/ins\u003e included in the executable. To run this executable you have to install the correct runtime first.\n\n## Antivirus software\n\nBe sure to add an exclusion for FastPack to your antivirus software.\nOtherwise the antivirus software will heavily reduce the speed of FastPack.\n\n# Usage\n\nFor help just call FastPack without any parameters:\n\n```\nCMD\u003e FastPack\nUsage: FastPack [options]\n\nOptions:\n  -a|--action\n    Specifies the action to execute\n    Valid values: about, help, pack, unpack, diff, info, licenses, version\n    Default: pack\n    Example: -a unpack\n    Example: --action unpack\n\n  -q|--quiet\n    Enables quiet-mode, only errors are shown\n\n  -lp|--looseparams\n    Ignores unknown parameters\n\n  -h|--help|-?|/?\n    Shows general or action related help\n\n  @AnyFileName\n    Can be used at any position to provide a file with input parameters\n    The provided file can have different formats that are detected by their file extension.\n    Supported formats: JSON (*.json), XML (*.xml), Text (*.*)\n    Default: Text\n    Example: @params.json\n\nActions:\n  -a about\n    Shows information about FastPack and the contributors.\n\n  -a help\n    Shows this help output.\n\n  -a pack\n    Used to compress the content of directories.\n\n  -a unpack\n    Used to decompress the content of an archive.\n\n  -a diff\n    Shows differences between 2 archives.\n\n  -a info\n    Shows information of an archive.\n\n  -a licenses\n    Shows the licenses of all 3rd party components used in FastPack.\n\n  -a version\n    Shows information about the version of FastPack.\n```\n\nTo show help for a specific action call:\n\n```\nCMD\u003e FastPack --action info --help\nUsage: FastPack -a info [options]\n\nOptions:\n  -i|--input\n    The fup file for which to show information\n    Valid values: a path to a file\n    Required: true\n    Example: -i \"C:\\Users\\user\\file.up\"\n\n  -d|--detailed\n    Flag for showing detailed information like folders and files.\n    Default: false\n    Required: false\n    Example: -d\n\n  -f|--format\n    The output format of the manifest information.\n    Valid values: Text, Json, Xml\n    Default: Text\n    Required: false\n    Example: -f Json\n\n  --pretty\n    Pretty print of the json or xml output.\n    Default: false\n    Required: false\n    Example: --pretty\n```\n\nFastPack uses \"**actions**\" to call different functionalities in FastPack. You provide the action using the action-parameter (-a|--action). The default action is \"**pack**\", for wich the action-parameter can be skipped. To call the **about**-action execute:\n\n```\nCMD\u003e FastPack --action about\n```\n\n## File parameters\n\nAs this tool is primary targeted for automated usage, we also considered advanced usages. This advanced usages could result in size limits for the command line because of too many parameters:\n\n* Windows (8191 chars)\n* Linux (~2 million chars)\n* MacOS (262,144 chars)\n\nIf you provide a lot of exclude/include-filters you can easily hit the limits of Windows.\nFor this case we provide file-parameters (@FilePathHere). You can provide multiple file-parameters at every position. Just put an \"@\" sign in front of an existing parameter-filename:\n\n```\nCMD\u003e FastPack -i C:\\temp -o C:\\temp.fup @Performance.txt @PackFilters.json @General.xml\n```\n\nThere are 3 different formats supported for the parameter-files. The formats are detected by the file extension. The fallback is the textfile format. For the above example the files could look like this\n\n### Plain Text (@Performance.txt)\n\nEach parameter and value has to be on its own line\n\n```\n--maxmemory\n2GB\n--parallelism\n16\n```\n\n### JSON (@PackFilters.json)\n\n```json\n{\n    \"args\": [\n        \"--include-filter\",\n        \"dll\",\n        \"--exclude-filter\",\n        \"dll/**/*.txt\"\n    ]\n}\n```\n\n### XML (@General.xml)\n\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cargs\u003e\n    \u003carg\u003e--no-progress\u003c/arg\u003e\n    \u003carg\u003e-c\u003c/arg\u003e\n    \u003carg\u003e\u003c![CDATA[\n    This\n    is\n    some\n    multiline\n    text\n    ]]\u003e\u003c/arg\u003e\n\u003c/args\u003e\n```\n\n## Filter options\n\nFor the \"pack\" and \"unpack\" action you can provide a filter type (Parameter: -ft). The following filter types are available:\n\n* [Glob](https://en.wikipedia.org/wiki/Glob_(programming)) (Default)\n* [Regex](https://docs.microsoft.com/en-us/dotnet/standard/base-types/regular-expression-language-quick-reference) (.NET Style)\n* [StartsWith](https://docs.microsoft.com/en-us/dotnet/api/system.string.startswith?view=net-6.0#system-string-startswith(system-string))\n\nThe filter type is used to evaluate the filter expressions passed via the following parameters:\n\n* **-if | --include-filter**: The filter for which files/folders are included during pack/unpack\n* **-ef | --exclude-filter**: The filter for which files/folders are excluded during pack/unpack\n\nFirst the include-filters are applied. Then on the remaining set of files/folders the exclude-filters are applied. The filters are applied to the path of a file/folder relative to the input directory for the pack-action. The filters are applied to the relative path of a file/folder in the archive for the unpack-action.\n\nBy **default** the filters are **case-sensitive**. You can change this by passing the following parameter:\n\n* **-fci | --filter-case-insensitive**: Makes the filters case insensitive\n\n## Performance options\n\nFor the \"**pack**\" and \"**unpack**\" action you can provide parameters for **managing memory** usage as well as **cpu usage**.\n\n### Parameter: -p | --parallelism\n\nThis parameter is used for specifying how many of your logical cpu cores are used for compressing and decompressing data. The values can be specified in different formats.\n\nFor all the non-absolute values your available logical cpu cores are used as a base for calculation. For the examples below we assume **16 logical cpu cores** as the base for calculation results.\n\n```\nCMD\u003e FastPack ... --parallelism -25% # Result: Use 12 Cores\nCMD\u003e FastPack ... --parallelism 25%  # Result: Use 4 Cores\nCMD\u003e FastPack ... --parallelism 10  # Result: Use 10 Cores\nCMD\u003e FastPack ... --parallelism -2  # Result: Use 14 Cores\n```\n\nBy **default all logical cores are used** if the \"parallelism\"-parameter is not provided.\n\n### Parameter: -mm | --maxmemory\n\nThis parameter is used for specifying how many RAM is used for compressing and decompressing data. The values can be specified in different formats.\n\nFor all the non-absolute values your available free RAM is used as a base for calculation. For the examples below we assume **16 GB RAM** of free RAM as the base for calculation results.\n\n```\nCMD\u003e FastPack ... --maxmemory -25% # Result: Use up to 12 GB of RAM\nCMD\u003e FastPack ... --maxmemory 25%  # Result: Use up to 4 GB of RAM\nCMD\u003e FastPack ... --maxmemory -1G  # Result: Use up to 15 GB of RAM\nCMD\u003e FastPack ... --maxmemory -2G  # Result: Use up to 2 GB of RAM\nCMD\u003e FastPack ... --maxmemory -500M  # Result: Use up to 500 MB of RAM\nCMD\u003e FastPack ... --maxmemory -900K  # Result: Use up to 900 KB of RAM\nCMD\u003e FastPack ... --maxmemory -900  # Result: Use up to 700 bytes of RAM\n```\n\nBy **default 80% of the available free RAM is used**. Which would be 12.8GB in our example.\nThe RAM used is not the real RAM used for the whole application. Its only the size of RAM used for loading files into RAM for (de)compression. **The overall RAM consumption can be much higher**.\n\nIf files do not fit into the specified memory limits, they are processed in a streamed manner, which slows the overall process.\n\n## Action: Pack\n\nTthe action \"**pack**\" is used for compressing a folders content. The pack-action is the default action if you dont provide the action-parameter (**-a|--action**).\n\n### Compressing the content of a folder with a comment\n\n```\nCMD\u003e FastPack -i C:\\FolderToCompress -o C:\\out\\OutputFile.fup --comment \"a comment for the archive\"\n```\n\n### Compressing the content of a folder with excludes using glob as filter type\n\nIn this example the \"**dll**\" folder and everything below as well as the \"**docs**\" file or folder is excluded during compression.\n\n```\nCMD\u003e FastPack -i C:\\FolderToCompress -o C:\\out\\OutputFile.fup -ft glob -ef dll/** -ef docs\n```\n\n### Deduplicating the content of a folder without compression\n\n```\nCMD\u003e FastPack -i C:\\FolderToCompress -o C:\\out\\OutputFile.fup --compressionalgorithm NoCompression\n```\n\n### Do a dry-run of the compression with simple text output\n\n```\nCMD\u003e FastPack -i C:\\FolderToCompress -o C:\\out\\OutputFile.fup --dryrun\n```\n\n### Do a dry-run of the compression with a pretty and detailed json output\n\n```\nCMD\u003e FastPack -i C:\\FolderToCompress -o C:\\out\\OutputFile.fup --dryrun detailed --dryrunformat json --pretty\n```\n\n## Action: Unpack\n\nThe action \"**unpack**\" is used to uncompress an archive which was compressed by the **pack** action.\n\n### Unpack the content of an archive\n\n```\nCMD\u003e FastPack -a unpack -i C:\\InputFile.fup -o C:\\TargetFolder\n```\n\n### Unpack the content of an archive with excludes using glob as filter type\n\nIn this example the \"**dll**\" folder and everything below as well as the \"**docs**\" file or folder is excluded from unpack.\n\nCMD\u003e FastPack -a unpack -i C:\\InputFile.fup -o C:\\TargetFolder -ft glob -ef dll/** -ef docs\n\n### Do a dry-run of the unpack with simple text output\n\n```\nCMD\u003e FastPack -a unpack -i C:\\InputFile.fup -o C:\\TargetFolder --dryrun\n```\n\n### Do a dry-run of the unpack with a pretty and detailed json output\n\n```\nCMD\u003e FastPack -a unpack -i C:\\InputFile.fup -o C:\\TargetFolder --dryrun detailed --dryrunformat json --pretty\n```\n\n## Action: diff\n\nThe action \"**diff**\" is used to compare two archives. By default only the structure (removed or added files/folders) is compared. But it can be configured to also compare size, dates and permissions. Its also possible to extract the differences to compare the content of files using a comparison tool.\n\n### Do a simple comparison (Structure only) with text output\n\n```\nCMD\u003e FastPack -a diff -1 C:\\FirstFile.fup -2 C:\\FirstFile.fup\n```\n\n### Do a simple comparison (Structure only) with text output and extract the differences\n\n```\nCMD\u003e FastPack -a diff -1 C:\\FirstFile.fup -2 C:\\FirstFile.fup -x C:\\DifferencesFolder\n```\n\n### Do a comparison including sizes, dates and permission with a pretty json output and extract the differences\n\n```\nCMD\u003e FastPack -a diff -1 C:\\FirstFile.fup -2 C:\\FirstFile.fup -x C:\\DifferencesFolder --format json --pretty -s Size -s Date -s Permission\n```\n\n## Action: info\n\nThe action \"**info**\" prints information for an archive. By default only some meta data of the archive is printed in plaintext to the console. But a more detailed output in different formats can be chosen, too.\n\n### Simple info about an archive\n\n```\nCMD\u003e FastPack -a info -i C:\\archive.fup\n```\n\n### Detailed info about an archive in a pretty json format\n\n```\nCMD\u003e FastPack -a info -i C:\\archive.fup --detailed --format json --pretty\n```\n\n## Action: licenses\n\nThe action \"**licenses**\" prints information about all the licenses of 3rd party libraries used in FastPack.\n\n```\nCMD\u003e FastPack -a licenses\n```\n\n## Action: about\n\nThe action \"**about**\" prints information about FastPack and its contributors as well as sponsors. It also allows to print the license of FastPack and 3rd parties.\n\n```\nCMD\u003e FastPack -a about\n```\n\n## Action: version\n\nThe action \"**version**\" prints the current version of FastPack. A shortcut parameter \"**--version**\" without \"**-a version**\" is supported\n\n```\nCMD\u003e FastPack --version\nCMD\u003e FastPack -a version\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FQuanosSolutions%2FFastPack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FQuanosSolutions%2FFastPack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FQuanosSolutions%2FFastPack/lists"}