{"id":13906987,"url":"https://github.com/tensorflow/compression","last_synced_at":"2025-05-14T04:06:08.333Z","repository":{"id":33636083,"uuid":"133584278","full_name":"tensorflow/compression","owner":"tensorflow","description":"Data compression in TensorFlow","archived":false,"fork":false,"pushed_at":"2025-04-29T21:46:42.000Z","size":1482,"stargazers_count":885,"open_issues_count":10,"forks_count":255,"subscribers_count":44,"default_branch":"master","last_synced_at":"2025-05-08T00:09:45.162Z","etag":null,"topics":["compression","data-compression","deep-learning","deep-neural-networks","machine-learning","ml","neural-network","python","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tensorflow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-15T23:32:19.000Z","updated_at":"2025-05-04T20:22:59.000Z","dependencies_parsed_at":"2023-02-18T04:31:25.465Z","dependency_job_id":"3a4a3de8-fc3c-4076-8f75-bfea1903204d","html_url":"https://github.com/tensorflow/compression","commit_stats":{"total_commits":295,"total_committers":21,"mean_commits":"14.047619047619047","dds":"0.30847457627118646","last_synced_commit":"389008c5d525111b50f28f3437eec444ca5a0ebe"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorflow%2Fcompression","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorflow%2Fcompression/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorflow%2Fcompression/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorflow%2Fcompression/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tensorflow","download_url":"https://codeload.github.com/tensorflow/compression/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254069217,"owners_count":22009511,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","data-compression","deep-learning","deep-neural-networks","machine-learning","ml","neural-network","python","tensorflow"],"created_at":"2024-08-06T23:01:45.919Z","updated_at":"2025-05-14T04:06:08.265Z","avatar_url":"https://github.com/tensorflow.png","language":"Python","readme":"# TensorFlow Compression\n\nTensorFlow Compression (TFC) contains data compression tools for TensorFlow.\n\nYou can use this library to build your own ML models with end-to-end optimized\ndata compression built in. It's useful to find storage-efficient representations\nof your data (images, features, examples, etc.) while only sacrificing a small\nfraction of model performance. Take a look at the [lossy data compression\ntutorial](https://www.tensorflow.org/tutorials/generative/data_compression) or\nthe [model compression\ntutorial](https://www.tensorflow.org/tutorials/optimization/compression) to get\nstarted.\n\nFor a more in-depth introduction from a classical data compression perspective,\nconsider our [paper on nonlinear transform\ncoding](https://arxiv.org/abs/2007.03034), or watch @jonarchists's [talk on\nlearned image compression](https://www.youtube.com/watch?v=x_q7cZviXkY). For an\nintroduction to lossy data compression from a machine learning perspective, take\na look at @yiboyang's [review paper](https://arxiv.org/abs/2202.06533).\n\nThe library contains (see the [API\ndocs](https://www.tensorflow.org/api_docs/python/tfc) for details):\n\n- Range coding (a.k.a. arithmetic coding) implementations in the form of\n  flexible TF ops written in C++. These include an optional \"overflow\"\n  functionality that embeds an Elias gamma code into the range encoded bit\n  sequence, making it possible to encode alphabets containing the entire set of\n  signed integers rather than just a finite range.\n\n- Entropy model classes which simplify the process of designing rate–distortion\n  optimized codes. During training, they act like likelihood models. Once\n  training is completed, they encode floating point tensors into optimized bit\n  sequences by automating the design of range coding tables and calling the\n  range coder implementation behind the scenes.\n\n- Additional TensorFlow functions and Keras layers that are useful in the\n  context of learned data compression, such as methods to numerically find\n  quantiles of density functions, take expectations with respect to dithering\n  noise, convolution layers with more flexible padding options and support for\n  reparameterizing kernels and biases in the Fourier domain, and an\n  implementation of generalized divisive normalization (GDN).\n\n**Important update:** As of February 1, 2024, TensorFlow Compression is in\nmaintenance mode. This means concretely:\n\n- The full feature set of TFC is frozen. No new features will be developed, but\n  the repository will receive maintenance fixes.\n\n- Going forward, new TFC packages will only work with TensorFlow 2.14. This is\n  due to an incompatibility introduced in the Keras version shipped with TF\n  2.15, which would require a rewrite of our layer and entropy model classes.\n\n- To ensure existing models can still be run with TF 2.15 and later, we are\n  releasing a new package\n  [tensorflow-compression-ops](https://github.com/tensorflow/compression/tree/master/tensorflow_compression_ops),\n  which only contains the C++ ops. These will be updated as long as possible for\n  newer TF versions.\n\n- Binary packages are provided for both options on pypi.org:\n  [TFC](https://pypi.org/project/tensorflow-compression/) and\n  [TFC ops](https://pypi.org/project/tensorflow-compression-ops/).\n\n\n## Documentation \u0026 getting help\n\nRefer to [the API documentation](https://www.tensorflow.org/api_docs/python/tfc)\nfor a complete description of the classes and functions this package implements.\n\nPlease post all questions or comments on\n[Discussions](https://github.com/tensorflow/compression/discussions). Only file\n[Issues](https://github.com/tensorflow/compression/issues) for actual bugs or\nfeature requests. On Discussions, you may get a faster answer, and you help\nother people find the question or answer more easily later.\n\n## Installation\n\n***Note: Precompiled packages are currently only provided for Linux and\nDarwin/Mac OS. To use these packages on Windows, consider installing TensorFlow\nusing the [instructions for\nWSL2](https://www.tensorflow.org/install/pip#windows_1) or using a [TensorFlow\nDocker image](https://www.tensorflow.org/install/docker), and then installing\nthe Linux package.***\n\nSet up an environment in which you can install precompiled binary Python\npackages using the `pip` command. Refer to the\n[TensorFlow installation instructions](https://www.tensorflow.org/install/pip)\nfor more information on how to set up such a Python environment.\n\nThe current version of TensorFlow Compression requires TensorFlow 2. For\nversions compatible with TensorFlow 1, see our [previous\nreleases](https://github.com/tensorflow/compression/releases).\n\n### pip\n\nTo install TFC via `pip`, run the following command:\n\n```bash\npython -m pip install tensorflow-compression\n```\n\nTo test that the installation works correctly, you can run the unit tests with:\n\n```bash\npython -m tensorflow_compression.all_tests\n```\n\nOnce the command finishes, you should see a message ```OK (skipped=29)``` or\nsimilar in the last line.\n\n### Colab\n\nYou can try out TFC live in a [Colab](https://colab.research.google.com/). The\nfollowing command installs the latest version of TFC that is compatible with the\ninstalled TensorFlow version. Run it in a cell before executing your Python\ncode:\n\n```\n%pip install tensorflow-compression~=$(pip show tensorflow | perl -p -0777 -e 's/.*Version: (\\d+\\.\\d+).*/\\1.0/sg')\n```\n\nNote: The binary packages of TFC are tied to TF with the same minor version\n(e.g., TFC 2.9.1 requires TF 2.9.x), and Colab sometimes lags behind a few days\nin deploying the latest version of TensorFlow. As a result, using `pip install\ntensorflow-compression` naively might attempt to upgrade TF, which can create\nproblems.\n\n### Docker\n\nTo use a Docker container (e.g. on Windows), be sure to install Docker\n(e.g., [Docker Desktop](https://www.docker.com/products/docker-desktop)),\nuse a [TensorFlow Docker image](https://www.tensorflow.org/install/docker),\nand then run the `pip install` command inside the Docker container, not on the\nhost. For instance, you can use a command line like this:\n\n```bash\ndocker run tensorflow/tensorflow:latest bash -c \\\n    \"python -m pip install tensorflow-compression \u0026\u0026\n     python -m tensorflow_compression.all_tests\"\n```\n\nThis will fetch the TensorFlow Docker image if it's not already cached, install\nthe pip package and then run the unit tests to confirm that it works.\n\n### Anaconda\n\nIt seems that [Anaconda](https://www.anaconda.com/distribution/) ships its own\nbinary version of TensorFlow which is incompatible with our pip package. To\nsolve this, always install TensorFlow via `pip` rather than `conda`. For\nexample, this creates an Anaconda environment with CUDA libraries, and then\ninstalls TensorFlow and TensorFlow Compression:\n\n```bash\nconda create --name ENV_NAME python cudatoolkit cudnn\nconda activate ENV_NAME\npython -m pip install tensorflow-compression\n```\n\nDepending on the requirements of the `tensorflow` pip package, you may need to\npin the CUDA libraries to specific versions. If you aren't using a GPU, CUDA is\nof course not necessary.\n\n## Usage\n\nWe recommend importing the library from your Python code as follows:\n\n```python\nimport tensorflow as tf\nimport tensorflow_compression as tfc\n```\n\n### Using a pre-trained model to compress an image\n\nIn the\n[models directory](https://github.com/tensorflow/compression/tree/master/models),\nyou'll find a python script `tfci.py`. Download the file and run:\n\n```bash\npython tfci.py -h\n```\n\nThis will give you a list of options. Briefly, the command\n\n```bash\npython tfci.py compress \u003cmodel\u003e \u003cPNG file\u003e\n```\n\nwill compress an image using a pre-trained model and write a file ending in\n`.tfci`. Execute `python tfci.py models` to give you a list of supported\npre-trained models. The command\n\n```bash\npython tfci.py decompress \u003cTFCI file\u003e\n```\n\nwill decompress a TFCI file and write a PNG file. By default, an output file\nwill be named like the input file, only with the appropriate file extension\nappended (any existing extensions will not be removed).\n\n### Training your own model\n\nThe\n[models directory](https://github.com/tensorflow/compression/tree/master/models)\ncontains several implementations of published image compression models to enable\neasy experimentation. Note that in order to reproduce published results, more\ntuning of the code and training dataset may be necessary. Use the `tfci.py`\nscript above to access published models.\n\nThe following instructions talk about a re-implementation of the model published\nin:\n\n\u003e \"End-to-end optimized image compression\"\u003cbr /\u003e\n\u003e J. Ballé, V. Laparra, E. P. Simoncelli\u003cbr /\u003e\n\u003e https://arxiv.org/abs/1611.01704\n\nNote that the models directory is not contained in the pip package. The models\nare meant to be downloaded individually. Download the file `bls2017.py` and run:\n\n```bash\npython bls2017.py -h\n```\n\nThis will list the available command line options for the implementation.\nTraining can be as simple as the following command:\n\n```bash\npython bls2017.py -V train\n```\n\nThis will use the default settings. Note that unless a custom training dataset\nis provided via `--train_glob`, the\n[CLIC dataset](https://www.tensorflow.org/datasets/catalog/clic) will be\ndownloaded using TensorFlow Datasets.\n\nThe most important training parameter is `--lambda`, which controls the\ntrade-off between bitrate and distortion that the model will be optimized for.\nThe number of channels per layer is important, too: models tuned for higher\nbitrates (or, equivalently, lower distortion) tend to require transforms with a\ngreater approximation capacity (i.e. more channels), so to optimize performance,\nyou want to make sure that the number of channels is large enough (or larger).\nThis is described in more detail in:\n\n\u003e \"Efficient nonlinear transforms for lossy image compression\"\u003cbr /\u003e\n\u003e J. Ballé\u003cbr /\u003e\n\u003e https://arxiv.org/abs/1802.00847\n\nIf you wish, you can monitor progress with Tensorboard. To do this, create a\nTensorboard instance in the background before starting the training, then point\nyour web browser to [port 6006 on your machine](http://localhost:6006):\n\n```bash\ntensorboard --logdir=/tmp/train_bls2017 \u0026\n```\n\nWhen training has finished, the Python script saves the trained model to the\ndirectory specified with `--model_path` (by default, `bls2017` in the current\ndirectory) in TensorFlow's `SavedModel` format. The script can then be used to\ncompress and decompress images as follows. The same saved model must be\naccessible to both commands.\n\n```bash\npython bls2017.py [options] compress original.png compressed.tfci\npython bls2017.py [options] decompress compressed.tfci reconstruction.png\n```\n\n## Building pip packages\n\nThis section describes the necessary steps to build your own pip packages of\nTensorFlow Compression. This may be necessary to install it on platforms for\nwhich we don't provide precompiled binaries (currently only Linux and Darwin).\n\nTo be compatible with the official TensorFlow pip package, the TFC pip package\nmust be linked against a matching version of the C libraries. For this reason,\nto build the official Linux pip packages, we use [these Docker\nimages](https://hub.docker.com/r/tensorflow/build) and use the same toolchain\nthat TensorFlow uses.\n\nInside the Docker container, the following steps need to be taken:\n\n1. Clone the `tensorflow/compression` repo from GitHub.\n2. Run `tools/build_pip_pkg.sh` inside the cloned repo.\n\nFor example:\n\n```bash\ngit clone https://github.com/tensorflow/compression.git /tensorflow_compression\ndocker run -i --rm \\\n    -v /tmp/tensorflow_compression:/tmp/tensorflow_compression\\\n    -v /tensorflow_compression:/tensorflow_compression \\\n    -w /tensorflow_compression \\\n    -e \"BAZEL_OPT=--config=manylinux_2_17_x86_64\" \\\n    tensorflow/build:latest-python3.10 \\\n    bash tools/build_pip_pkg.sh /tmp/tensorflow_compression \u003ccustom-version\u003e\n```\n\nFor Darwin, the Docker image and specifying the toolchain is not necessary. We\njust build the package like this (note that you may want to create a clean\nPython virtual environment to do this):\n\n```bash\ngit clone https://github.com/tensorflow/compression.git /tensorflow_compression\ncd /tensorflow_compression\nBAZEL_OPT=\"--macos_minimum_os=10.14\" bash \\\n  tools/build_pip_pkg.sh \\\n  /tmp/tensorflow_compression \u003ccustom-version\u003e\n```\n\nIn both cases, the wheel file is created inside `/tmp/tensorflow_compression`.\n\nTo test the created package, first install the resulting wheel file:\n\n```bash\npython -m pip install /tmp/tensorflow_compression/tensorflow_compression-*.whl\n```\n\nThen run the unit tests (Do not run the tests in the workspace directory where\nthe `WORKSPACE` file lives. In that case, the Python interpreter would attempt\nto import `tensorflow_compression` packages from the source tree, rather than\nfrom the installed package system directory):\n\n```bash\npushd /tmp\npython -m tensorflow_compression.all_tests\npopd\n```\n\nWhen done, you can uninstall the pip package again:\n\n```bash\npython -m pip uninstall tensorflow-compression\n```\n\n## Evaluation\n\nWe provide evaluation results for several image compression methods in terms of\ndifferent metrics in different colorspaces. Please see the\n[results subdirectory](https://github.com/tensorflow/compression/tree/master/results/image_compression)\nfor more information.\n\n## Citation\n\nIf you use this library for research purposes, please cite:\n```\n@software{tfc_github,\n  author = \"Ballé, Jona and Hwang, Sung Jin and Agustsson, Eirikur\",\n  title = \"{T}ensor{F}low {C}ompression: Learned Data Compression\",\n  url = \"http://github.com/tensorflow/compression\",\n  version = \"2.14.1\",\n  year = \"2024\",\n}\n```\nIn the above BibTeX entry, names are top contributors sorted by number of\ncommits. Please adjust version number and year according to the version that was\nactually used.\n\nNote that this is not an officially supported Google product.\n","funding_links":[],"categories":["Tensorflow实用程序","其他_机器学习与深度学习"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftensorflow%2Fcompression","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftensorflow%2Fcompression","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftensorflow%2Fcompression/lists"}