{"id":15633578,"url":"https://github.com/angusg/tensorflow-xnor-bnn","last_synced_at":"2025-08-22T22:37:38.017Z","repository":{"id":87480075,"uuid":"98679503","full_name":"AngusG/tensorflow-xnor-bnn","owner":"AngusG","description":"BinaryNets in TensorFlow with XNOR GEMM op","archived":false,"fork":false,"pushed_at":"2017-08-23T00:50:24.000Z","size":194,"stargazers_count":156,"open_issues_count":10,"forks_count":39,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-03-28T05:07:41.384Z","etag":null,"topics":["binary-neural-networks","deep-learning","machine-learning","tensorflow","xnor-convolutions","xnor-net"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AngusG.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-07-28T19:01:53.000Z","updated_at":"2025-03-03T06:48:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"2b2d055f-4ac0-4e98-9cb3-3fc28d4b4ffb","html_url":"https://github.com/AngusG/tensorflow-xnor-bnn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AngusG%2Ftensorflow-xnor-bnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AngusG%2Ftensorflow-xnor-bnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AngusG%2Ftensorflow-xnor-bnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AngusG%2Ftensorflow-xnor-bnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AngusG","download_url":"https://codeload.github.com/AngusG/tensorflow-xnor-bnn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248915018,"owners_count":21182546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary-neural-networks","deep-learning","machine-learning","tensorflow","xnor-convolutions","xnor-net"],"created_at":"2024-10-03T10:49:24.724Z","updated_at":"2025-04-14T16:24:27.908Z","avatar_url":"https://github.com/AngusG.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tensorflow-xnor-bnn\nBinaryNets in TensorFlow with XNOR GEMM op\n\n### Dependencies\nThe project was tested with:\n* `python 3.6.1`\n* `tensorflow 1.2.1`\n* `numpy 1.13.1`\n* `g++ 4.8.4`\n* `Cuda compilation tools, release 8.0, V8.0.44`\n\n## Using this repo\n\n### 1 - Compile the gemm_op.so library\n1. Run `source setenv.sh` to set `TF_INC` variable with location to core tensorflow headers (you do not need to have source installed). \n2. In project root run `mkdir obj libs`, this is where `gemm_op.so` will be placed.\n3. Run `make`. If you want to make changes to the op without changing the kernels, there is a `cpp` target to save time. \n\n### 2 - Confirm the op yields same results as tf.matmul()\n* Run `python test_gemm_op.py` which generates two matrices of +1/-1 and compares the results from `xnor_gemm` to `tf.matmul`.\n\n### 3 - Run benchmarks\n* Run `python matmul_bench.py` to compare the GEMM performance between the `xnor_gemm` and `tf.matmul`. The speedup is less than that reported in https://arxiv.org/abs/1602.02830 because we're comparing to a highly optimized kernel, not the unoptimized base kernel. The results should be similar to the improvement over cuBLAS (2-3x for large matrices). Some results for three GPUs are reported below, where N is the size of square input matrices.\n\nGTX-680-4GB\n\n|N     | RUNS| Avg\t(s)| Std\t(s)| Avg (s) | Std\t(s)| Speedup |\n|------|:----|:--------|:--------|:--------|:--------|:-------:|\n|1024\t |20\t |0.00608\t |0.00051\t |0.00875\t |0.01861\t |1.44     |\n|2048\t |10\t |0.01877\t |0.00235\t |0.02770\t |0.02294\t |1.48     |\n|4096\t |10\t |0.07897\t |0.00325\t |0.11908\t |0.02427\t |1.51     |\n|8192\t |10\t |0.36292\t |0.00331\t |0.75703\t |0.02268\t |2.09     |\n\nGTX-TITAN-BLACK-6GB\n\n|N | RUNS\t| Avg\t(s) | Std\t(s) | Avg (s)\t| Std\t(s) | Speedup |\n|------|:----|:--------|:--------|:--------|:--------|:----:|\n| 1024 | 20  | 0.00473 | 0.00021 | 0.00362 | 0.00199 | 0.76 |\n| 2048 | 10  | 0.01184 | 0.00007 | 0.01364 | 0.00879 | 1.15 |\n| 4096 | 10  | 0.04598 | 0.00524 | 0.06320 | 0.01995 | 1.37 |\n| 8192 | 10  | 0.19189 | 0.00323 | 0.35513 | 0.08722 | 1.85 |\n\nTESLA-P100-PCIE-12GB\n\n| N    | RUNS | Avg (s) | Med (s)  | Avg (s) | Med (s) | Speedup (avg) | Speedup (med) |\n|------|------|---------|----------|---------|---------|---------------|---------------|\n| 1024 | 19   | 0.00316 | 0.00317  | 0.00264 | 0.00202 | 0.83          | 0.64          |\n| 2048 | 9    | 0.00804 | 0.008029 | 0.01028 | 0.00698 | 1.28          | 0.87          |\n| 4096 | 9    | 0.02665 | 0.02647  | 0.04669 | 0.03353 | 1.75          | 1.27          |\n| 8192 | 9    | 0.10526 | 0.10534  | 0.23801 | 0.19075 | 2.26          | 1.81          |\n\n### 4 - Train MNIST\n\n```\nusage: mnist_fc_bnn.py [-h] [--log_dir LOG_DIR] [--n_hidden N_HIDDEN]\n                       [--reg REG] [--lr LR] [--batch_size BATCH_SIZE]\n                       [--max_steps MAX_STEPS] [--eval_every_n EVAL_EVERY_N]\n                       [--binary] [--xnor] [--batch_norm] [--debug]\n                       data_dir\n\npositional arguments:\n  data_dir              directory for storing input data\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --log_dir LOG_DIR     root path for logging events and checkpointing\n  --n_hidden N_HIDDEN   number of hidden units\n  --reg REG             l1 regularization penalty\n  --lr LR               learning rate\n  --batch_size BATCH_SIZE\n                        examples per mini-batch\n  --max_steps MAX_STEPS\n                        maximum training steps\n  --eval_every_n EVAL_EVERY_N\n                        validate model every n steps\n  --binary              should weights and activations be constrained to -1,\n                        +1\n  --xnor                if binary flag is passed, determines if xnor_gemm cuda\n                        kernel is used to accelerate training, otherwise no\n                        effect\n  --batch_norm          batch normalize activations\n  --debug               run with tfdbg\n```  \n\nThe training script has reasonable defaults. Running `python mnist_fc_bnn.py /path/to/download/mnist/` results in:\n```\nstep 0, loss = 5.7175, test accuracy 0.1050 (243.8 ex/s)\nstep 100, loss = 2.0587, test accuracy 0.9217 (11449.2 ex/s)\nstep 200, loss = 1.6467, test accuracy 0.9433 (2106.6 ex/s)\nstep 300, loss = 1.1352, test accuracy 0.9470 (15324.5 ex/s)\nstep 400, loss = 1.0114, test accuracy 0.9551 (14653.6 ex/s)\nstep 500, loss = 0.8495, test accuracy 0.9578 (13741.0 ex/s)\nstep 600, loss = 0.6992, test accuracy 0.9586 (15455.5 ex/s)\nstep 700, loss = 0.6375, test accuracy 0.9578 (18959.0 ex/s)\nstep 800, loss = 0.5467, test accuracy 0.9522 (13496.9 ex/s)\nstep 900, loss = 0.5446, test accuracy 0.9602 (14288.7 ex/s)\nFinal test accuracy 0.9644\nAvg ex/s = 9139.7\nMed ex/s = 13472.4\n```\n\nPassing the `log_dir` argument will automatically create a unique subfolder with a name based on the provided arguments (if all arguments are the same, a simple counter is incremented). The training/test loss/accuracy scalars are logged as well as histograms of weights (real valued and quantized, activations, and gradients). \n\n![TensorBoard](img/tensorboard.png \"TensorBoard\")\n\nThis command will run a simulated binary net (weights +/- 1 but using tf.matmul) and log to `/scratch/user/logs/tf-bnn/bin/matmul/hid_1024/batch_norm/bs_1024/0.0/1`\n```\npython mnist_fc_bnn.py /path/to/mnist/ --log_dir /scratch/user/logs/tf-bnn --batch_size 1024 --n_hidden 1024 --reg 0 --batch_norm --lr 0.00001 --binary\nFinal test accuracy 0.9022\n```\n\nThe following will run an equivalent full precision net and log to `/scratch/user/logs/tf-bnn/fp/hid_1024/bs_1024/0.0/1`, \n```\npython mnist_fc_bnn.py /path/to/mnist/ --log_dir /scratch/user/logs/tf-bnn --batch_size 1024 --n_hidden 1024 --reg 0 --lr 0.00001\nFinal test accuracy 0.9765\n```\nNote that the accuracy drop of 7.43% due to quantization is less than reported in the original paper, but this was an arbitrary choice of hyperparameters that trains quickly. An excellent discussion on choosing hyperparameters for training BinaryNets can be found in `How to Train a Compact Binary Neural Network with High Accuracy? - Tang et al AAAI-17`. In general, we really shouldn't be using any L1/L2 regularization (`--reg 0`) as this causes instability and more frequent sign changes.\n\nSome things to keep in mind with training speed benchmarks:\n* - We only get a speedup in forward pass since backprop is done with full precision gradients.\n* - The default `binary_net` configuration has 4 layers, but weights aren't quantized in the input/output layers given signficant evidence in the literature that this has a disproportionate adverse impact on accuracy. Thus, when we run a `--xnor` `--binary` net, we're actually only quantizing half of the layers in the network. The training speedup should increase with additional layers.\n\nGTX-TITAN-BLACK-6GB\n\n| batch_size | n_hidden | steps | Avg (ex/s) | Med (ex/s) | Avg (ex/s) | Med (ex/s) | Speedup (avg) | Speedup (med) |\n|------------|----------|-------|------------|------------|------------|------------|---------------|---------------|\n| 512        | 512      | 1000  | 59,468.4   | 61,277.0   | 79,116.0   | 82,863.2   | 0.75          | 0.74          |\n| 1024       | 512      | 1000  | 63,060.3   | 64,415.5   | 73,987.8   | 76,055.5   | 0.85          | 0.85          |\n| 2048       | 512      | 500   | 42,262.9   | 42,864.1   | 41,527.4   | 42,075.2   | 1.02          | 1.02          |\n| 4096       | 512      | 500   | 16,255.7   | 16,579.1   | 13,750.4   | 13,866.2   | 1.18          | 1.20          |\n| 8192       | 512      | 300   | 4,628.4    | 4,591.8    | 3,799.0    | 3,798.9    | 1.22          | 1.21          |\n\n\n## Limitations\n - XNOR GEMM op currently only works for square matrices that are powers of 2, with smallest N being 512.\n - `tf.sign()` used for quantization is leaky and outputs 0 if the input is exactly 0. In practice this doesn't hurt accuracy too much. \n\n## Relevant links\n- https://github.com/tensorflow/tensorflow/issues/1592\n- https://www.tensorflow.org/extend/adding_an_op\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fangusg%2Ftensorflow-xnor-bnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fangusg%2Ftensorflow-xnor-bnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fangusg%2Ftensorflow-xnor-bnn/lists"}