{"id":13542771,"url":"https://github.com/attractivechaos/kann","last_synced_at":"2025-04-04T22:08:25.130Z","repository":{"id":41579852,"uuid":"83609139","full_name":"attractivechaos/kann","owner":"attractivechaos","description":"A lightweight C library for artificial neural networks","archived":false,"fork":false,"pushed_at":"2021-12-03T17:06:34.000Z","size":780,"stargazers_count":698,"open_issues_count":31,"forks_count":115,"subscribers_count":49,"default_branch":"master","last_synced_at":"2025-03-28T21:08:58.380Z","etag":null,"topics":["deep-learning","neural-network"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/attractivechaos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-01T22:42:04.000Z","updated_at":"2025-03-21T18:13:34.000Z","dependencies_parsed_at":"2022-08-20T10:10:58.652Z","dependency_job_id":null,"html_url":"https://github.com/attractivechaos/kann","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/attractivechaos%2Fkann","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/attractivechaos%2Fkann/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/attractivechaos%2Fkann/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/attractivechaos%2Fkann/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/attractivechaos","download_url":"https://codeload.github.com/attractivechaos/kann/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247256115,"owners_count":20909240,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","neural-network"],"created_at":"2024-08-01T11:00:17.403Z","updated_at":"2025-04-04T22:08:25.108Z","avatar_url":"https://github.com/attractivechaos.png","language":"C","funding_links":[],"categories":["AI","Machine Learning","C","Machine Learning \u0026 AI on MCU","微控制器 MCU 端","人工智能"],"sub_categories":["USB","Awesome-Embedded Repository"],"readme":"## Getting Started\n```sh\n# acquire source code and compile\ngit clone https://github.com/attractivechaos/kann\ncd kann; make  # or \"make CBLAS=/path/to/openblas\" for faster matrix multiplication\n# learn unsigned addition (30000 samples; numbers within 10000)\nseq 30000 | awk -v m=10000 '{a=int(m*rand());b=int(m*rand());print a,b,a+b}' \\\n  | ./examples/rnn-bit -m7 -o add.kan -\n# apply the model (output 1138429, the sum of the two numbers)\necho 400958 737471 | ./examples/rnn-bit -Ai add.kan -\n```\n\n## Introduction\n\nKANN is a standalone and lightweight library in C for constructing and training\nsmall to medium artificial neural networks such as [multi-layer\nperceptrons][mlp], [convolutional neural networks][cnn] and [recurrent neural\nnetworks][rnn] (including [LSTM][lstm] and [GRU][gru]). It implements\ngraph-based reverse-mode [automatic differentiation][ad] and allows to build\ntopologically complex neural networks with recurrence, shared weights and\nmultiple inputs/outputs/costs. In comparison to mainstream deep learning\nframeworks such as [TensorFlow][tf], KANN is not as scalable, but it is close\nin flexibility, has a much smaller code base and only depends on the standard C\nlibrary. In comparison to other lightweight frameworks such as [tiny-dnn][td],\nKANN is still smaller, times faster and much more versatile, supporting RNN,\nVAE and non-standard neural networks that may fail these lightweight\nframeworks.\n\nKANN could be potentially useful when you want to experiment small to medium\nneural networks in C/C++, to deploy no-so-large models without worrying about\n[dependency hell][dh], or to learn the internals of deep learning libraries.\n\n### Features\n\n* Flexible. Model construction by building a computational graph with\n  operators. Support RNNs, weight sharing and multiple inputs/outputs.\n\n* Efficient. Reasonably optimized matrix product and convolution. Support\n  mini-batching and effective multi-threading. Sometimes faster than mainstream\n  frameworks in their CPU-only mode.\n\n* Small and portable. As of now, KANN has less than 4000 lines of code in four\n  source code files, with no non-standard dependencies by default. Compatible with \n  ANSI C compilers.\n\n### Limitations\n\n* CPU only. As such, KANN is **not** intended for training huge neural\n  networks.\n\n* Lack of some common operators and architectures such as batch normalization.\n\n* Verbose APIs for training RNNs.\n\n## Installation\n\nThe KANN library is composed of four files: `kautodiff.{h,c}` and `kann.{h,c}`.\nYou are encouraged to include these files in your source code tree. No\ninstallation is needed. To compile examples:\n```sh\nmake\n```\nThis generates a few executables in the [examples](examples) directory.\n\n## Documentations\n\nComments in the header files briefly explain the APIs. More documentations can\nbe found in the [doc](doc) directory. Examples using the library are in the\n[examples](examples) directory.\n\n### A tour of basic KANN APIs\n\nWorking with neural networks usually involves three steps: model construction,\ntraining and prediction. We can use layer APIs to build a simple model:\n```c\nkann_t *ann;\nkad_node_t *t;\nt = kann_layer_input(784); // for MNIST\nt = kad_relu(kann_layer_dense(t, 64)); // a 64-neuron hidden layer with ReLU activation\nt = kann_layer_cost(t, 10, KANN_C_CEM); // softmax output + multi-class cross-entropy cost\nann = kann_new(t, 0);                   // compile the network and collate variables\n```\nFor this simple feedforward model with one input and one output, we can train\nit with:\n```c\nint n;     // number of training samples\nfloat **x; // model input, of size n * 784\nfloat **y; // model output, of size n * 10\n// fill in x and y here and then call:\nkann_train_fnn1(ann, 0.001f, 64, 25, 10, 0.1f, n, x, y);\n```\nWe can save the model to a file with `kann_save()` or use it to classify a\nMNIST image:\n```c\nfloat *x;       // of size 784\nconst float *y; // this will point to an array of size 10\n// fill in x here and then call:\ny = kann_apply1(ann, x);\n```\n\nWorking with complex models requires to use low-level APIs. Please see\n[01user.md](doc/01user.md) for details.\n\n### A complete example\n\nThis example learns to count the number of \"1\" bits in an integer (i.e.\npopcount):\n```c\n// to compile and run: gcc -O2 this-prog.c kann.c kautodiff.c -lm \u0026\u0026 ./a.out\n#include \u003cstdlib.h\u003e\n#include \u003cstdio.h\u003e\n#include \"kann.h\"\n\nint main(void)\n{\n\tint i, k, max_bit = 20, n_samples = 30000, mask = (1\u003c\u003cmax_bit)-1, n_err, max_k;\n\tfloat **x, **y, max, *x1;\n\tkad_node_t *t;\n\tkann_t *ann;\n\t// construct an MLP with one hidden layers\n\tt = kann_layer_input(max_bit);\n\tt = kad_relu(kann_layer_dense(t, 64));\n\tt = kann_layer_cost(t, max_bit + 1, KANN_C_CEM); // output uses 1-hot encoding\n\tann = kann_new(t, 0);\n\t// generate training data\n\tx = (float**)calloc(n_samples, sizeof(float*));\n\ty = (float**)calloc(n_samples, sizeof(float*));\n\tfor (i = 0; i \u003c n_samples; ++i) {\n\t\tint c, a = kad_rand(0) \u0026 (mask\u003e\u003e1);\n\t\tx[i] = (float*)calloc(max_bit, sizeof(float));\n\t\ty[i] = (float*)calloc(max_bit + 1, sizeof(float));\n\t\tfor (k = c = 0; k \u003c max_bit; ++k)\n\t\t\tx[i][k] = (float)(a\u003e\u003ek\u00261), c += (a\u003e\u003ek\u00261);\n\t\ty[i][c] = 1.0f; // c is ranged from 0 to max_bit inclusive\n\t}\n\t// train\n\tkann_train_fnn1(ann, 0.001f, 64, 50, 10, 0.1f, n_samples, x, y);\n\t// predict\n\tx1 = (float*)calloc(max_bit, sizeof(float));\n\tfor (i = n_err = 0; i \u003c n_samples; ++i) {\n\t\tint c, a = kad_rand(0) \u0026 (mask\u003e\u003e1); // generating a new number\n\t\tconst float *y1;\n\t\tfor (k = c = 0; k \u003c max_bit; ++k)\n\t\t\tx1[k] = (float)(a\u003e\u003ek\u00261), c += (a\u003e\u003ek\u00261);\n\t\ty1 = kann_apply1(ann, x1);\n\t\tfor (k = 0, max_k = -1, max = -1.0f; k \u003c= max_bit; ++k) // find the max\n\t\t\tif (max \u003c y1[k]) max = y1[k], max_k = k;\n\t\tif (max_k != c) ++n_err;\n\t}\n\tfprintf(stderr, \"Test error rate: %.2f%%\\n\", 100.0 * n_err / n_samples);\n\tkann_delete(ann); // TODO: also to free x, y and x1\n\treturn 0;\n}\n```\n\n## Benchmarks\n\n* First of all, this benchmark only evaluates relatively small networks, but\n  in practice, it is huge networks on GPUs that really demonstrate the true\n  power of mainstream deep learning frameworks. *Please don't read too much into\n  the table*.\n\n* \"Linux\" has 48 cores on two Xeno E5-2697 CPUs at 2.7GHz. MKL, NumPy-1.12.0\n  and Theano-0.8.2 were installed with Conda; Keras-1.2.2 installed with pip.\n  The official TensorFlow-1.0.0 wheel does not work with Cent OS 6 on this\n  machine, due to glibc. This machine has one Tesla K40c GPU installed. We are\n  using by CUDA-7.0 and cuDNN-4.0 for training on GPU.\n\n* \"Mac\" has 4 cores on a Core i7-3667U CPU at 2GHz. MKL, NumPy and Theano came\n  with Conda, too. Keras-1.2.2 and Tensorflow-1.0.0 were installed with pip. On\n  both machines, Tiny-DNN was acquired from github on March 1st, 2017.\n\n* mnist-mlp implements a simple MLP with one layer of 64 hidden neurons.\n  mnist-cnn applies two convolutional layers with 32 3-by-3 kernels and ReLU\n  activation, followed by 2-by-2 max pooling and one 128-neuron dense layer.\n  mul100-rnn uses two GRUs of size 160. Both input and output are 2-D\n  binary arrays of shape (14,2) -- 28 GRU operations for each of the 30000\n  training samples.\n\n|Task       |Framework    |Machine|Device   |Real     |CPU     |Command line |\n|:----------|:------------|:------|--------:|--------:|-------:|:------------|\n|mnist-mlp  |KANN+SSE     |Linux  |1 CPU    | 31.3s   | 31.2s  |mlp -m20 -v0|\n|           |             |Mac    |1 CPU    | 27.1s   | 27.1s  ||\n|           |KANN+BLAS    |Linux  |1 CPU    | 18.8s   | 18.8s  ||\n|           |Theano+Keras |Linux  |1 CPU    | 33.7s   | 33.2s  |keras/mlp.py -m20 -v0|\n|           |             |       |4 CPUs   | 32.0s   |121.3s  ||\n|           |             |Mac    |1 CPU    | 37.2s   | 35.2s  ||\n|           |             |       |2 CPUs   | 32.9s   | 62.0s  ||\n|           |TensorFlow   |Mac    |1 CPU    | 33.4s   | 33.4s  |tensorflow/mlp.py -m20|\n|           |             |       |2 CPUs   | 29.2s   | 50.6s  |tensorflow/mlp.py -m20 -t2|\n|           |Tiny-dnn     |Linux  |1 CPU    | 2m19s   | 2m18s  |tiny-dnn/mlp -m20|\n|           |Tiny-dnn+AVX |Linux  |1 CPU    | 1m34s   | 1m33s  ||\n|           |             |Mac    |1 CPU    | 2m17s   | 2m16s  ||\n|mnist-cnn  |KANN+SSE     |Linux  |1 CPU    |57m57s   |57m53s  |mnist-cnn -v0 -m15|\n|           |             |       |4 CPUs   |19m09s   |68m17s  |mnist-cnn -v0 -t4 -m15|\n|           |Theano+Keras |Linux  |1 CPU    |37m12s   |37m09s  |keras/mlp.py -Cm15 -v0|\n|           |             |       |4 CPUs   |24m24s   |97m22s  ||\n|           |             |       |1 GPU    |2m57s    |        |keras/mlp.py -Cm15 -v0|\n|           |Tiny-dnn+AVX |Linux  |1 CPU    |300m40s  |300m23s |tiny-dnn/mlp -Cm15|\n|mul100-rnn |KANN+SSE     |Linux  |1 CPU    |40m05s   |40m02s  |rnn-bit -l2 -n160 -m25 -Nd0|\n|           |             |       |4 CPUs   |12m13s   |44m40s  |rnn-bit -l2 -n160 -t4 -m25 -Nd0|\n|           |KANN+BLAS    |Linux  |1 CPU    |22m58s   |22m56s  |rnn-bit -l2 -n160 -m25 -Nd0|\n|           |             |       |4 CPUs   |8m18s    |31m26s  |rnn-bit -l2 -n160 -t4 -m25 -Nd0|\n|           |Theano+Keras |Linux  |1 CPU    |27m30s   |27m27s  |rnn-bit.py -l2 -n160 -m25|\n|           |             |       |4 CPUs   |19m52s   |77m45s  ||\n\n* In the single thread mode, Theano is about 50% faster than KANN probably due\n  to efficient matrix multiplication (aka. `sgemm`) implemented in MKL. As is\n  shown in a [previous micro-benchmark][matmul], MKL/OpenBLAS can be twice as\n  fast as the implementation in KANN.\n\n* KANN can optionally use the `sgemm` routine from a BLAS library (enabled by\n  macro `HAVE_CBLAS`). Linked against OpenBLAS-0.2.19, KANN matches the\n  single-thread performance of Theano on Mul100-rnn. KANN doesn't reduce\n  convolution to matrix multiplication, so MNIST-cnn won't benefit from\n  OpenBLAS. We observed that OpenBLAS is slower than the native KANN\n  implementation when we use a mini-batch of size 1. The cause is unknown.\n\n* KANN's intra-batch multi-threading model is better than Theano+Keras.\n  However, in its current form, this model probably won't get alone well with\n  GPUs.\n\n\n\n[mlp]: https://en.wikipedia.org/wiki/Multilayer_perceptron\n[cnn]: https://en.wikipedia.org/wiki/Convolutional_neural_network\n[rnn]: https://en.wikipedia.org/wiki/Recurrent_neural_network\n[gru]: https://en.wikipedia.org/wiki/Gated_recurrent_unit\n[lstm]: https://en.wikipedia.org/wiki/Long_short-term_memory\n[ad]: https://en.wikipedia.org/wiki/Automatic_differentiation\n[dh]: https://en.wikipedia.org/wiki/Dependency_hell\n[ae]: https://en.wikipedia.org/wiki/Autoencoder\n[tf]: https://www.tensorflow.org\n[td]: https://github.com/tiny-dnn/tiny-dnn\n[matmul]: https://github.com/attractivechaos/matmul\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fattractivechaos%2Fkann","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fattractivechaos%2Fkann","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fattractivechaos%2Fkann/lists"}