{"id":17063624,"url":"https://github.com/ahmetaa/fast-dnn","last_synced_at":"2025-06-18T19:35:45.521Z","repository":{"id":36933216,"uuid":"41240505","full_name":"ahmetaa/fast-dnn","owner":"ahmetaa","description":"A fast deep neural network library (CPU) for speech recognition","archived":false,"fork":false,"pushed_at":"2019-03-20T12:28:35.000Z","size":7039,"stargazers_count":84,"open_issues_count":1,"forks_count":17,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-26T12:45:39.483Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahmetaa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-23T07:37:05.000Z","updated_at":"2024-12-05T09:15:34.000Z","dependencies_parsed_at":"2022-07-07T20:44:55.625Z","dependency_job_id":null,"html_url":"https://github.com/ahmetaa/fast-dnn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmetaa%2Ffast-dnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmetaa%2Ffast-dnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmetaa%2Ffast-dnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmetaa%2Ffast-dnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahmetaa","download_url":"https://codeload.github.com/ahmetaa/fast-dnn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248610690,"owners_count":21132984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T10:52:35.284Z","updated_at":"2025-04-12T18:21:59.700Z","avatar_url":"https://github.com/ahmetaa.png","language":"C++","funding_links":[],"categories":["人工智能"],"sub_categories":["机器学习"],"readme":"# fast-dnn\nThis is a fast deep neural network library designed for DNNs used in Speech Recognition systems.  \nThis is a runtime library designed to run on x86-64 CPUs. Training DNNs is not in the scope of this library. \n\nImplementation improves the speed of the DNN calculations on CPUs using SIMD instructions, linear quantization, batch processing, sigmoid lookup and lazy output calculation. \nIdeas are taken from Vanhoucke et-al's [Improving the speed of neural networks on CPUs] (http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37631.pdf) paper. Library contains the C++ implementation and a Java API. \n\n## Basic usage\nFirst a proper DNN model file needs to be created. System uses a text DNN file and converts it to a binary form. \nFor now, it accepts [Kaldi] (http://kaldi.sourceforge.net/dnn.html) style network files. Also, neural network input and hidden layer node sizes needs to be aligned to 4 and 16 respectively.\nA Java API is provided for these operations.\n\nFor converting network:\n\n\tFeedForwardNetwork network = FeedForwardNetwork.loadFromTextFile(\n\t  new File(\"dnn.txt\"),\n\t  new File(\"feature_transform\")\n\t);\n\nFirst file represents the network, second represents the input transformation. Second file contains two vectors. Each input vector is transformed by adding the first vector and multiplying the second. Then padding and binarization is applied.\n\n\tnetwork.align(4,16); // pads input to a factor of 4 and hidden layer node counts to a factor of 16\n\tnetwork.saveBinary(new File(\"dnn.bin\"));\n\nOperations above are only required to run once and not required during runtime. Once the network is ready, it is used via *QuantizedDnn* in runtime:\n\n\tQuantizedDnn dnn = QuantizedDnn.loadFromFile(new File(\"dnn.bin\"));\n\tfloat[][] input = ... input vectors as a matrix. it must match input dimension\n\tfloat[][] results = dnn.calculate(input); // output soft-max result.\n\t \nThere is also a lazy output calculation option. This is task specific to Speech Recognition engines. Basically during recognition, for each input vector\nnot all output probabilities needs to be calculated. In average %30-50 of output probabilities are required. Therefore, output activation calculations\ncan be made lazily. However, user of the API must provide an output-length byte array that contains \"1\"'s as active outputs. Here is an example:\n\n\tQuantizedDnn dnn = QuantizedDnn.loadFromFile(new File(\"dnn.bin\"));\n\tfloat[][] input = ... input vectors as a matrix. it must match input dimension\n    QuantizedDnn.LazyContext context = dnn.getNewLazyContext(input.length);\n    context.calculateUntilOutput(input);\n    for (each input vector) {\n         byte[] mask = ... // get a byte array from ASR system. Array length is equal to output size. 1 values in  \n                           // the array represents active outputs to be calculated.\n         float[] softMaxResult = context.calculateForOutputNodes(mask); // output for current input         \n    }\n\nIf amount of outputs for each input is below %50 of the total outputs, lazy calculation may give around 5-10% speed increase. However, because of the JNI round trips, lazy calculation is not as effective as it should be.  \n\n## How does it work?\n\nThe DNNs used in Automatic Speech Recognition (ASR) systems are usually very large. Especially server side applications use networks with sometimes more than 40 million parameters. In common ASR systems, for 1 seconds of speech, around 100 full network output activations needs to be calculated. This makes around 3-4 billion multiplication and sum operations for 1 second of speech.   \n\nOne idea is to use GPUs for this task. Indeed they work and they are very fast and should be preferred for batch processing if possible. But they are not as ubiquitous as CPUs and they may not be so practical for real-time speech processing.\nSo, for some applications those DNNs needs to run fast in CPUs. Conventional calculation techniques becomes too slow for practical use, as stated in the paper, processing 1 second of speech takes around 4 seconds using\nusing naive floating point matrix multiplications. Using floating point SIMD instructions comes to mind, but that only brings down the number to around 1 seconds. This is still not good enough (Libraries like Eigen and Blas does a much better job though). \n  \nQuantization comes to the rescue.\nInstead of using 32 bit floating numbers for weights and sigmoid activations, 8 bit signed and unsigned numbers can be used.\n32 bit values are linearly quantized to 8 bit numbers. This is possible because the weights are usually lie \nbetween -1 and 1 (Actually a Gaussian curve with small variance), and sigmoid activation values are always between 0 and 1. \nThen, using a special SIMD operation, 16 signed integers\nare multiplied with 16 unsigned integers and results are summed nicely with a couple of SIMD instructions. There are some exceptions and caveats but long story short, this reduces the time required for processing 1 second \nof speech to around 0.25-0.3 seconds. Which is acceptable even for the online systems. For details, please refer to the paper.\n\n## Actual Speed\nIn general, this network is about a magnitude of order faster than a naive C++/Java implementation. According to my tests, it is about 2 times faster than networks that uses BLAS (Via JBlas). Speed difference is lower when compared to C++ Blas, [it is around %25-30 faster] (https://plus.google.com/+AhmetAAkın/posts/RQwqZh9GyPg). When using Java API, it may take a small hit because of the JNI. This library allows usage of very large DNNs (such as 7 2048 node hidden layers and 8000 output nodes). But for small networks, speed difference may not be significant.\n\n## Limitations\n* Only tested in Ubuntu Linux x86-64 (Event then, C++ side may need to be re-compiled). \n* Works in CPU's with SSE4.1 support.\n* Hidden layers must be in equal size.\n* Hidden layer activations are Sigmoid and output activations are SoftMax functions.\n\n## Alternatives\nRecently announced Tensorflow seems to [quantize] (https://petewarden.com/2016/05/03/how-to-quantize-neural-networks-with-tensorflow/) networks for sigmoid and Relu activations using a low precision hand tuned matrix multiplication library is open sourced by Google [gemlowp] (https://github.com/google/gemmlowp). \n\n## TODO\n* Load NN file and pad network in C++ side\n* Implement Java API for network node reduction.\n* Implement ReLu activation quantization network \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahmetaa%2Ffast-dnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahmetaa%2Ffast-dnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahmetaa%2Ffast-dnn/lists"}