{"id":20286317,"url":"https://github.com/alexeytochin/tf_seq2seq_losses","last_synced_at":"2025-04-11T08:59:40.804Z","repository":{"id":43843496,"uuid":"422689467","full_name":"alexeytochin/tf_seq2seq_losses","owner":"alexeytochin","description":"TensorFlow implementations of losses for sequence to sequence machine learning models ","archived":false,"fork":false,"pushed_at":"2024-06-22T12:23:00.000Z","size":75,"stargazers_count":10,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T06:24:38.723Z","etag":null,"topics":["2nd-derivative","artificial-intelligence","artificial-neural-networks","ctc","ctc-loss","ctc-loss-implemenetation","deep-learning","fast-ctc-loss","fast-ctc-loss-implementation","hessian","loss","loss-functions","python","second-derivative","seq2seq","sequence-recognition","sequence-to-sequence","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexeytochin.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-29T19:20:13.000Z","updated_at":"2025-02-18T00:37:50.000Z","dependencies_parsed_at":"2024-06-21T09:18:56.156Z","dependency_job_id":"192b4043-ce1a-427a-baed-2bed65e1388e","html_url":"https://github.com/alexeytochin/tf_seq2seq_losses","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexeytochin%2Ftf_seq2seq_losses","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexeytochin%2Ftf_seq2seq_losses/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexeytochin%2Ftf_seq2seq_losses/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexeytochin%2Ftf_seq2seq_losses/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexeytochin","download_url":"https://codeload.github.com/alexeytochin/tf_seq2seq_losses/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248363673,"owners_count":21091414,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["2nd-derivative","artificial-intelligence","artificial-neural-networks","ctc","ctc-loss","ctc-loss-implemenetation","deep-learning","fast-ctc-loss","fast-ctc-loss-implementation","hessian","loss","loss-functions","python","second-derivative","seq2seq","sequence-recognition","sequence-to-sequence","tensorflow"],"created_at":"2024-11-14T14:32:44.959Z","updated_at":"2025-04-11T08:59:40.786Z","avatar_url":"https://github.com/alexeytochin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tf-seq2seq-losses\nTensorflow implementations for\n[Connectionist Temporal Classification](file:///home/alexey/Downloads/Connectionist_temporal_classification_Labelling_un.pdf)\n(CTC) loss functions that are fast and support second-order derivatives.\n\n## Installation\n```bash\n$ pip install tf-seq2seq-losses\n```\n\n## Why Use This Package?\n### 1. Faster Performance\nOfficial CTC loss implementation, \n[`tf.nn.ctc_loss`](https://www.tensorflow.org/api_docs/python/tf/nn/ctc_loss),\nis significantly slower.\nOur implementation is approximately 30 times faster, as shown by the benchmark results:\n\n|        Name        | Forward Time (ms) | Gradient Calculation Time (ms) |                 \n|:------------------:|:-----------------:|:------------------------------:|\n|  `tf.nn.ctc_loss`  |    13.2 ± 0.02    |            10.4 ± 3            |\n| `classic_ctc_loss` |   0.138 ± 0.006   |          0.28 ± 0.01           |\n| `simple_ctc_loss`  |  0.0531 ± 0.003   |         0.119 ± 0.004          |\n\nTested on a single GPU: GeForce GTX 970, Driver Version: 460.91.03, CUDA Version: 11.2. For the experimental setup, see\n[`benchmark.py`](tests/performance_test.py)\nTo reproduce this benchmark, run the following command from the project root directory \n(install `pytest` and `pandas` if needed):\n```bash\n$ pytest -o log_cli=true --log-level=INFO tests/benchmark.py\n```\nHere, `classic_ctc_loss` is the standard version of CTC loss with token collapsing, e.g., `a_bb_ccc_c -\u003e abcc`. \nThe `simple_ctc_loss` is a simplified version that removes blanks trivially, e.g., `a_bb_ccc_c -\u003e abbcccc`.\n\n### 2. Supports Second-Order Derivatives\nThis implementation supports second-order derivatives without using TensorFlow's autogradient. \nInstead, it uses a custom approach similar to the one described\n[here](https://www.tensorflow.org/api_docs/python/tf/nn/ctc_loss)\nwith a complexity of \n$O(l^4)$, \nwhere \n$l$\nis the sequence length. The gradient complexity is \n$O(l^2)$.\n\nExample usage:\n```python\nimport tensorflow as tf\nfrom tf_seq2seq_losses import classic_ctc_loss \n\nbatch_size = 2\nnum_tokens = 3\nlogit_length = 5\nlabels = tf.constant([[1, 2, 2, 1], [1, 2, 1, 0]], dtype=tf.int32)\nlabel_length = tf.constant([4, 3], dtype=tf.int32)\nlogits = tf.zeros(shape=[batch_size, logit_length, num_tokens], dtype=tf.float32)\nlogit_length = tf.constant([5, 4], dtype=tf.int32)\n\nwith tf.GradientTape(persistent=True) as tape1: \n    tape1.watch([logits])\n    with tf.GradientTape() as tape2:\n        tape2.watch([logits])\n        loss = tf.reduce_sum(classic_ctc_loss(\n            labels=labels,\n            logits=logits,\n            label_length=label_length,\n            logit_length=logit_length,\n            blank_index=0,\n        ))\n    gradient = tape2.gradient(loss, sources=logits)\nhessian = tape1.batch_jacobian(gradient, source=logits, experimental_use_pfor=False)\n# shape = [2, 5, 3, 5, 3]\n```\n\n### 3. Numerical Stability\n1. The proposed implementation is more numerically stable, \nproducing reasonable outputs even for logits of order `1e+10` and `-tf.inf`.\n2. If the logit length is too short to predict the label output, \nthe loss is `tf.inf` for that sample, unlike `tf.nn.ctc_loss`, which might output `707.13184`.\n\n\n### 4. Pure Python Implementation\nThis is a pure Python/TensorFlow implementation, eliminating the need to build or compile any C++/CUDA components.\n\n\n## Usage\nThe interface is identical to `tensorflow.nn.ctc_loss` with `logits_time_major=False`.\n\nExample:\n```python\nimport tensorflow as tf\nfrom tf_seq2seq_losses import classic_ctc_loss\n\nbatch_size = 1\nnum_tokens = 3 # = 2 tokens + 1 blank token\nlogit_length = 5\nloss = classic_ctc_loss(\n    labels=tf.constant([[1, 2, 2, 1]], dtype=tf.int32),\n    logits=tf.zeros(shape=[batch_size, logit_length, num_tokens], dtype=tf.float32),\n    label_length=tf.constant([4], dtype=tf.int32),\n    logit_length=tf.constant([logit_length], dtype=tf.int32),\n    blank_index=0,\n)\n```\n\n## Under the Hood\nThe implementation uses TensorFlow operations such as tf.while_loop and tf.TensorArray. \nThe main computational bottleneck is the iteration over the logit length to calculate α and β \n(as described in the original\n[CTC paper](file:///home/alexey/Downloads/Connectionist_temporal_classification_Labelling_un.pdf)). \nThe expected gradient GPU calculation time is linear with respect to the logit length.\n\n## Known Issues\n### 1. Warning:\n\u003e AutoGraph could not transform \u003cfunction classic_ctc_loss at ...\u003e and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 \n(on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n\nObserved with TensorFlow version 2.4.1. \nThis warning does not affect performance and is caused by the use of Union in type annotations.\n\n### 2. UnimplementedError:\nUsing `tf.jacobian` and `tf.batch_jacobian` for the second derivative of classic_ctc_loss with \n`experimental_use_pfor=False` in `tf.GradientTape` may cause an unexpected `UnimplementedError` \nin TensorFlow version 2.4.1 or later. \nThis can be avoided by setting `experimental_use_pfor=True` \nor by using `ClassicCtcLossData.hessian` directly without `tf.GradientTape`.\n\nFeel free to reach out if you have any questions or need further clarification.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexeytochin%2Ftf_seq2seq_losses","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexeytochin%2Ftf_seq2seq_losses","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexeytochin%2Ftf_seq2seq_losses/lists"}