{"id":19352013,"url":"https://github.com/lukashedegaard/continual-inference","last_synced_at":"2025-04-06T00:08:14.447Z","repository":{"id":38833755,"uuid":"394247985","full_name":"LukasHedegaard/continual-inference","owner":"LukasHedegaard","description":"A Python library for Continual Inference Networks in PyTorch","archived":false,"fork":false,"pushed_at":"2025-03-15T11:39:15.000Z","size":2059,"stargazers_count":49,"open_issues_count":0,"forks_count":11,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-29T23:08:22.666Z","etag":null,"topics":["cnn","continual-inference","convolutional-neural-networks","deep-learning","deep-neural-networks","efficiency","machine-learning","neural-networks","pytorch","recurrent-neural-network"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LukasHedegaard.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-09T10:30:15.000Z","updated_at":"2025-03-15T11:32:32.000Z","dependencies_parsed_at":"2023-12-26T06:35:12.532Z","dependency_job_id":"6f3c305f-cf06-4f5d-ae5a-8b4b55a8dcb6","html_url":"https://github.com/LukasHedegaard/continual-inference","commit_stats":{"total_commits":435,"total_committers":2,"mean_commits":217.5,"dds":0.03448275862068961,"last_synced_commit":"b75acad64abf26ffd5ae693bf6eecff7536468cf"},"previous_names":[],"tags_count":46,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasHedegaard%2Fcontinual-inference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasHedegaard%2Fcontinual-inference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasHedegaard%2Fcontinual-inference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasHedegaard%2Fcontinual-inference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LukasHedegaard","download_url":"https://codeload.github.com/LukasHedegaard/continual-inference/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247415967,"owners_count":20935387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cnn","continual-inference","convolutional-neural-networks","deep-learning","deep-neural-networks","efficiency","machine-learning","neural-networks","pytorch","recurrent-neural-network"],"created_at":"2024-11-10T04:37:51.941Z","updated_at":"2025-04-06T00:08:14.420Z","avatar_url":"https://github.com/LukasHedegaard.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"https://raw.githubusercontent.com/LukasHedegaard/continual-inference/main/figures/logo/logo_name.svg\" style=\"width: 400px;\"\u003e\n\n__A Python library for Continual Inference Networks in PyTorch__\n\n[Quick-start](https://continual-inference.readthedocs.io/en/latest/generated/README.html#quick-start) • \n[Docs](https://continual-inference.readthedocs.io/en/latest/generated/README.html) • \n[Principles](https://continual-inference.readthedocs.io/en/latest/generated/README.html#library-principles) • \n[Paper](https://arxiv.org/abs/2204.03418) • \n[Examples](https://continual-inference.readthedocs.io/en/latest/generated/README.html#composition-examples) • \n[Modules](https://continual-inference.readthedocs.io/en/latest/common/modules.html) • \n[Model Zoo](https://continual-inference.readthedocs.io/en/latest/generated/README.html#model-zoo-and-benchmarks) • \n[Contribute](https://continual-inference.readthedocs.io/en/latest/generated/CONTRIBUTING.html) • \n[License](https://github.com/LukasHedegaard/continual-inference/blob/main/LICENSE)\n\n\u003cdiv\u003e\n  \u003ca href=\"https://pypi.org/project/continual-inference/\" style=\"display:inline-block;\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/pyversions/continual-inference\" height=\"20\" \u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://badge.fury.io/py/continual-inference\" style=\"display:inline-block;\"\u003e\n    \u003cimg src=\"https://badge.fury.io/py/continual-inference.svg\" height=\"20\" \u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://continual-inference.readthedocs.io/en/latest/generated/README.html\" style=\"display:inline-block;\"\u003e\n    \u003cimg src=\"https://readthedocs.org/projects/continual-inference/badge/?version=latest\" alt=\"Documentation Status\" height=\"20\"/\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pepy.tech/project/continual-inference\" style=\"display:inline-block;\"\u003e\n    \u003cimg src=\"https://static.pepy.tech/badge/continual-inference\" height=\"20\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://codecov.io/gh/LukasHedegaard/continual-inference\" style=\"display:inline-block;\"\u003e\n    \u003cimg src=\"https://codecov.io/gh/LukasHedegaard/continual-inference/branch/main/graph/badge.svg?token=XW1UQZSEOG\" height=\"20\"/\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://opensource.org/licenses/Apache-2.0\" style=\"display:inline-block;\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/License-Apache%202.0-blue.svg\" height=\"20\"\u003e\n  \u003c/a\u003e\n  \u003c!-- \u003ca href=\"https://arxiv.org/abs/2204.03418\" style=\"display:inline-block;\"\u003e\n    \u003cimg src=\"http://img.shields.io/badge/paper-arxiv.2204.03418-B31B1B.svg\" height=\"20\" \u003e\n  \u003c/a\u003e --\u003e\n  \u003ca href=\"https://github.com/psf/black\" style=\"display:inline-block;\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/code%20style-black-000000.svg\" height=\"20\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.codefactor.io/repository/github/lukashedegaard/continual-inference/overview/main\" style=\"display:inline-block;\"\u003e\n    \u003cimg src=\"https://www.codefactor.io/repository/github/lukashedegaard/continual-inference/badge/main\" alt=\"We match PyTorch interfaces exactly. Method arguments named 'input' reduce the codefactor to 'A-'\" height=\"20\" /\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n## Continual Inference Networks ensure efficient stream processing\nMany of our favorite Deep Neural Network architectures (e.g., [CNNs](https://arxiv.org/abs/2106.00050) and [Transformers](https://arxiv.org/abs/2201.06268)) were built with offline-processing for offline processing. Rather than processing inputs one sequence element at a time, they require the whole (spatio-)temporal sequence to be passed as a single input.\nYet, **many important real-life applications need online predictions on a continual input stream**. \nWhile CNNs and Transformers can be applied by re-assembling and passing sequences within a sliding window, this is _inefficient_ due to the redundant intermediary computations from overlapping clips.\n\n**Continual Inference Networks** (CINs) are built to ensure efficient stream processing by employing an alternative computational ordering, which allows sequential computations without the use of sliding window processing.\nIn general, CINs requires approx. _L_ ×  fewer FLOPs per prediction compared to sliding window-based inference with non-CINs, where _L_ is the corresponding sequence length of a non-CIN network. For more details, check out the videos below describing Continual 3D CNNs [[1](https://arxiv.org/abs/2106.00050)] and Transformers [[2](https://arxiv.org/abs/2201.06268)].\n\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"http://www.youtube.com/watch?feature=player_embedded\u0026v=Jm2A7dVEaF4\" target=\"_blank\"\u003e\n     \u003cimg src=\"http://img.youtube.com/vi/Jm2A7dVEaF4/hqdefault.jpg\" alt=\"Presentation of Continual 3D CNNs\" style=\"width:240px;height:auto;\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"http://www.youtube.com/watch?feature=player_embedded\u0026v=gy802Tlp-eQ\" target=\"_blank\"\u003e\n     \u003cimg src=\"http://img.youtube.com/vi/gy802Tlp-eQ/hqdefault.jpg\" alt=\"Presentation of Continual Transformers\" style=\"width:240px;height:auto;\" /\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n## News\n- 2022-12-02: ONNX compatibility for all modules is available from v1.0.0. See [test_onnx.py](tests/continual/test_onnx.py) for examples.\n\n\n## Quick-start\n\n### Install \n```bash\npip install continual-inference\n```\n\n\n\n### Example\n`co` modules are weight-compatible drop-in replacement for `torch.nn`, enhanced with the capability of efficient _continual inference_:\n\n```python3\nimport torch\nimport continual as co\n                                                           \n#                      B, C, T, H, W\nexample = torch.randn((1, 1, 5, 3, 3))\n\nconv = co.Conv3d(in_channels=1, out_channels=1, kernel_size=(3, 3, 3))\n\n# Same exact computation as torch.nn.Conv3d ✅\noutput = conv(example)\n\n# But can also perform online inference efficiently 🚀\nfirsts = conv.forward_steps(example[:, :, :4])\nlast = conv.forward_step(example[:, :, 4])\n\nassert torch.allclose(output[:, :, : conv.delay], firsts)\nassert torch.allclose(output[:, :, conv.delay], last)\n\n# Temporal properties\nassert conv.receptive_field == 3\nassert conv.delay == 2\n```\n\nSee the [network composition](#composition) and [model zoo](#model-zoo-and-benchmarks) sections for additional examples.\n\n## Library principles\n\n### Forward modes\nThe library components feature three distinct forward modes, which are handy for different situations, namely `forward`, `forward_step`, and `forward_steps`:\n\n#### `forward(input)`\nPerforms a forward computation over multiple time-steps. This function is identical to the corresponding module in _torch.nn_, ensuring cross-compatibility. Moreover, it's handy for efficient training on clip-based data.\n\n```\n         O            (O: output)\n         ↑ \n         N            (N: network module)\n         ↑ \n -----------------    (-: aggregation)\n P   I   I   I   P    (I: input frame, P: padding)\n```\n\n\n#### `forward_step(input, update_state=True)`\nPerforms a forward computation for a single frame and (optionally) updates internal states accordingly. This function performs efficient continual inference.\n\n```\nO+S O+S O+S O+S   (O: output, S: updated internal state)\n ↑   ↑   ↑   ↑ \n N   N   N   N    (N: network module)\n ↑   ↑   ↑   ↑ \n I   I   I   I    (I: input frame)\n```\n\n#### `forward_steps(input, pad_end=False, update_state=True)`\nPerforms a forward computation across multiple time-steps while updating internal states for continual inference (if update_state=True).\nStart-padding is always accounted for, but end-padding is omitted per default in expectance of the next input step. It can be added by specifying pad_end=True. If so, the output-input mapping the exact same as that of forward.\n```\n         O            (O: output)\n         ↑ \n -----------------    (-: aggregation)\n O  O+S O+S O+S  O    (O: output, S: updated internal state)\n ↑   ↑   ↑   ↑   ↑\n N   N   N   N   N    (N: network module)\n ↑   ↑   ↑   ↑   ↑\n P   I   I   I   P    (I: input frame, P: padding)\n```\n\n#### `__call__`\nPer default, the `__call__` function operates identically to _torch.nn_ and executes forward. We supply two options for changing this behavior, namely the _call_mode_ property and the _call_mode_ context manager. An example of their use follows:\n\n```python\ntimeseries = torch.randn(batch, channel, time)\ntimestep = timeseries[:, :, 0]\n\nnet(timeseries)  # Invokes net.forward(timeseries)\n\n# Assign permanent call_mode property\nnet.call_mode = \"forward_step\"\nnet(timestep)  # Invokes net.forward_step(timestep)\n\n# Assign temporary call_mode with context manager\nwith co.call_mode(\"forward_steps\"):\n    net(timeseries)  # Invokes net.forward_steps(timeseries)\n\nnet(timestep)  # Invokes net.forward_step(timestep) again\n```\n\n### Composition\n\nContinual Inference Networks require strict handling of internal data delays to guarantee correspondence between [forward modes](#forward-modes). While it is possible to compose neural networks by defining _forward_, _forward_step_, and _forward_steps_ manually, correct handling of delays is cumbersome and time-consuming. Instead, we provide a rich interface of container modules, which handles delays automatically. On top of `co.Sequential` (which is a drop-in replacement of _torch.nn.Sequential_), we provide modules for handling parallel and conditional dataflow. \n\n- [`co.Sequential`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Sequential.html): Invoke modules sequentially, passing the output of one module onto the next.\n- [`co.Broadcast`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Broadcast.html): Broadcast one stream to multiple.\n- [`co.Parallel`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Parallel.html): Invoke modules in parallel given each their input.\n- [`co.ParallelDispatch`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.ParallelDispatch.html): Dispatch multiple input streams to multiple output streams flexibly.\n- [`co.Reduce`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Reduce.html): Reduce multiple input streams to one.\n- [`co.BroadcastReduce`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.BroadcastReduce.html): Shorthand for Sequential(Broadcast, Parallel, Reduce).\n- [`co.Residual`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Residual.html): Residual connection.\n- [`co.Conditional`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Conditional.html): Conditionally checks whether to invoke a module (or another) at runtime.\n\n\n#### Composition examples:\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eResidual module\u003c/b\u003e\u003c/summary\u003e\n\nShort-hand:\n```python3\nresidual = co.Residual(co.Conv3d(32, 32, kernel_size=3, padding=1))\n```\n\nExplicit:\n```python3\nresidual = co.Sequential(\n    co.Broadcast(2),\n    co.Parallel(\n        co.Conv3d(32, 32, kernel_size=3, padding=1),\n        co.Delay(2),\n    ),\n    co.Reduce(\"sum\"),\n)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e3D MobileNetV2 Inverted residual block\u003c/b\u003e\u003c/summary\u003e\n\nContinual 3D version of the [MobileNetV2 Inverted residual block](https://arxiv.org/pdf/1801.04381.pdf).\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/LukasHedegaard/continual-inference/main/figures/examples/mb_conv.png\" style=\"width: 15vw; min-width: 200px;\"\u003e\n  \u003cbr\u003e\n  MobileNetV2 Inverted residual block. Source: https://arxiv.org/pdf/1801.04381.pdf\n\u003c/div\u003e\n\n```python3\nmb_conv = co.Residual(\n    co.Sequential(\n      co.Conv3d(32, 64, kernel_size=(1, 1, 1)),\n      nn.BatchNorm3d(64),\n      nn.ReLU6(),\n      co.Conv3d(64, 64, kernel_size=(3, 3, 3), padding=(1, 1, 1), groups=64),\n      nn.ReLU6(),\n      co.Conv3d(64, 32, kernel_size=(1, 1, 1)),\n      nn.BatchNorm3d(32),\n    )\n)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e3D Squeeze-and-Excitation module\u003c/b\u003e\u003c/summary\u003e\n\nContinual 3D version of the [Squeeze-and-Excitation module](https://arxiv.org/pdf/1709.01507.pdf)\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/LukasHedegaard/continual-inference/main/figures/examples/se_block.png\" style=\"width: 15vw; min-width: 200px;\"\u003e\n  \u003cbr\u003e\n  Squeeze-and-Excitation block. \n  Scale refers to a broadcasted element-wise multiplication.\n  Adapted from: https://arxiv.org/pdf/1709.01507.pdf\n\u003c/div\u003e\n\n```python3\nse = co.Residual(\n    co.Sequential(\n        OrderedDict([\n            (\"pool\", co.AdaptiveAvgPool3d((1, 1, 1), kernel_size=7)),\n            (\"down\", co.Conv3d(256, 16, kernel_size=1)),\n            (\"act1\", nn.ReLU()),\n            (\"up\", co.Conv3d(16, 256, kernel_size=1)),\n            (\"act2\", nn.Sigmoid()),\n        ])\n    ),\n    reduce=\"mul\",\n)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e3D Inception module\u003c/b\u003e\u003c/summary\u003e\n\nContinual 3D version of the [Inception module](https://arxiv.org/pdf/1409.4842v1.pdf):\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/LukasHedegaard/continual-inference/main/figures/examples/inception_block.png\" style=\"width: 25vw; min-width: 350px;\"\u003e\n  \u003cbr\u003e\n  Inception module. Source: https://arxiv.org/pdf/1409.4842v1.pdf\n   \n\u003c/div\u003e\n\n```python3\ndef norm_relu(module, channels):\n    return co.Sequential(\n        module,\n        nn.BatchNorm3d(channels),\n        nn.ReLU(),\n    )\n\ninception_module = co.BroadcastReduce(\n    co.Conv3d(192, 64, kernel_size=1),\n    co.Sequential(\n        norm_relu(co.Conv3d(192, 96, kernel_size=1), 96),\n        norm_relu(co.Conv3d(96, 128, kernel_size=3, padding=1), 128),\n    ),\n    co.Sequential(\n        norm_relu(co.Conv3d(192, 16, kernel_size=1), 16),\n        norm_relu(co.Conv3d(16, 32, kernel_size=5, padding=2), 32),\n    ),\n    co.Sequential(\n        co.MaxPool3d(kernel_size=(1, 3, 3), padding=(0, 1, 1), stride=1),\n        norm_relu(co.Conv3d(192, 32, kernel_size=1), 32),\n    ),\n    reduce=\"concat\",\n)\n```\n\u003c/details\u003e\n\n\n### Input shapes\nWe enforce a unified ordering of input dimensions for all library modules, namely:\n\n    (batch, channel, time, optional_dim2, optional_dim3)\n\n### Outputs\nThe outputs produces by `forward_step` and `forward_steps` are identical to those of `forward`, provided the same data was input beforehand and state update was enabled. We know that input and output shapes aren't necessarily the same when using `forward` in the PyTorch library, and  generally depends on padding, stride and receptive field of a module. \n\nFor the `forward_step` function, this comes to show by some `None`-valued outputs. Specifically, modules with a _delay_ (i.e. with receptive fields larger than the padding + 1) will produce `None` until the input count exceeds the delay. Moreover, _stride_ \u003e 1 will produce `Tensor` outputs every _stride_ steps and `None` the remaining steps. A visual example is shown below:\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/LukasHedegaard/continual-inference/main/figures/continual/continual-stride.png\" style=\"width:300px;height:auto;\"/\u003e\n  \u003c/br\u003e\n  A mixed example of delay and outputs under padding and stride. Here, we illustrate the step-wise operation of two co module layers, l1 with with receptive_field = 3, padding = 2, and stride = 2 and l2 with receptive_field = 3, no padding and stride = 1. ⧇ denotes a padded zero, ■ is a non-zero step-feature, and ☒ is an empty output.\n\u003c/div\u003e\n\nFor more information, please see the [library paper](https://arxiv.org/abs/2204.03418).\n\n### Handling state\nDuring stream processing, network modules which operate over multiple time-steps, e.g., a convolution with `kernel_size \u003e 1` in the temporal dimension, will aggregate and cache state internally. Each module has its own local state, which can be inspected using `module.get_state()`. During `forward_step` and `forward_steps`, the state is updated unless the forward_step(s) is invoked with an `update_state = False` argument.\n\nA __state cleanup__ can be accomplished via `module.clean_state()`.\n\n\n## Module library\n_Continual Inference_ features a rich collection of modules for defining Continual Inference Networks. Specific care was taken to create CIN versions of the PyTorch modules found in [_torch.nn_](https://pytorch.org/docs/stable/nn.html):\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eConvolutions\u003c/b\u003e\u003c/summary\u003e\n\n- [`co.Conv1d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Conv1d.html)\n- [`co.Conv2d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Conv2d.html)\n- [`co.Conv3d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Conv3d.html)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003ePooling\u003c/b\u003e\u003c/summary\u003e\n\n  - [`co.AvgPool1d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.AvgPool1d.html)\n  - [`co.AvgPool2d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.AvgPool2d.html)\n  - [`co.AvgPool3d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.AvgPool3d.html)\n  - [`co.MaxPool1d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.MaxPool1d.html)\n  - [`co.MaxPool2d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.MaxPool2d.html)\n  - [`co.MaxPool3d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.MaxPool3d.html)\n  - [`co.AdaptiveAvgPool2d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.AdaptiveAvgPool2d.html)\n  - [`co.AdaptiveAvgPool3d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.AdaptiveAvgPool3d.html)\n  - [`co.AdaptiveMaxPool2d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.AdaptiveMaxPool2d.html)\n  - [`co.AdaptiveMaxPool3d`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.AdaptiveMaxPool3d.html)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eLinear\u003c/b\u003e\u003c/summary\u003e\n\n  - [`co.Linear`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Linear.html)\n  - [`co.Identity`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Identity.html): Maps input to output without modification.\n  - [`co.Add`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Add.html): Adds a constant value.\n  - [`co.Multiply`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Multiply.html): Multiplies with a constant factor.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eRecurrent\u003c/b\u003e\u003c/summary\u003e\n\n  - [`co.RNN`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.RNN.html)\n  - [`co.LSTM`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.LSTM.html)\n  - [`co.GRU`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.GRU.html)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eTransformers\u003c/b\u003e\u003c/summary\u003e\n\n  - [`co.TransformerEncoder`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.TransformerEncoder.html)\n  - [`co.TransformerEncoderLayerFactory`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.TransformerEncoderLayerFactory.html): Factory function corresponding to `nn.TransformerEncoderLayer`.\n  - [`co.SingleOutputTransformerEncoderLayer`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.SingleOutputTransformerEncoderLayer.html): SingleOutputMHA version of `nn.TransformerEncoderLayer`.\n  - [`co.RetroactiveTransformerEncoderLayer`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.RetroactiveTransformerEncoderLayer.html): RetroactiveMHA version of `nn.TransformerEncoderLayer`.\n  - [`co.RetroactiveMultiheadAttention`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.retroactive_mha.html.RetroactiveMultiheadAttention): Retroactive version of `nn.MultiheadAttention`.\n  - [`co.SingleOutputMultiheadAttention`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.single_output_mha.html.SingleOutputMultiheadAttention): Single-output version of `nn.MultiheadAttention`.\n  - [`co.RecyclingPositionalEncoding`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.RecyclingPositionalEncoding.html): Positional Encoding used for Continual Transformers.\n\n\u003c/details\u003e\n\n\nModules for composing and converting networks. Both _composition_ and _utility_ modules can be used for regular definition of PyTorch modules as well.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eComposition modules\u003c/b\u003e\u003c/summary\u003e\n\n  - [`co.Sequential`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Sequential.html): Invoke modules sequentially, passing the output of one module onto the next.\n  - [`co.Broadcast`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Broadcast.html): Broadcast one stream to multiple.\n  - [`co.Parallel`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Parallel.html): Invoke modules in parallel given each their input.\n  - [`co.ParallelDispatch`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.ParallelDispatch.html): Dispatch multiple input streams to multiple output streams flexibly.\n  - [`co.Reduce`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Reduce.html): Reduce multiple input streams to one.\n  - [`co.BroadcastReduce`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.BroadcastReduce.html): Shorthand for Sequential(Broadcast, Parallel, Reduce).\n  - [`co.Residual`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Residual.html): Residual connection.\n  - [`co.Conditional`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Conditional.html): Conditionally checks whether to invoke a module (or another) at runtime.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eUtility modules\u003c/b\u003e\u003c/summary\u003e\n\n  - [`co.Delay`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Delay.html): Pure delay module (e.g. needed in residuals).\n  - [`co.Skip`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Skip.html): Skip a predefined number of input steps.\n  - [`co.Reshape`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Reshape.html): Reshape non-temporal dimensions.\n  - [`co.Lambda`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Lambda.html): Lambda module which wraps any function.\n  - [`co.Constant`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Constant.html): Maps input to and output with constant value.\n  - [`co.Zero`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.Zero.html): Maps input to output of zeros.\n  - [`co.One`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.One.html): Maps input to output of ones.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eConverters\u003c/b\u003e\u003c/summary\u003e\n\n  - [`co.continual`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.continual.html): conversion function from `torch.nn` modules to `co` modules.\n  - [`co.forward_stepping`](https://continual-inference.readthedocs.io/en/latest/common/generated/continual.forward_stepping.html): functional wrapper, which enhances temporally local `torch.nn` modules with the forward_stepping functions.\n\n\u003c/details\u003e\n\nWe support drop-in interoperability with with the following _torch.nn_ modules:\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eActivation\u003c/b\u003e\u003c/summary\u003e\n\n  - `nn.Threshold`\n  - `nn.ReLU`\n  - `nn.RReLU`\n  - `nn.Hardtanh`\n  - `nn.ReLU6`\n  - `nn.Sigmoid`\n  - `nn.Hardsigmoid`\n  - `nn.Tanh`\n  - `nn.SiLU`\n  - `nn.Hardswish`\n  - `nn.ELU`\n  - `nn.CELU`\n  - `nn.SELU`\n  - `nn.GLU`\n  - `nn.GELU`\n  - `nn.Hardshrink`\n  - `nn.LeakyReLU`\n  - `nn.LogSigmoid`\n  - `nn.Softplus`\n  - `nn.Softshrink`\n  - `nn.PReLU`\n  - `nn.Softsign`\n  - `nn.Tanhshrink`\n  - `nn.Softmin`\n  - `nn.Softmax`\n  - `nn.Softmax2d`\n  - `nn.LogSoftmax`\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eNormalization\u003c/b\u003e\u003c/summary\u003e\n\n  - `nn.BatchNorm1d`\n  - `nn.BatchNorm2d`\n  - `nn.BatchNorm3d`\n  - `nn.GroupNorm`,\n  - `nn.InstanceNorm1d` (affine=True, track_running_stats=True required)\n  - `nn.InstanceNorm2d` (affine=True, track_running_stats=True required)\n  - `nn.InstanceNorm3d` (affine=True, track_running_stats=True required)\n  - `nn.LayerNorm` (only non-temporal dimensions must be specified)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eDropout\u003c/b\u003e\u003c/summary\u003e\n\n  - `nn.Dropout`\n  - `nn.Dropout1d`\n  - `nn.Dropout2d`\n  - `nn.Dropout3d`\n  - `nn.AlphaDropout`\n  - `nn.FeatureAlphaDropout`\n\n\u003c/details\u003e\n\n\n## Model Zoo and Benchmarks\n\n### Continual 3D CNNs\n\nBenchmark results for 1-view testing on __Kinetics400__. For reference, _X3D-L_ scores 69.3% top-1 acc with 19.2 GFLOPs per prediction. \n\nArch     | Avg. pool size | Top 1 (%) | FLOPs (G) per step | FLOPs reduction | Params (M) | Code                                                                   | Weights\n-------- | -------------- | --------- | ------------------ | --------------- | ---------- | ---------------------------------------------------------------------- | ---- \nCoX3D-L  | 64             | 71.6      | 1.25               | 15.3x           | 6.2        | [link](https://github.com/LukasHedegaard/co3d/tree/main/models/cox3d)  | [link](https://dl.fbaipublicfiles.com/pytorchvideo/model_zoo/kinetics/X3D\\_L.pyth)\nCoX3D-M  | 64             | 71.0      | 0.33               | 15.1x           | 3.8        | [link](https://github.com/LukasHedegaard/co3d/tree/main/models/cox3d)  | [link](https://dl.fbaipublicfiles.com/pytorchvideo/model_zoo/kinetics/X3D\\_M.pyth)\nCoX3D-S  | 64             | 64.7      | 0.17               | 12.1x           | 3.8        | [link](https://github.com/LukasHedegaard/co3d/tree/main/models/cox3d)  | [link](https://dl.fbaipublicfiles.com/pytorchvideo/model_zoo/kinetics/X3D\\_S.pyth)\nCoSlow   | 64             | 73.1      | 6.90               |  8.0x           | 32.5       | [link](https://github.com/LukasHedegaard/co3d/tree/main/models/coslow) | [link](https://dl.fbaipublicfiles.com/pytorchvideo/model_zoo/kinetics/SLOW\\_8x8\\_R50.pyth)\nCoI3D    | 64             | 64.0      | 5.68               |  5.0x           | 28.0       | [link](https://github.com/LukasHedegaard/co3d/tree/main/models/coi3d)  | [link](https://dl.fbaipublicfiles.com/pytorchvideo/model_zoo/kinetics/I3D\\_8x8\\_R50.pyth)\n\nFLOPs reduction is noted relative to non-continual inference.\nNote that [on-hardware inference](https://arxiv.org/abs/2106.00050) doesn't reach the same speedups as \"FLOPs reductions\" might suggest due to overhead of state reads and writes. This overhead is less important for large batch sizes. This applies to all models in the model zoo.\n\n### Continual ST-GCNs\n\nBenchmark results for on __NTU RGB+D 60__ for the joint modality. For reference, _ST-GCN_ achieves 86% X-Sub and 93.4 X-View accuracy with 16.73 GFLOPs per prediction. \n\nArch      | Receptive field | X-Sub Acc (%) | X-View Acc (%) | FLOPs (G) per step | FLOPs reduction | Params (M) | Code                                                                  \n--------  | --------------- | ------------- | -------------- | ------------------ | --------------- | ---------- | -----\nCoST-GCN  | 300             | 86.3          | 93.8           | 0.16               | 107.7x          | 3.1        | [link](https://github.com/LukasHedegaard/continual-skeletons/blob/main/models/cost_gcn_mod/cost_gcn_mod.py)\nCoA-GCN   | 300             | 84.1          | 92.6           | 0.17               | 108.7x          | 3.5        | [link](https://github.com/LukasHedegaard/continual-skeletons/blob/main/models/coa_gcn_mod/coa_gcn_mod.py)\nCoST-GCN  | 300             | 86.3          | 92.4           | 0.15               | 107.6x          | 3.1        | [link](https://github.com/LukasHedegaard/continual-skeletons/blob/main/models/cos_tr_mod/cos_tr_mod.py)\n\n[Here](https://drive.google.com/drive/u/4/folders/1m6aV5Zv8tAytvxF6qY4m9nyqlkKv0y72), you can download pre-trained,model weights for the above architectures on NTU RGB+D 60, NTU RGB+D 120, and Kinetics-400 on joint and bone modalities.\n\n\n### Continual Transformers\n\nBenchmark results for on __THUMOS14__ on top of features extracted using a TSN-ResNet50 backbone pre-trained on Kinetics400. For reference, _OadTR_ achieves 64.4 % mAP with 2.5 GFLOPs per prediction. \n\nArch        | Receptive field | mAP (%) | FLOPs (G) per step |  Params (M) | Code                                                                  \n----------  | --------------- | ------- | ------------------ |  ---------- | -----\nCoOadTR-b1  | 64              | 64.2    | 0.41               |  15.9       | [link](https://github.com/LukasHedegaard/CoOadTR)\nCoOadTR-b2  | 64              | 64.4    | 0.01               |   9.6       | [link](https://github.com/LukasHedegaard/CoOadTR)\n\nThe library features complete implementations of the [one](https://github.com/LukasHedegaard/continual-inference/blob/9895344f50a93ebb5cf5c4f26ecfdf27b6a3fe75/tests/continual/test_transformer.py#L8)- and [two](https://github.com/LukasHedegaard/continual-inference/blob/9895344f50a93ebb5cf5c4f26ecfdf27b6a3fe75/tests/continual/test_transformer.py#L59)-block continual transformer encoders as well.\n\n\n## Compatibility\nThe library modules are built to integrate seamlessly with other PyTorch projects.\nSpecifically, extra care was taken to ensure out-of-the-box compatibility with:\n- [pytorch-lightning](https://github.com/PyTorchLightning/pytorch-lightning)\n- [ptflops](https://github.com/sovrasov/flops-counter.pytorch)\n- [ride](https://github.com/LukasHedegaard/ride)\n- [onnx](https://github.com/onnx/onnx)\n\u003c!-- - [onnxruntime](https://github.com/microsoft/onnxruntime) --\u003e\n\n\n## Citation\n\u003ca href=\"https://arxiv.org/abs/2204.03418\" style=\"display:inline-block;\"\u003e\n  \u003cimg src=\"http://img.shields.io/badge/paper-arxiv.2204.03418-B31B1B.svg\" height=\"20\" \u003e\n\u003c/a\u003e\n\n```bibtex\n@inproceedings{hedegaard2022colib,\n  title={Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch},\n  author={Lukas Hedegaard and Alexandros Iosifidis},\n  booktitle={European Conference on Computer Vision Workshops (ECCVW)},\n  year={2022}\n}\n```\n\n\n## Acknowledgement\nThis work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871449 (OpenDR).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukashedegaard%2Fcontinual-inference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flukashedegaard%2Fcontinual-inference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukashedegaard%2Fcontinual-inference/lists"}