{"id":19352058,"url":"https://github.com/lukashedegaard/co3d","last_synced_at":"2025-04-23T07:31:17.063Z","repository":{"id":41657506,"uuid":"372476785","full_name":"LukasHedegaard/co3d","owner":"LukasHedegaard","description":"Official source code for \"Continual 3D Convolutional Neural Networks for Real-time Processing of Videos\" [ECCV2022]","archived":false,"fork":false,"pushed_at":"2022-12-06T13:47:39.000Z","size":110491,"stargazers_count":43,"open_issues_count":1,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-02T09:51:16.051Z","etag":null,"topics":["3d-cnn","cnn","continual-inference","convolutional-neural-networks","deep-learning","human-activity-recognition","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LukasHedegaard.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-31T11:05:22.000Z","updated_at":"2025-02-23T09:10:22.000Z","dependencies_parsed_at":"2023-01-24T02:00:10.689Z","dependency_job_id":null,"html_url":"https://github.com/LukasHedegaard/co3d","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasHedegaard%2Fco3d","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasHedegaard%2Fco3d/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasHedegaard%2Fco3d/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasHedegaard%2Fco3d/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LukasHedegaard","download_url":"https://codeload.github.com/LukasHedegaard/co3d/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250391166,"owners_count":21422850,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-cnn","cnn","continual-inference","convolutional-neural-networks","deep-learning","human-activity-recognition","pytorch"],"created_at":"2024-11-10T04:38:00.514Z","updated_at":"2025-04-23T07:31:12.042Z","avatar_url":"https://github.com/LukasHedegaard.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Continual 3D Convolutional Neural Networks\n[![Paper](http://img.shields.io/badge/paper-arxiv.2106.00050-B31B1B.svg)](https://arxiv.org/abs/2106.00050)\n[![Framework](https://img.shields.io/badge/Built_to-Ride-643DD9.svg)](https://github.com/LukasHedegaard/ride)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n\u003cdiv align=\"center\"\u003e\n\u003cvideo src='https://github.com/LukasHedegaard/co3d/raw/c5891aaf6f76bb8bda0ddef238dd7a0feb1afc38/presentation/5612.mp4' width=512/\u003e\n\u003c/div\u003e\n\nContinual 3D Convolutional Neural Networks (Co3D CNNs) are a novel computational formulation of spatio-temporal 3D CNNs, in which videos are processed frame-by-frame rather than by clip.\n\nIn online processing tasks demanding frame-wise predictions, Co3D CNNs dispense with the computational redundancies of regular 3D CNNs, namely the repeated convolutions over frames, which appear in multiple clips.\n\nCo3D CNNs are weight-compatible with regular 3D CNNs, do not need further training, and reduce the floating point operations for frame-wise computations by more than an order of magnitude!\n\n## News\n- 2022-07-04 Our paper _\"Continual 3D Convolutional Neural Networks for Real-time Processing of Videos\"_ has been accepted at the [European Conference on Computer Vision (ECCV) 2022](https://eccv2022.ecva.net).\n\n\n## Principle \n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"figures/coconv.png\" width=\"500\"\u003e\n  \u003cbr\u003e\n  Continual Convolution. \n\tAn input (green d or e) is convolved with a kernel (blue α, β). The intermediary feature-maps corresponding to all but the last temporal position are stored, while the last feature map and prior memory are summed to produce the resulting output. For a continual stream of inputs, Continual Convolutions produce identical outputs to regular convolutions.\n\u003c/div\u003e\n\n\n## Results\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"figures/acc-vs-flops.png\" width=\"500\"\u003e\n  \u003cbr\u003e\n  Accuracy/complexity trade-off for Continual X3D CoX3D and recent state-of-the-art 3D CNNs on Kinetics-400 using 1-clip/frame testing. \n  For regular 3D CNNs, the FLOPs per clip ■ are noted, while the FLOPs per frame ● are shown for the Continual 3D CNNs. \n  The CoX3D models used the weights from the X3D models without further fine-tuning.\n  The global average pool size for the network is noted in each point.\n  The diagonal and vertical arrows indicate respectively a transfer from regular to Continual 3D CNN and an extension of receptive field.\n\n  \u003cbr\u003e\n  \u003cbr\u003e\n\n\u003cimg src=\"figures/results.png\"\u003e\n\u003cbr\u003e\n  Benchmark of state-of-the-art methods on Kinetics-400. The noted accuracy is the single clip or frame top-1 score using RGB as the only input-modality. \n  The performance was evaluated using publicly available pre-trained models without any further fine-tuning.\n  For thoughput comparison, evaluations per second denote frames per second for the CoX3D models and clips per second for the remaining models. Throughput results are the mean +/- std of 100 measurements. Pareto-optimal models are marked with bold. Mem. is the maximum allocated memory during inference noted in megabytes.\n\u003c/div\u003e\n\n\n\n# Setup\n\n1. Clone the project code\n    ```bash\n    git clone https://github.com/LukasHedegaard/co3d\n    cd co3d\n    ```\n\n1. Create and activate `conda` environent (optional)\n    ```bash\n    conda create --name co3d python=3.8\n    conda activate co3d\n    ```\n\n1. Install Python dependencies\n    ```bash\n    pip install -e .[dev]\n    ``` \n\n1. Install [FFMPEG](https://ffmpeg.org) and [UNRAR](https://www.rarlab.com/rar_add.htm)\n\n1. Fill in the information on your dataset folder path in the `.env` file:\n    ```bash\n    DATASETS_PATH=/path/to/datasets\n    LOGS_PATH=/path/to/logs\n    CACHE_PATH=.cache\n    ```\n\n1. Download dataset using [these instructions](datasets/README.md)\n\n\n# Models\n\n## [CoX3D](models/cox3d/README.md)\n_CoX3D_ is the Continual-CNN implementation of X3D.\nIn contrast to regular 3D CNNs, which take a whole video clip as input, Continual CNNs operate frame-by-frame and can thus speed up computation by a significant margin.\n\n\n## [CoSlow](models/coslow/README.md)\n_CoSlow_ is the Continual-CNN implementation of Slow.\n\n\n## [CoI3D](models/coi3d/README.md)\n_CoSlow_ is the Continual-CNN implementation of I3d.\n\n\n## [X3D](models/x3d/README.md)\n_X3D_ [[ArXiv](https://arxiv.org/abs/2004.04730), [Repo](https://github.com/facebookresearch/SlowFast)] is a family of 3D variants of the EfficientNet achitecture, which produce state-of-the-art results for lightweight human activity recognition.\n\n\n## [R(2+1)D](models/r2plus1d/README.md)\n_R(2+1)D_ [[ArXiv](https://arxiv.org/abs/1705.07750), [Repo](https://pytorch.org/vision/stable/models.html#torchvision.models.video.r2plus1d_18)] is a CNN for activity recognition, which separates the 3D convolution into a spatial 2D convolution and a temporal 1D convolution in order to reduce the number of parameters and increase the network efficiency.\n\n\n## [I3D](models/i3d/README.md)\n_I3D_ [[ArXiv](https://arxiv.org/abs/1705.07750), [Repo](https://github.com/deepmind/kinetics-i3d)] is a 3D CNN for activity recognition, proposed to \"inflate\" the weights from a 2D CNN pretrained on ImageNet in the initialisation of the 3D CNN, thereby improving accuracy and reducing training time.\n\nThe implementation here is a port of the one found in the [SlowFast Repo](https://github.com/facebookresearch/SlowFast).\n\n\n## [SlowFast](models/slowfast/README.md)\n_SlowFast_ [[ArXiv](https://arxiv.org/abs/1812.03982), [Repo](https://github.com/facebookresearch/SlowFast)] is two-stream 3D CNNs architecture for video-recognition. The structure includes two pathways with one pathway operating at a slower frame-rate than the other.\n\n\n## [Slow](models/coresnet/README.md)\n_Slow_ is the \"slow\" branch of the SlowFast network [[ArXiv](https://arxiv.org/abs/1812.03982), [Repo](https://github.com/facebookresearch/SlowFast)]\n\n# Usage\nThe project code written in PyTorch and uses [Ride](https://github.com/LukasHedegaard/ride) to provide implementations of training, evaluations, and benchmarking methods.\nA plethora of usage options are available, which are best explored in the [Ride docs](https://ride.readthedocs.io) or the command-line help, e.g.:\n```bash\npython models/cox3d/main.py --help \n```\n\nThis repository contains the implementations of Continual X3D (CoX3D), as well as number of 3D-CNN baselines.\n\nEach model has its own folder with a self-contained implementation, scripts, weight download utilities, hparams and profiling results. \nOverview tables for scripts used to download weights, run the model test-sequences, and throughput benchmarks are found below:\n\n## Download weights\n| Model         | Dataset  | Download |\n| -------       | -------- | -------- |\n| I3D-R50       | Kinetics | [download](https://dl.fbaipublicfiles.com/pyslowfast/model_zoo/kinetics400/I3D_8x8_R50.pkl)\n| R(2+1)D-18    | Kinetics | [download](https://download.pytorch.org/models/r2plus1d_18-91a641e6.pth)\n| SlowFast-8x8  | Kinetics | [download](https://dl.fbaipublicfiles.com/pyslowfast/model_zoo/kinetics400/SLOWFAST_8x8_R50.pkl)\n| SlowFast-4x16 | Kinetics | [download](https://dl.fbaipublicfiles.com/pyslowfast/model_zoo/kinetics400/SLOWFAST_4x16_R50.pkl)\n| Slow-8x8      | Kinetics | [download](https://dl.fbaipublicfiles.com/pytorchvideo/model_zoo/kinetics/SLOW_8x8_R50.pyth)\n| (Co)X3D-XS    | Kinetics | [download](https://dl.fbaipublicfiles.com/pyslowfast/x3d_models/x3d_xs.pyth)\n| (Co)X3D-S     | Kinetics | [download](https://dl.fbaipublicfiles.com/pyslowfast/x3d_models/x3d_s.pyth)\n| (Co)X3D-M     | Kinetics | [download](https://dl.fbaipublicfiles.com/pyslowfast/x3d_models/x3d_m.pyth)\n| (Co)X3D-L     | Kinetics | [download](https://dl.fbaipublicfiles.com/pyslowfast/x3d_models/x3d_l.pyth)\n| (Co)Slow-8x8  | Charades | [download](https://dl.fbaipublicfiles.com/pytorchvideo/model_zoo/charades/SLOW_8x8_R50.pyth)\n\n\n## Evaluate on Kinetics400\nEvaluate the 1-clip accuracy of pretrained models. \nThe scripts should be executed from project root.\n\n| Model         | Script |\n| -------       | -------- | \n| I3D-R50       | [`./models/i3d/scripts/test/kinetics400.sh`](models/i3d/scripts/test/kinetics400.sh) | \n| R(2+1)D-18    | [`./models/r2plus1d/scripts/test/kinetics400.sh`](models/r2plus1d/scripts/test/kinetics400.sh) | \n| SlowFast      | [`./models/slowfast/scripts/test/kinetics400.sh`](models/slowfast/scripts/test/kinetics400.sh) | \n| Slow          | [`./models/slow/scripts/test/kinetics400.sh`](models/slow/scripts/test/kinetics400.sh) | \n| X3D           | [`./models/x3d/scripts/test/kinetics400.sh`](models/x3d/scripts/test/kinetics400.sh) | \n| CoX3D         | [`./models/cox3d/scripts/test/kinetics400.sh`](models/cox3d/scripts/test/kinetics400.sh) | \n| CoSlow        | [`./models/coslow/scripts/test/kinetics400.sh`](models/coslow/scripts/test/kinetics400.sh) | \n| CoI3D         | [`./models/coi3d/scripts/test/kinetics400.sh`](models/coi3d/scripts/test/kinetics400.sh) | \n\n\n## Evaluate on Charades\nEvaluate the 1-clip accuracy of pretrained models. \nThe scripts should be executed from project root.\n\n| Model         | Script |\n| -------       | -------- | \n| (Co)Slow-8x8       | [`./models/coslow/scripts/test/charades.sh`](models/coslow/scripts/test/charades.sh) | \n\n\n## Benchmark FLOPs and throughput\nThe scripts should be executed from project root.\n\n| Model         | Script |\n| -------       | -------- | \n| I3D-R50       | [`./models/i3d/scripts/profile/kinetics400.sh`](models/i3d/scripts/profile/kinetics400.sh) | \n| R(2+1)D-18    | [`./models/r2plus1d/scripts/profile/kinetics400.sh`](models/r2plus1d/scripts/profile/kinetics400.sh) | \n| SlowFast      | [`./models/slowfast/scripts/profile/kinetics400.sh`](models/slowfast/scripts/profile/kinetics400.sh) | \n| Slow          | [`./models/slow/scripts/profile/kinetics400.sh`](models/slow/scripts/profile/kinetics400.sh) | \n| X3D           | [`./models/x3d/scripts/profile/kinetics400.sh`](models/x3d/scripts/profile/kinetics400.sh) | \n| CoX3D         | [`./models/cox3d/scripts/profile/kinetics400.sh`](models/cox3d/scripts/profile/kinetics400.sh) | \n| CoI3D         | [`./models/coi3d/scripts/profile/kinetics400.sh`](models/coi3d/scripts/profile/kinetics400.sh) | \n| CoSlow        | [`./models/coslow/scripts/profile/kinetics400.sh`](models/coslow/scripts/profile/kinetics400.sh) | \n\n\n# Citation   \n```\n@inproceedings{hedegaard2022continual,\n    title={Continual 3D Convolutional Neural Networks for Real-time Processing of Videos},\n    author={Lukas Hedegaard and Alexandros Iosifidis},\n    booktitle={European Conference on Computer Vision (ECCV)},\n    year={2022},\n}\n```\n\n## Acknowledgement\nThis work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871449 (OpenDR).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukashedegaard%2Fco3d","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flukashedegaard%2Fco3d","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukashedegaard%2Fco3d/lists"}