{"id":15628925,"url":"https://github.com/fcakyon/content-moderation-deep-learning","last_synced_at":"2026-02-21T19:07:12.005Z","repository":{"id":65088991,"uuid":"540129543","full_name":"fcakyon/content-moderation-deep-learning","owner":"fcakyon","description":"Deep learning based content moderation from text, audio, video \u0026 image input modalities.","archived":false,"fork":false,"pushed_at":"2024-12-16T13:32:20.000Z","size":193,"stargazers_count":330,"open_issues_count":0,"forks_count":18,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-02-04T19:13:03.123Z","etag":null,"topics":["content-moderation","content-ratings","genre-classification","movie-content-filter","movie-trailer","multimodal-deep-learning","nsfw-recognition","nudity-detection","profanity-detection","violence-detection"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2212.04533","language":null,"has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fcakyon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"fcakyon"}},"created_at":"2022-09-22T19:02:29.000Z","updated_at":"2025-01-25T07:48:40.000Z","dependencies_parsed_at":"2024-10-23T00:38:24.375Z","dependency_job_id":null,"html_url":"https://github.com/fcakyon/content-moderation-deep-learning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fcakyon%2Fcontent-moderation-deep-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fcakyon%2Fcontent-moderation-deep-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fcakyon%2Fcontent-moderation-deep-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fcakyon%2Fcontent-moderation-deep-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fcakyon","download_url":"https://codeload.github.com/fcakyon/content-moderation-deep-learning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246223323,"owners_count":20743167,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["content-moderation","content-ratings","genre-classification","movie-content-filter","movie-trailer","multimodal-deep-learning","nsfw-recognition","nudity-detection","profanity-detection","violence-detection"],"created_at":"2024-10-03T10:24:49.893Z","updated_at":"2025-10-18T06:34:47.030Z","avatar_url":"https://github.com/fcakyon.png","language":null,"funding_links":["https://github.com/sponsors/fcakyon"],"categories":[],"sub_categories":[],"readme":"# deep-learning-content-moderation\n\nVarious sources for deep learning based content moderation, sensitive content detection, scene genre classification, nudity detection, violence detection, substance detection from text, audio, video \u0026amp; image input modalities.\n\n## citation\n\nIf you find this source useful, please consider citing it in your work as:\n\n```bib\n@INPROCEEDINGS{10193621,\n  author={Akyon, Fatih Cagatay and Temizel, Alptekin},\n  booktitle={2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)}, \n  title={State-of-the-Art in Nudity Classification: A Comparative Analysis}, \n  year={2023},\n  pages={1-5},\n  keywords={Analytical models;Convolution;Conferences;Transfer learning;Benchmark testing;Transformers;Safety;content moderation;nudity detection;safety;transformers},\n  doi={10.1109/ICASSPW59220.2023.10193621}}\n```\n```bib\n@article{akyon2022contentmoderation,\n  title={Deep Architectures for Content Moderation and Movie Content Rating},\n  author={Akyon, Fatih Cagatay and Temizel, Alptekin},\n  journal={arXiv},\n  doi={https://doi.org/10.48550/arXiv.2212.04533},\n  year={2022}\n}\n```\n\n## table of contents\n\n* [datasets](https://github.com/fcakyon/content-moderation-deep-learning#datasets)\n    * [movie and content moderation datasets](https://github.com/fcakyon/content-moderation-deep-learning#movie-and-content-moderation-datasets)\n* [techniques](https://github.com/fcakyon/content-moderation-deep-learning#techniques)\n    * [sensitive content detection](https://github.com/fcakyon/content-moderation-deep-learning#sensitive-content-detection)\n        * [movie content rating](https://github.com/fcakyon/content-moderation-deep-learning#movie-content-rating)\n        * [content moderation](https://github.com/fcakyon/content-moderation-deep-learning#content-moderation)\n    * [movie/scene genre classification](https://github.com/fcakyon/content-moderation-deep-learning#movie/scene-genre-classification)\n    * [multimodal architectures](https://github.com/fcakyon/content-moderation-deep-learning#multimodal-architectures)\n        * [synchronous multimodal architectures](https://github.com/fcakyon/content-moderation-deep-learning#synchronous-multimodal-architectures)\n        * [asynchronous multimodal architectures](https://github.com/fcakyon/content-moderation-deep-learning#asynchronous-multimodal-architectures)\n    * [action recognition](https://github.com/fcakyon/content-moderation-deep-learning#action-recognition)\n        * [with transformers](https://github.com/fcakyon/content-moderation-deep-learning#with-transformers)\n        * [with 3D CNNs](https://github.com/fcakyon/content-moderation-deep-learning#with-3d-cnns)\n    * [contrastive representation learning](https://github.com/fcakyon/content-moderation-deep-learning#contrastive-representation-learning)\n    * [review papers](https://github.com/fcakyon/content-moderation-deep-learning#review-papers)\n* [tools](https://github.com/fcakyon/content-moderation-deep-learning#tools)\n\n## datasets\n\n### movie and content moderation datasets\n\n| name | paper | year | url | input modality | task | labels |\n|--- |--- |--- |--- |--- |--- |--- |\n| LSPD | [pdf](https://inass.org/wp-content/uploads/2021/09/2022022819-4.pdf) | 2022 | [page](https://sites.google.com/uit.edu.vn/LSPD) | image, video | image/video classification, instance segmentation | porn, normal, sexy, hentai, drawings, female/male genital, female breast, anus |\n| MM-Trailer | [pdf](https://aclanthology.org/2021.ranlp-1.146.pdf) | 2021 | [page](https://ritual.uh.edu/RANLP2021/) | video | video classification | age rating |\n| Movienet | [scholar](https://scholar.google.com/scholar?cluster=7273702520677604457\u0026hl=en\u0026as_sdt=0,5) | 2021 | [page](https://movienet.github.io/) | image, video, text | object detection, video classification | scene level actions and places, character bboxes |\n| Movie script severity dataset | [pdf](https://arxiv.org/pdf/2109.09276.pdf) | 2021 | [github](https://github.com/RiTUAL-UH/Predicting-Severity-in-Movie-Scripts) | text | text classification | frightening, mild, moderate, severe |\n| LVU | [pdf](https://chaoyuan.org/papers/lvu.pdf) | 2021 | [page](https://chaoyuan.org/lvu/) | video | video classification | relationship, place, like ration, view count, genre, writer, year per movie scene |\n| Violence detection dataset | [scholar](https://scholar.google.com/scholar?cluster=839991967738597651\u0026hl=en\u0026as_sdt=0,5) | 2020 | [github](https://github.com/airtlab/A-Dataset-for-Automatic-Violence-Detection-in-Videos) | video | video classification | violent, not-violent |\n| Movie script dataset | [pdf](https://sail.usc.edu/~mica/pubs/martinez2019violence.pdf) | 2019 | [github](https://github.com/usc-sail/mica-violence-ratings) | text | text classification | violent or not |\n| Nudenet | [github](https://github.com/notAI-tech/NudeNet) | 2019 | [archive.org](https://archive.org/details/NudeNet_classifier_dataset_v1) | image | image classification | nude or not |\n| Adult content dataset | [pdf](https://arxiv.org/abs/1612.09506) | 2017 | [contact](https://drive.google.com/file/d/1MrzzFxQ9t56i9d3gMAI3SaHXfVd1UUDt/view) | image | image classification | nude or not |\n| Substance use dataset | [pdf](https://web.cs.ucdavis.edu/~hpirsiav/papers/substance_ictai17.pdf) | 2017 | [first author](https://www.linkedin.com/in/roy-arpita/) | image | image classification | drug related or not |\n| NDPI2k dataset | [pdf](https://www.researchgate.net/profile/Sandra-Avila-5/publication/308398120_Pornography_Classification_The_Hidden_Clues_in_Video_Space-Time/links/5a205361a6fdcccd30e00d4a/Pornography-Classification-The-Hidden-Clues-in-Video-Space-Time.pdf) | 2016 | [contact](https://www.ic.unicamp.br/~rocha/mail/index.html) | video | video classification | porn or not |\n| Violent Scenes Dataset | [springer](https://link.springer.com/article/10.1007/s11042-014-1984-4) | 2014 | [page](https://www.interdigital.com/data_sets/violent-scenes-dataset) | video | video classification | blood, fire, gun, gore, fight |\n| VSD2014 | [pdf](http://www.cp.jku.at/people/schedl/Research/Publications/pdf/schedl_cbmi_2015.pdf) | 2014 | [download](http://www.cp.jku.at/datasets/VSD2014/VSD2014.zip) | video | video classification | blood, fire, gun, gore, fight |\n| AIIA-PID4 | [pdf](https://www.igi-global.com/gateway/article/full-text-pdf/79140\u0026riu=true) | 2013 | - | image | image classification | bikini, porn, skin, non-skin |\n| NDPI800 dataset | [scholar](https://scholar.google.com/scholar?cluster=7836593192753784698\u0026hl=en\u0026as_sdt=0,5) | 2013 | [page](https://sites.google.com/site/pornographydatabase/) | video | video classification | porn or not |\n| HMDB-51 | [scholar](https://scholar.google.com/scholar?cluster=5533751878557083824\u0026hl=en\u0026as_sdt=0,5) | 2011 | [page](https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/#overview) | video |video classification | smoke, drink |\n\n## techniques\n\n### sensitive content detection\n\n#### movie content rating\n\n| name | paper | year | model | features | datasets | tasks | context |\n|--- |--- |--- |--- |--- |--- |--- |--- |\n| Movies2Scenes: Learning Scene Representations Using Movie Similarities | [scholar](https://scholar.google.com/scholar?cluster=10458780403101083615\u0026hl=en\u0026as_sdt=0,5) | 2022 | ViT-like video encoder + MLP | ViT-like video encoder embedings | Private, Movienet, LVU | movie scene representation learning, video classifcation (sex, violence, drug-use) | movie scene content rating |\n| Detection and Classification of Sensitive Audio-Visual Content for Automated Film Censorship and Rating | [pdf](https://www.researchgate.net/profile/Mahmudul-Haque-4/publication/359704298_Detection_and_Classification_of_Sensitive_Audio-Visual_Content_for_Automated_Film_Censorship_and_Rating/links/6249a3aa8068956f3c668be5/Detection-and-Classification-of-Sensitive-Audio-Visual-Content-for-Automated-Film-Censorship-and-Rating.pdf) | 2022 | CNN + GRU + MLP | CNN embeddings from video frames | Violence detection dataset | violent/non-violent classification from videos | movie scene content rating |\n| Automatic parental guide ratings for short movies | [page](http://eprints.utar.edu.my/4250/) | 2021 | separate model for each task: concat + LSTM, object detector, one-class CNN embeddings | video frame pixel values, image embeddings, text | Nudenet, private dataset | profanity, violence, nudity, drug classification | movie content rating |\n| From None to Severe: Predicting Severity in Movie Scripts | [scholar](https://scholar.google.com/scholar?cluster=13391231771783888164\u0026hl=en\u0026as_sdt=0,5) | 2021 | multi-task pairwise ranking-classification network | GloVe, Bert and TextCNN text embeddings | Movie script severity dataset | rating classifcation (frightening, mild, moderate, severe) | movie content rating |\n| A Case Study of Deep Learning-Based Multi-Modal Methods for Labeling the Presence of Questionable Content in Movie Trailers | [scholar](https://scholar.google.com/scholar?cluster=15170598737548453750\u0026hl=en\u0026as_sdt=0,5) | 2021 | multi-modal + multi output concat+MLP | CNN+LSTM video features, Bert and DeepMoji text embeddings, MFCC audio features | MM-Trailer | rating classifcation (red, yellow, green) | movie trailer content rating |\n| Automatic Parental Guide Scene Classification Menggunakan Metode Deep Convolutional Neural Network Dan Lstm | [scholar](https://scholar.google.com/scholar?cluster=2229349718067727385\u0026hl=en\u0026as_sdt=0,5) | 2020 | 3 CNN model for 3 modality, multi-label dataset | CNN video and audio embeddings, LSTM text (subitle) embeddings | private dataset | gore, nudity, drug, profanity classification from video and subtitle | movie scene content rating |\n| Multimodal data fusion for sensitive scene localization | [scholar](https://scholar.google.com/scholar?cluster=16351890771413386481\u0026hl=en\u0026as_sdt=0,5) | 2019 | meta-learning with Naive Bayes, SVM | MFCC and  prosodic features from audio, HOG and TRoF features from images | Pornography-2k dataset, VSD2014 | violent and pornographic scene localization from video | movie scene content rating |\n| A Deep Learning approach for the Motion Picture Content Rating | [scholar](https://scholar.google.com/scholar?cluster=14110689052300247206\u0026hl=en\u0026as_sdt=0,5) | 2019 | MLP + rule-based decision | InceptionV3 image embeddings | Violent Scenes Dataset, private dataset | violence (shooting, blood, fire, weapon) classification from video | movie scene content rating |\n| Hybrid System for MPAA Ratings of Movie Clips Using Support Vector Machine | [springer](https://link.springer.com/chapter/10.1007/978-981-13-1595-4_45) | 2019 | SVM | DCT features from image | private dataset | movie content rating classification from images | movie content rating |\n| Inappropriate scene detection in a video stream | [page](https://dspace.bracu.ac.bd/xmlui/handle/10361/9469) | 2017 | SVM classifier + Lenet image classifier + rules-based decision | HoG and CNN features for image | private dataset | image classification: no/mild/high violence, safe/unsafe/pornoghraphy | movie frame content rating |\n\n#### content moderation\n\n| name | paper | year | model | features | datasets | tasks | context |\n|--- |--- |--- |--- |--- |--- |--- |--- |\n| State-of-the-Art in Nudity Classification: A Comparative Analysis | [ieee](https://ieeexplore.ieee.org/document/10193621) | 2023 | CNN, Transformers | EfficientNet, ViT, ConvNeXT image embeddings | LSPD, Nudenet, NDPI2k | nudity classification from images | general content moderation |\n| Reliable Decision from Multiple Subtasks through Threshold Optimization: Content Moderation in the Wild | [scholar](https://scholar.google.com/scholar?cluster=6441617450690688428\u0026hl=en\u0026as_sdt=0,5) | 2022 | novel threshold optimization tech. (TruSThresh) | prediction scores | UnSmile (Korean hatespeech dataset) | optimum threshold prediction | social media content moderation |\n| On-Device Content Moderation | [scholar](https://scholar.google.com/scholar?cluster=5550217109642251291\u0026hl=en\u0026as_sdt=0,5) | 2021 | mobilenet v3 + SSD object detector | mobilenet v3 image embeddings | private dataset | object detection + nudity classification from images | on-device content moderation |\n| Gore Classification and Censoring in Images | [scholar](https://scholar.google.com/scholar?cluster=1477214110429045154\u0026hl=en\u0026as_sdt=0,5) | 2021 | ensemble of CNN + MLP | mobilenet v2, densenent, vgg16 image embeddings | private dataset | gore classification from images | general content moderation |\n| Automated Censoring of Cigarettes in Videos Using Deep Learning Techniques | [scholar](https://scholar.google.com/scholar?cluster=14277066392319361311\u0026hl=en\u0026as_sdt=0,5) | 2020 | CNN + MLP | inception v3 image embeddings | private dataset | cigarette classification from video | general content moderation |\n| A Multimodal CNN-based Tool to Censure Inappropriate Video Scenes | [scholar](https://scholar.google.com/scholar?cluster=11897464348691991288\u0026hl=en\u0026as_sdt=0,5) | 2019 | CNN + SVM | InceptionV3 image embeddings, AudioVGG audio embeddings | private dataset | inappropriate (nudity+gore) classification from video | general video content moderation |\n| A baseline for NSFW video detection in e-learning environments | [scholar](https://scholar.google.com/scholar?cluster=12765316397376090724\u0026hl=en\u0026as_sdt=0,5) | 2019 | concat + SVM, MLP | InceptionV3 image embeddings, AudioVGG audio embeddings | YouTube8M, NDPI, Cholec80 | nudity classification from video | e-learning content moderation |\n| Bringing the kid back into youtube kids: Detecting inappropriate content on video streaming platforms | [scholar](https://scholar.google.com/scholar?cluster=17864610709056312820\u0026hl=en\u0026as_sdt=0,5) | 2019 | CNN + LSTM (late fusion) | CNN based encoder for image, video and audio spectrograms | private dataset | video classification: orignal, fake explicit, fake violent | social media content moderation |\n\n### movie/scene genre classification\n\n| name | paper | year | model | features | datasets | tasks |\n|--- |--- |--- |--- |--- |--- |--- |\n| Effectively leveraging Multi-modal Features for Movie Genre Classification | [scholar](https://scholar.google.com/scholar?cluster=7914951466429935825\u0026hl=en\u0026as_sdt=0,5) | 2022 | embeddings + fusion + MLP | CLIP image embeddings, PANNs audio embeddings, CLIP text embeddings | MovieNet | movie genre classification |\n| OS-MSL: One Stage Multimodal Sequential Link Framework for Scene Segmentation and Classification | [scholar](https://scholar.google.com/scholar?cluster=12395964150974750638\u0026hl=en\u0026as_sdt=0,5) | 2022 | embeddings + novel transformer | ResNet-18 image embeddings, ResNet-VLAD audio embeddings | TI-News | news scene segmentation/classification (studio, outdoor, interview) |\n| Detection of Animated Scenes Among Movie Trailers | [scholar](https://scholar.google.com/scholar?cluster=6655397332509975816\u0026hl=en\u0026as_sdt=0,5) | 2022 | CNN + GRU | EfficientNet image embeddings | Private dataset | genre classification from movie trailer scenes |\n| A multi-label movie genre classification scheme based on the movie's subtitles | [springer](https://link.springer.com/article/10.1007/s11042-022-12961-6) | 2022 | KNN | text frequency vectors | Private dataset | genre classification from movie subtitle text |\n| A multimodal approach for multi-label movie genre classification | [scholar](https://scholar.google.com/scholar?cluster=9310008085751443571\u0026hl=en\u0026as_sdt=0,5) | 2020 | CNN + LSTM | MFCCs/SSD/LBP from audio, LBP/3DCNN from video frames, Inception-v3 from poster, TFIDF from text | Private dataset | genre classification from movie trailers |\n| Genre classification of movie trailers using 3d convolutional neural networks | [ieee](https://ieeexplore.ieee.org/abstract/document/9121148) | 2020 | 3D CNN | images | Private dataset | genre classification from movie trailer scenes |\n| A unified framework of deep networks for genre classification using movie trailer | [scholar](https://scholar.google.com/scholar?cluster=14452117618428848943\u0026hl=en\u0026as_sdt=0,5) | 2020 | CNN + LSTM | Inception V4 image embeddings | EmoGDB | genre classification from movie trailer scenes |\n| Towards story-based classification of movie scenes | [scholar](https://scholar.google.com/scholar?cluster=3605673709859574181\u0026hl=en\u0026as_sdt=0,5) | 2020 | logistic regression | manually extracted categorical features | Flintstones Scene Dataset | scene classification (Obstacle, Midpoint, Climax of Act 1) |\n\n### multimodal architectures\n\n#### synchronous multimodal architectures\n\n| name | paper | year | model | features | datasets | tasks | modalities |\n|--- |--- |--- |--- |--- |--- |--- |--- |\n| M\u0026M Mix: A Multimodal Multiview Transformer Ensemble | [scholar](https://scholar.google.com/scholar?cluster=14299004100101928418\u0026hl=en\u0026as_sdt=0,5) | 2022 | transformer with 2 cls heads | ViT image embeddings from audio spect., frame image, optical flow | Epic-Kitchens | video/action classification | image + audio + optical flow |\n| MultiMAE: Multi-modal Multi-task Masked Autoencoders | [scholar](https://scholar.google.com/scholar?cluster=7235983779434806126\u0026hl=en\u0026as_sdt=0,5) | 2022 | transformer with 3 decoder + cls heads | ViT-like image enc. patch embeddings (optional modalities) | ImageNet: Pseudo labeled multi-task training dataset (depth, segm) | image cs., semantic segm., depth est. | image + depth map |\n| Data2vec: A general framework for self-supervised learning in speech, vision and language | [scholar](https://scholar.google.com/scholar?cluster=12686412422242429370\u0026hl=en\u0026as_sdt=0,5) | 2022 | single encoder | transformer based audio, text, image encoder embeddings | ImageNet, Librispeech | masked pretraining | image + audio + text |\n| VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text | [scholar](https://scholar.google.com/scholar?cluster=7327595990658945420\u0026hl=en\u0026as_sdt=0,5) | 2022 | 1 encoder per modality | transformer based audio, text, image encoder embeddings | AudioSet, HowTo100M | pretraining + video/audio classification | image + audio + text |\n| Expanding Language-Image Pretrained Models for General Video Recognition | [scholar](https://scholar.google.com/scholar?cluster=1144066736404687657\u0026hl=en\u0026as_sdt=0,5) | 2022 | 1 encoder per modality | transformer based video, text encoder embeddings | HMDB-51, UCF-101 | contrastive pretraining | video + text |\n| Audio-Visual Instance Discrimination with Cross-Modal Agreement | [scholar](https://scholar.google.com/scholar?cluster=885326186401082715\u0026hl=en\u0026as_sdt=0,5) | 2021 | 1 encoder per modality | CNN based audio, video encoder embeddings | HMDB-51, UCF-101 | video/audio classification | video + audio |\n| Robust Audio-Visual Instance Discrimination | [scholar](https://scholar.google.com/scholar?cluster=9200438422642254466\u0026hl=en\u0026as_sdt=0,5) | 2021 | 1 encoder per modality | CNN based audio, video encoder embeddings | HMDB-51, UCF-101 | video/audio classification | video + audio |\n| Learning transferable visual models from natural language supervision | [scholar](https://scholar.google.com/scholar?cluster=15031020161691567042\u0026hl=en\u0026as_sdt=0,5) | 2021 | 1 encoder per modality | transformer based image, text encoder embeddings | JFT-300M | contrastive pretraining | image + text |\n| Self-supervised multimodal versatile networks | [scholar](https://scholar.google.com/scholar?cluster=16748353423289036473\u0026hl=en\u0026as_sdt=0,5) | 2020 | multiple encoders | CNN based image/audio embeddings, word2vec text embeddings | UCF101, Kinetics, AudioSet | contrastive pretraining + classification | image + audio + text |\n| Uniter: Universal image-text representation learning | [scholar](https://scholar.google.com/scholar?cluster=3224460637705754187\u0026hl=en\u0026as_sdt=0,5) | 2020 | multimodal encoder | combined embeddings | COCO, Visual Genome, Conceptual Captions | qa/image-text retrieval | image + text |\n| 12-in-1: Multi-task vision and language representation learning | [scholar](https://scholar.google.com/scholar?cluster=17276757515931533114\u0026hl=en\u0026as_sdt=0,5) | 2020 | multimodal encoder | combined embeddings | COCO, Flickr30k | qa/image-text retrieval | image + text |\n| Two-stream convolutional networks for action recognition in videos | [scholar](https://scholar.google.com/scholar?cluster=582514008712420788\u0026hl=en\u0026as_sdt=0,5) | 2014 | 1 encoder per modality | CNN based audio, text encoder embeddings | HMDB-51, UCF-101 | video/audio classification | video + optical flow |\n\n#### asynchronous multimodal architectures\n\n| name | paper | year | model | features | datasets | tasks | modalities |\n|--- |--- |--- |--- |--- |--- |--- |--- |\n| OmniMAE: Single Model Masked Pretraining on Images and Videos | [scholar](https://scholar.google.com/scholar?cluster=4636381983240187321\u0026hl=en\u0026as_sdt=0,5) | 2022 | transformer with 1 cls. head | ViT-like image/video enc. patch embeddings | ImageNet, SSv2 | video/action classification | image + video |\n| OMNIVORE: A Single Model for Many Visual Modalities | [scholar](https://scholar.google.com/scholar?cluster=7775892631592466813\u0026hl=en\u0026as_sdt=0,5\u0026as_ylo=2021) | 2022 | transformer with 3 cls. heads | ViT-like image/video enc. patch embeddings | ImageNet, Kinetics, SSv2, SUN RGB-D | image cls., action recog., depth est. | image + video + depth map |\n| Polyvit: Co-training vision transformers on images, videos and audio | [scholar](https://scholar.google.com/scholar?cluster=2433441885724580400\u0026hl=en\u0026as_sdt=0,5) | 2021 | transformer with 9 cls. heads | ViT-like image/video/audio enc. embeddings | ImageNet, CIFAR, Kinetics, Moments in Time, AudioSet, VGGSound | image cls., video cls., audio cls. | image + video + audio |\n\n### action recognition\n\n#### with transformers\n\n| name | paper | year | model | features | datasets | tasks |\n|--- |--- |--- |--- |--- |--- |--- |\n| Frozen CLIP Models are Efficient Video Learners | [scholar](https://scholar.google.com/scholar?cluster=16057670792750577500\u0026hl=en\u0026as_sdt=0,5) | 2022 | transformer with 1 cls head | CLIP image embeddings | ImageNet, Kinetics, SSv2 | action recognition |\n| Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training | [scholar](https://scholar.google.com/scholar?cluster=8140812159859442226\u0026hl=en\u0026as_sdt=0,5) | 2022 | transformer with 1 cls head | ViT-like video enc. patch embeddings | Kinetics, SSv2 | action recognition |\n| Bevt: Bert pretraining of video transformers | [scholar](https://scholar.google.com/scholar?cluster=9527303198700083047\u0026hl=en\u0026as_sdt=0,5) | 2022 | encoder-decoder transformer | VideoSwin image/video enc. embeddings | Kinetics, SSv2 | action recognition |\n| Video swin transformer | [scholar](https://scholar.google.com/scholar?cluster=5833041667751260373\u0026hl=en\u0026as_sdt=0,5) | 2022 | Swin trans. with cls.head | Swin video enc. embeddings | Kinetics, SSv2 | action recognition |\n| Is space-time attention all you need for video understanding? | [scholar](https://scholar.google.com/scholar?cluster=6828425192739736056\u0026hl=en\u0026as_sdt=0,5) | 2021 | transformer with cls. head | ViT-like video enc. patch embeddings | Kinetics, SSv2 | action recognition |\n\n#### with 3D CNNs\n\n| name | paper | year | model | features | datasets | tasks |\n|--- |--- |--- |--- |--- |--- |--- |\n| X3d: Expanding architectures for efficient video recognition | [scholar](https://scholar.google.com/scholar?cluster=5426206565542427464\u0026hl=en\u0026as_sdt=0,5) | 2020 | CNN with cls. head | 3D CNN based video enc. embeddings | Kinetics, SSv2 | action recognition |\n| Slowfast networks for video recognition | [scholar](https://scholar.google.com/scholar?cluster=1892562522989461632\u0026hl=en\u0026as_sdt=0,5) | 2019 | CNN with cls. head | 3D CNN based video enc. embeddings | Kinetics, SSv2 | action recognition |\n| A closer look at spatiotemporal convolutions for action recognition (R2+1D) | [scholar](https://scholar.google.com/scholar?cluster=9524036545693727210\u0026hl=en\u0026as_sdt=0,5) | 2018 | CNN with cls. head | 3D CNN based video enc. embeddings | Kinetics, HMDB-51, UCF-101 | action recognition |\n| Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (I3D) | [scholar](https://scholar.google.com/scholar?cluster=9581219496538221166\u0026hl=en\u0026as_sdt=0,5) | 2017 | CNN with cls. head | 3D CNN based video enc. embeddings | Kinetics, HMDB-51, UCF-101 | action recognition |\n\n### contrastive representation learning\n\n| name | paper | date |\n|--- |--- |--- |\n| Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text | [scholar](https://scholar.google.com.tr/scholar?cluster=7327595990658945420) | 2021 |\n| Supervised contrastive learning | [scholar](https://scholar.google.com.tr/scholar?cluster=2159203708190499110) | 2020 |\n\n### review papers\n\n| name | paper | date |\n|--- |--- |--- |\n| Machine Learning Models for Content Classification in Film Censorship and Rating | [pdf](https://www.researchgate.net/profile/Mahmudul-Haque-4/publication/360812725_Machine_Learning_Models_for_Content_Classification_in_Film_Censorship_and_Rating/links/629219a58d19206823e2cf70/Machine-Learning-Models-for-Content-Classification-in-Film-Censorship-and-Rating.pdf) | 2022 |\n| A survey of artificial intelligence strategies for automatic detection of sexually explicit videos | [scholar](https://scholar.google.com/scholar?cluster=3622219384521520101\u0026hl=en\u0026as_sdt=0,5) | 2022 |\n| A survey on video content rating: taxonomy, challenges and open issues | [pdf](https://link.springer.com/content/pdf/10.1007/s11042-021-10838-8.pdf) | 2021 |\n| Multimodal Learning with Transformers: A Survey  | [scholar](https://scholar.google.com/scholar?cluster=6631830511436466415\u0026hl=en\u0026as_sdt=0,5) | 2022 |\n| A Survey Paper on Movie Trailer Genre Detection | [scholar](https://scholar.google.com/scholar?cluster=18275675471389373237\u0026hl=en\u0026as_sdt=0,5) | 2020 |\n\n## tools\n\n| name | url | description |\n|--- |--- |--- |\n| safetext | [github](https://github.com/viddexa/safetext) | multilingual swear word detection and filtering |\n| PySceneDetect | [github](https://github.com/Breakthrough/PySceneDetect) | Python and OpenCV-based scene cut/transition detection program \u0026 library |\n| LAION safety toolkit | [github](https://github.com/LAION-AI/LAION-SAFETY) | NSFW detector trained on LAION dataset |\n| pysrt | [github](https://github.com/byroot/pysrt) | Python parser for SubRip (srt) files |\n| ffsubsync | [github](https://github.com/smacke/ffsubsync) | Automagically synchronize subtitles with video. |\n| MoviePy | [github](https://github.com/Zulko/moviepy) | Video editing with Python |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffcakyon%2Fcontent-moderation-deep-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffcakyon%2Fcontent-moderation-deep-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffcakyon%2Fcontent-moderation-deep-learning/lists"}