{"id":28357267,"url":"https://github.com/mazleon/hateful_meme_challenge","last_synced_at":"2026-05-03T15:32:47.645Z","repository":{"id":129405199,"uuid":"355794051","full_name":"mazleon/Hateful_Meme_Challenge","owner":"mazleon","description":"Hateful Memes dataset contains real hate speech. The Real Hateful Memes dataset consists of more than 10,000 newly created examples by Facebook AI. ","archived":false,"fork":false,"pushed_at":"2021-05-17T08:07:47.000Z","size":1445,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-19T20:43:46.647Z","etag":null,"topics":["computer-vision","hateful-memes-challenge","multimodal-deep-learning","nlp","pytorch","pytorch-lightning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mazleon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-04-08T06:49:50.000Z","updated_at":"2024-08-05T04:18:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"ee9e0b99-aa24-474e-b15f-2229c2ec3455","html_url":"https://github.com/mazleon/Hateful_Meme_Challenge","commit_stats":null,"previous_names":["mazleon/hateful_meme_challenge"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mazleon/Hateful_Meme_Challenge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mazleon%2FHateful_Meme_Challenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mazleon%2FHateful_Meme_Challenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mazleon%2FHateful_Meme_Challenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mazleon%2FHateful_Meme_Challenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mazleon","download_url":"https://codeload.github.com/mazleon/Hateful_Meme_Challenge/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mazleon%2FHateful_Meme_Challenge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32575076,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T06:36:36.687Z","status":"ssl_error","status_checked_at":"2026-05-03T06:36:09.306Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","hateful-memes-challenge","multimodal-deep-learning","nlp","pytorch","pytorch-lightning"],"created_at":"2025-05-28T07:11:14.974Z","updated_at":"2026-05-03T15:32:47.624Z","avatar_url":"https://github.com/mazleon.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hateful_Meme_Challenge\nHateful Memes dataset contains real hate speech. The Real Hateful Memes dataset consists of more than 10,000 newly created examples by Facebook AI. \n\nHateful Memes dataset contains real hate speech. The Real Hateful Memes dataset consists of more\nthan 10,000 newly created examples by Facebook AI.\nSolutions: As the memes contain images with text, it becomes related to computer vision and natural\nlanguage processing problems.The target is to classify the memes in either hateful or not hateful class, so\nI have to analyze the images, texts which is why I chose multimodal architecture for hateful memes\nclassification. For multimodal architecture I chose the ResNet-152(vision), fastText(language) model.\nComparing their results. I believe that multimodality holds the key to problems with as much variety as\nnatural language and understanding computer vision evaluation, and embodied AI.\nBrief description of sample code:\nFirst, for deep analysis in the dataset, I have to know some statistics of different objects that are present\nin thoses images. To find out, I applied an object detection deep learning model called “Detectron 2”. In\norder to that model, I installed detectron2 libraries and imported others libraries. After importing libraries, I\ncreated a detectron2 config and a detectron2 DefaultPredictor to run inference on images. Then I loaded the\nweights/pretrained model in “DefaultPredictor'' that comes from “from detectron2 import model_zoo”.\nThen I went through my dataset and applied DefaultPredictor for prediction classes, and collected those\nclasses found in images into an object list.. Then I filtered the total object found in a single image and the\ntotal unique number objects found in a single image and stored them in a pandas dataframe. Later I\nvisualized them into different formats by using matplotlib libraries and also tried to find the different\ncategories' answers from the dataframe, for example, which images contained above 5 unique objects\nand how many total images were there that contained most objects in a single image. You can find those\ncode in this link: [Data Analysis of Hateful Memes]\n\n## Dataset PreProcessing: \nThe dataset contains 3 files and 1 directory. The directory is called\nimg and it contains all the meme images, where I have access: train, dev and test. The image\nfiles were named \u003cid\u003e.png, where id is a unique 5 digit number. Each remaining jsonl file\ncontains id, img, text and label. I loaded them by using “pd.read_json(filepath, lines=True)”. I\nalso made a HatefulMemesDataset class which extended the torch.utils.data.Dataset class. In\nthis class, it takes data_path, img_dir, image_transformation, text_transformation, balance,dev_limit, ranodm_state as an input argument. With this class I could load the dataset into a\npytorch dataloader like trainloader, valloader. [source code]\n\n## Vision Module : \nIn the vision module, I used the ResNet-152 model. The ResNet-152 was\nimported from torchvision libraries with pretrained weights. I extracted the features from the\nimages rather than classifying to overwrite the last layer with an identity transformation. Then I\nreduced the dimension using a linear layer and the resnet output was 2048.\n\n```vision_module=torchvision.models.resnet152(pretrained=True)\nvision_module.fc = torch.nn.Linear(in_features=2048, out_features=self.vision_feature_dim)\n```\n\n\n## Language Module:\nIt’s a task to extract features from text. And for extracting these features I used fastText libraries’\nmodel.FastText is an open-source, free, lightweight library that allows users to learn text representations\nand text classifiers.\n```\nlanguage_transform = fasttext.train_unsupervised(\nstr(ft_path),\nmodel= self.hparams.get(\"fasttext_model\", \"cbow\"),\ndim=self.embedding_dim\n)\n```\nI passed the outputs of our text to transform through an additional trainable layer rather than fine-tuning\nthe transform. For that I added another linear layer.\n```\nlanguage_module = torch.nn.Linear(\nin_features=self.embedding_dim,\nout_features=self.language_feature_dim\n)\n```\n## The Late Fusion:\nThere are many fusions available in the research work. Among which I chose the later fusion\nmethod. That means the feature extracted from the vision and language module will be fusioned\nand passed through a linear layer.\n```\nself.fusion = torch.nn.Linear(\nin_features=(language_feature_dim + vision_feature_dim),\nout_features=fusion_output_size\n)\n```\n## The model forward method:\nThis method receives a text and an image. Firstly it passes the text through our language\nmodule and the output comes from a language module which will be passed through an\nactivation function called relu. Like this, the image will be passed through our vision module and\nthe output from a vision module will also be passed through relu activation function. Then I\ncombined the text feature and image features and applied the late fusion method. After applying\nlate fusion, I passed the output through a fully connected linear layer. The input feature size of\nthe linear layer is the output size of fusion and the output feature size is equal to the number of\nclasses. For finding the class’ probability distribution I used a softmax activation function. Thenthe prediction was passed through a loss function. Finally, this method returns the prediction\nand loss value.\n```\ndef forward(self, text, image, label=None):\n  text_features = torch.nn.functional.relu(\n  self.language_module(text)\n  )\n  image_features = torch.nn.functional.relu(\n  self.vision_module(image)\n  )\n  combined = torch.cat(\n  [text_features, image_features], dim=1\n  )\n  fused = self.dropout(\n  torch.nn.functional.relu(\n  self.fusion(combined)\n  )\n )\n  logits = self.fc(fused)\n  pred = torch.nn.functional.softmax(logits)\n  loss = (\n  self.loss_fn(pred, label)\n  if label is not None else label\n  )\n  return (pred, loss)\n\n```\n\nThe rest of the code is done using PyTorch Lightning libraries which are basically based on the PyTorch\nframework. And the other parameter that I used for solving this problem is given below.\n```\nhparams = {\n  \"train_path\": train_path,\n  \"dev_path\": dev_path,\n  \"img_dir\": data_dir,\n  # Optional params\n  \"embedding_dim\": 150,\n  \"language_feature_dim\": 300,\n  \"vision_feature_dim\": 300,\n  \"fusion_output_size\": 256,\n  \"output_path\": \"model-outputs\",\n  \"dev_limit\": None,\n  \"lr\": 0.00005,\n  \"max_epochs\": 10,\n  \"n_gpu\": 1,\n  \"batch_size\": 4,\n  # allows us to \"simulate\" having larger batches\n  \"accumulate_grad_batches\": 16,\n  \"early_stop_patience\": 3,\n}\n```\n\nYou can find the original source code of the project in the following link :\nhttps://github.com/princexoleo/Hateful_Meme_Challenge\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmazleon%2Fhateful_meme_challenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmazleon%2Fhateful_meme_challenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmazleon%2Fhateful_meme_challenge/lists"}