{"id":28509514,"url":"https://github.com/finnff/vlm-bias-decomposition","last_synced_at":"2025-07-02T23:31:13.370Z","repository":{"id":296096182,"uuid":"992255426","full_name":"finnff/vlm-bias-decomposition","owner":"finnff","description":"VLM bias mitigation via vector decomposition in CLIP","archived":false,"fork":false,"pushed_at":"2025-06-23T22:50:30.000Z","size":4075,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-23T23:31:34.348Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/finnff.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-28T21:34:09.000Z","updated_at":"2025-06-23T22:50:34.000Z","dependencies_parsed_at":"2025-06-15T12:34:05.495Z","dependency_job_id":null,"html_url":"https://github.com/finnff/vlm-bias-decomposition","commit_stats":null,"previous_names":["finnff/vlm-bias-decomposition"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/finnff/vlm-bias-decomposition","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finnff%2Fvlm-bias-decomposition","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finnff%2Fvlm-bias-decomposition/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finnff%2Fvlm-bias-decomposition/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finnff%2Fvlm-bias-decomposition/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/finnff","download_url":"https://codeload.github.com/finnff/vlm-bias-decomposition/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finnff%2Fvlm-bias-decomposition/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263232468,"owners_count":23434671,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-08T22:08:55.910Z","updated_at":"2025-07-02T23:31:13.246Z","avatar_url":"https://github.com/finnff.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# vlm-bias-decomposition\n\nVLM bias mitigation via vector decomposition in CLIP\n\n## Requirements\n\n```bash\nconda init #(if you haven't done this before)\n\nconda create -n clipenv python=3.8 --yes\nconda activate clipenv\n\nconda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=11.0\npip install ftfy regex tqdm pandas matplotlib requests gdown scipy scikit-learn seaborn adjustText\npip install git+https://github.com/openai/CLIP.git\n```\n\n\n## Usage\n\n1. First run `python celeba_downloader_clip_test.py` to ensure you have the CelebA dataset downloaded and CLIP model loaded.\n2. Then run `python distinct_prompt_evaluation.py` to perform the CLIP Attribute Detection and Correlation Analysis.\n\n\n### Nvidia CUDA\n\nRepository is setup for usage with a CUDA which requires a Nvidia GPU, but will work on CPU as well (But a lot slower!). \nEg. running both scripts with 10000 and 100000 samples respectively on a 8 year old Nvidia GTX 1080 with 8GB VRAM took less than 10 minutes in total.\nThe same scripts on my laptop(i7 8650u) would take over 4 hours with CPU processing, which is ~25x slower.\nSo please decrease the `NUM_SAMPLES` at the top of each script to get a more reasonable run time if you don't have a Nvidia GPU.\n\n### celeba_downloader_clip_test.py\n\n\n\n![img](./celeba_demo.png)\n\nrunning `./celeba_downloader_clip_test.py` gives us: \n```\nUsing device: cuda\nLoading CLIP model...\nCelebA Dataset + CLIP Analysis\n==================================================\nLoading CelebA dataset...\nFiles already downloaded and verified\n✓ Successfully loaded CelebA dataset with 162770 images\n\n==================================================\nCELEBA DATASET EXPLORATION\n==================================================\nTotal number of images: 162770\nNumber of attributes: 40\n\nAnalyzing attributes distribution...\nSampling attributes: 100%|████████████████████████████████████████████████████████████████████████████| 162770/162770 [01:40\u003c00:00, 1626.66it/s]\n\n Attributes and how frequent they are:\n Attribute Count Percentage\n No_Beard 135779.0 83.41\n Young 126788.0 77.89\n Attractive 83603.0 51.36\nMouth_Slightly_Open 78486.0 48.21\n Smiling 78080.0 47.96\n Wearing_Lipstick 76437.0 46.96\n High_Cheekbones 73645.0 45.24\n Male 68261.0 41.93\n Heavy_Makeup 62555.0 38.43\n Wavy_Hair 51982.0 31.93\n Oval_Face 46101.0 28.32\n Pointy_Nose 44846.0 27.55\n Arched_Eyebrows 43278.0 26.58\n Big_Lips 39213.0 24.09\n Black_Hair 38906.0 23.90\n Big_Nose 38341.0 23.55\n Straight_Hair 33947.0 20.85\n Bags_Under_Eyes 33280.0 20.44\n Brown_Hair 33192.0 20.39\n Wearing_Earrings 30362.0 18.65\n Bangs 24685.0 15.16\n Blond_Hair 24267.0 14.90\n Bushy_Eyebrows 23386.0 14.36\n Wearing_Necklace 19764.0 12.14\n Narrow_Eyes 18869.0 11.59\n 5_o_Clock_Shadow 18177.0 11.16\n Receding_Hairline 13040.0 8.01\n Wearing_Necktie 11890.0 7.30\n Rosy_Cheeks 10525.0 6.46\n Eyeglasses 10521.0 6.46\n Goatee 10337.0 6.35\n Chubby 9389.0 5.76\n Sideburns 9156.0 5.62\n Blurry 8362.0 5.13\n Wearing_Hat 8039.0 4.93\n Double_Chin 7571.0 4.65\n Pale_Skin 7005.0 4.30\n Gray_Hair 6896.0 4.23\n Mustache 6642.0 4.08\n Bald 3713.0 2.28\n\nVisualizing 8 sample images...\n\nRunning CLIP analysis on 162770 images...\nProcessing batches: 100%|█████████████████████████████████████████████████████████████████████████████████████| 636/636 [06:26\u003c00:00, 1.64it/s]\n\nAnalyzing CLIP results...\n\nAverage CLIP confidences by text prompt:\na photo of a woman with makeup: 0.213\na photo of a woman smiling: 0.209\na photo of a man with a receding hairline: 0.198\na photo of a woman with blonde hair: 0.173\na photo of an attractive man: 0.130\na photo of a man with glasses: 0.077\n\nCorrelation analysis:\nMale images - average 'man' prompts confidence: 0.167\nFemale images - average 'woman' prompts confidence: 0.330\nImages with glasses - 'glasses' prompts confidence: 0.593\nImages without glasses - 'glasses' prompts confidence: 0.042\nImages with receding hairline - 'receding hairline' confidence: 0.431\nImages without receding hairline - 'receding hairline' confidence: 0.177\nAttractive images - 'attractive' prompts confidence: 0.106\nNon-attractive images - 'attractive' prompts confidence: 0.154\nSmiling images - 'smiling' prompts confidence: 0.347\nNon-smiling images - 'smiling' prompts confidence: 0.081\nHeavy makeup images - 'makeup' prompts confidence: 0.380\nNo heavy makeup images - 'makeup' prompts confidence: 0.109\nBlonde hair images - 'blonde hair' confidence: 0.714\nNon-blonde hair images - 'blonde hair' confidence: 0.079\n\nAnalysis complete!\nCheck 'celeba_samples.png' for sample images visualization\npython3 celeba_downloader_clip_test.py 1568.52s user 95.02s system 332% cpu 8:21.06 total\n```\n\n\n\n\n### distinct_prompt_evaluation.py\n\n```\n\nUsing device: cuda\nLoading CLIP model...\nLoading CelebA dataset...\nFiles already downloaded and verified\n✓ Successfully loaded CelebA dataset with 162770 images\n\nDataset loaded with 162770 images\nNumber of attributes: 40\nAttributes: 5_o_Clock_Shadow, Arched_Eyebrows, Attractive, Bags_Under_Eyes, Bald, Bangs, Big_Lips, Big_Nose, Black_Hair, Blond_Hair...\n\n============================================================================\nIMPROVED CLIP ANALYSIS WITH BINARY COMPARISONS\n============================================================================\n\nRunning improved CLIP analysis on 100000 images (batch size: 256, num_workers: 16, prefetch_factor: 2)...\nProcessing batches: 100%|█████████████████████████████████████████████████████████████████████████████████████| 391/391 [05:20\u003c00:00, 1.22it/s]\n\nImproved CLIP Analysis Results:\n\nGender Classification Accuracy:\nMale accuracy: 41173/41991 = 98.1%\nFemale accuracy: 57749/58009 = 99.6%\n\nAttribute Detection Performance:\n\nMetrics:\n- Avg Score (Has): How confident CLIP is when the attribute is present (higher is better)\n- Avg Score (Has not): How confident CLIP is when attribute is absent (lower is better)\n- Discrimination: Difference between the two (positive = CLIP can detect this attribute)\n\nAttribute | Avg Score (Has) | Avg Score (Has not) | Discrimination\n----------------------------------------------------------------------------\nmale | 0.945 | 0.033 | +0.912\nblond_hair | 0.917 | 0.264 | +0.653\nbald | 0.658 | 0.063 | +0.595\neyeglasses | 0.785 | 0.214 | +0.571\nsmiling | 0.604 | 0.159 | +0.445\npale_skin | 0.684 | 0.318 | +0.366\nbangs | 0.617 | 0.268 | +0.349\nyoung | 0.759 | 0.414 | +0.345\nwearing_hat | 0.778 | 0.474 | +0.304\nblack_hair | 0.677 | 0.419 | +0.257\ngray_hair | 0.489 | 0.278 | +0.211\nwavy_hair | 0.647 | 0.459 | +0.187\nheavy_makeup | 0.585 | 0.400 | +0.185\nbushy_eyebrows | 0.555 | 0.379 | +0.176\nstraight_hair | 0.915 | 0.763 | +0.152\nchubby | 0.574 | 0.438 | +0.136\nblurry | 0.367 | 0.233 | +0.134\nbrown_hair | 0.774 | 0.659 | +0.115\nmustache | 0.353 | 0.253 | +0.100\nwearing_lipstick | 0.718 | 0.636 | +0.082\nwearing_necktie | 0.688 | 0.607 | +0.082\nwearing_earrings | 0.712 | 0.635 | +0.077\narched_eyebrows | 0.742 | 0.671 | +0.071\ndouble_chin | 0.669 | 0.605 | +0.064\nattractive | 0.345 | 0.286 | +0.060\nnarrow_eyes | 0.641 | 0.588 | +0.053\nbags_under_eyes | 0.573 | 0.554 | +0.019\nmouth_slightly_open | 0.580 | 0.562 | +0.018\npointy_nose | 0.291 | 0.283 | +0.009\nbig_lips | 0.234 | 0.228 | +0.006\noval_face | 0.561 | 0.557 | +0.004\nbig_nose | 0.307 | 0.306 | +0.002\nwearing_necklace | 0.645 | 0.653 | -0.008\nhigh_cheekbones | 0.368 | 0.384 | -0.015\nreceding_hairline | 0.553 | 0.573 | -0.020\nsideburns | 0.510 | 0.547 | -0.037\ngoatee | 0.453 | 0.521 | -0.068\nrosy_cheeks | 0.227 | 0.302 | -0.075\nno_beard | 0.742 | 0.838 | -0.097\n5_o_clock_shadow | 0.182 | 0.490 | -0.309\n\nInteresting Cases:\n\n============================================================================\nEMBEDDING SPACE CORRELATION ANALYSIS\n============================================================================\n\nComputing embedding similarities for 100000 images...\n\nCLIP-CelebA Attribute Correlations:\n\nMetrics:\n- Correlation: How well CLIP's similarity scores align with CelebA labels (-1 to +1, higher (absolute) magnitude = better)\n- Mean Sim (WITH): Average cosine similarity (dot product of normalized embeddings) between images WITH the attribute and its text description\n- Mean Sim (WITHOUT): Average cosine similarity between images WITHOUT the attribute and its text description\n- Higher correlation means CLIP understands this attribute; larger gap between WITH/WITHOUT is better\n\nAttribute | Correlation | Mean Sim (WITH) | Mean Sim (WITHOUT)\n----------------------------------------------------------------------------\nSmiling | 0.640 | 0.241 | 0.220\nBlond_Hair | 0.554 | 0.252 | 0.222\nBangs | 0.519 | 0.238 | 0.211\nNo_Beard | -0.514 | 0.210 | 0.230\nHeavy_Makeup | 0.501 | 0.227 | 0.211\nWearing_Hat | 0.441 | 0.255 | 0.225\nBlack_Hair | 0.406 | 0.233 | 0.216\nWavy_Hair | 0.397 | 0.235 | 0.220\nGoatee | 0.393 | 0.245 | 0.218\nEyeglasses | 0.392 | 0.232 | 0.206\nWearing_Earrings | 0.389 | 0.246 | 0.232\nWearing_Lipstick | 0.378 | 0.222 | 0.211\nWearing_Necktie | 0.339 | 0.240 | 0.222\nBrown_Hair | 0.337 | 0.245 | 0.231\nArched_Eyebrows | 0.295 | 0.250 | 0.240\nBald | 0.290 | 0.245 | 0.210\nMouth_Slightly_Open | 0.265 | 0.235 | 0.228\nReceding_Hairline | 0.243 | 0.246 | 0.229\nMustache | 0.232 | 0.230 | 0.213\nYoung | 0.218 | 0.225 | 0.221\nBlurry | 0.214 | 0.229 | 0.217\nAttractive | 0.191 | 0.212 | 0.207\nBushy_Eyebrows | 0.187 | 0.242 | 0.235\nPale_Skin | 0.177 | 0.226 | 0.211\nGray_Hair | 0.167 | 0.234 | 0.221\nStraight_Hair | 0.128 | 0.230 | 0.224\nOval_Face | 0.111 | 0.242 | 0.240\nWearing_Necklace | 0.105 | 0.233 | 0.229\n5_o_Clock_Shadow | 0.102 | 0.224 | 0.221\nSideburns | 0.086 | 0.230 | 0.225\nBig_Lips | 0.078 | 0.222 | 0.220\nBags_Under_Eyes | -0.076 | 0.233 | 0.235\nPointy_Nose | 0.069 | 0.230 | 0.228\nChubby | 0.054 | 0.210 | 0.206\nHigh_Cheekbones | -0.025 | 0.212 | 0.214\nBig_Nose | -0.022 | 0.226 | 0.227\nDouble_Chin | 0.021 | 0.214 | 0.212\nMale | -0.012 | 0.224 | 0.224\nNarrow_Eyes | 0.005 | 0.227 | 0.227\nRosy_Cheeks | 0.005 | 0.220 | 0.220\n\nSaving analysis results...\nAnalysis complete! Results saved to 'clip_celeba_improved_results.json'\n\npython3 distinct_prompt_evaluation.py 1087.39s user 164.20s system 216% cpu 9:38.30 total\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffinnff%2Fvlm-bias-decomposition","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffinnff%2Fvlm-bias-decomposition","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffinnff%2Fvlm-bias-decomposition/lists"}