{"id":18690312,"url":"https://github.com/cdancette/detect-shortcuts","last_synced_at":"2025-04-12T05:54:32.618Z","repository":{"id":94538732,"uuid":"350766038","full_name":"cdancette/detect-shortcuts","owner":"cdancette","description":"Repo for ICCV 2021 paper: Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering ","archived":false,"fork":false,"pushed_at":"2024-07-01T18:17:48.000Z","size":60,"stargazers_count":26,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-12T05:54:25.824Z","etag":null,"topics":["biases","deep-learning","visual-question-answering"],"latest_commit_sha":null,"homepage":"https://cdancette.fr/projects/vqa-ce/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cdancette.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-23T15:37:56.000Z","updated_at":"2025-01-13T23:54:47.000Z","dependencies_parsed_at":"2023-03-13T16:59:08.895Z","dependency_job_id":null,"html_url":"https://github.com/cdancette/detect-shortcuts","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdancette%2Fdetect-shortcuts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdancette%2Fdetect-shortcuts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdancette%2Fdetect-shortcuts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdancette%2Fdetect-shortcuts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cdancette","download_url":"https://codeload.github.com/cdancette/detect-shortcuts/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525156,"owners_count":21118616,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["biases","deep-learning","visual-question-answering"],"created_at":"2024-11-07T10:47:16.850Z","updated_at":"2025-04-12T05:54:32.610Z","avatar_url":"https://github.com/cdancette.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Rule-Mining for shortcut discovery (VQA-CE)\n\n\u003cimg width=\"1477\" alt=\"vqa-ce\" src=\"https://user-images.githubusercontent.com/10550327/132852024-0dac6c04-3854-4007-a4b8-6017b2496ecd.png\"\u003e\n\n\nThis repo contains the rule mining pipeline described in the article : \n**Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering** by Corentin Dancette, Rémi Cadène, Damien Teney and \nMatthieu Cord (https://arxiv.org/abs/2104.03149).\nIt also provides the VQA-CE dataset.\n\nWebsite here: https://cdancette.fr/projects/vqa-ce/\n\nThis code was developped with python 3.7 and pytorch 1.7.0.\n\n### VQA-CE\nThe VQA-CE counterexamples subset can be downloaded here : \n- counterexamples: https://github.com/cdancette/detect-shortcuts/releases/download/v1.0/counterexamples.json\n- hard: https://github.com/cdancette/detect-shortcuts/releases/download/v1.0/hard.json\n\nThe \"easy\" subset can be obtained by substracting counterexamples and hard from all question_ids.\n\n## Usage\n\n### Installing requirements\nFirst, you need to install gminer. Follow instructions at https://github.com/cdancette/GMiner.\n\nFor python requirements, run `pip install -r requirements.txt`. This will install pytorch, numpy and tqdm.\n### Visual Question Answering (VQA)\n\n#### Download VQA and COCO data\n\nFirst, run `./download.sh`. Data will be downloaded in the `./data` directory. \n\n#### Run the rule mining pipeline\n\nThen run `python vqa.py --gminer_path \u003cpath_to_gminer\u003e` to run our pipeline on the VQA v2 dataset.\nYou can change the parameters, see the end of the `vqa.py` file or run `python vqa.py --help`.\n\nThis will save in logs/vqa2 various files containing the rules found in the dataset, \nthe question_ids for easy and counterexamples splits, and the predictions made by the rule model.\n\nTo evaluate predictions, you can use the [multimodal](https://github.com/cdancette/multimodal) library: \n\n```bash\npip install multimodal\npython -m multimodal vqa2-eval -p logs/vqa2/rules_predictions.json --split val\n```\n\n\n### Other task\n\n\n#### fit\nYou can use our library to extract rule for any other dataset.\n\nTo do so, you can use the `fit` function in our `rule_mining.py`\nIt takes the following arguments : \n`fit(dataset, answer_ids, gminer_support=0.01, gminer_max_length=0, gminer_path=None)`, where : \n\n- `dataset` is a list of transactions. Each transaction is a list of integers describing tokens. \n- `answer_ids` is a list of integers, describing answer ids. They should be contained between 0 and max answer id.\n- `gminer_support` is the minimum support used to mine frequent itemset.\n- `gminer_max_length`: minimum length of an itemset. By default no minimum length\n- `gminer_path`: path to the gminer binary you compiled (see top of the readme).\n\n\nThe function returns a list of rules, contained in namedtuples: `Rule = namedtuple(\"Rule\", [\"itemset\", \"ans\", \"sup\", \"conf\"])`.\n\nThe itemset contains the input token ids, ans is the answer id, sup and conf are the support and the confidence of this rule.\n\n#### match_rules\n\nWe provide a function to get, for each example in your dataset, all rules matching its input.\n\n`match_rules(dataset, rules, answers=None, bsize=500)`\n\nThis will return `(matching_rules, correct_rules)`, where `matching_rules` is a list of the same length as the dataset, giving for each example, the matching rules. \n\nYou can use this to build your counterexamples subset (examples where all rules are incorrect), or your easy subset (where at least one rule is correct).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcdancette%2Fdetect-shortcuts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcdancette%2Fdetect-shortcuts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcdancette%2Fdetect-shortcuts/lists"}