{"id":19401180,"url":"https://github.com/google-research/fool-me-twice","last_synced_at":"2026-04-02T02:05:38.876Z","repository":{"id":50762172,"uuid":"349044477","full_name":"google-research/fool-me-twice","owner":"google-research","description":"Game code and data for Fool Me Twice: Entailment from Wikipedia Gamification https://arxiv.org/abs/2104.04725","archived":false,"fork":false,"pushed_at":"2025-02-13T16:52:14.000Z","size":13596,"stargazers_count":19,"open_issues_count":24,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-03T01:01:50.078Z","etag":null,"topics":["entailment","fever","firebase","game","nlp","verification","wikipedia"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-18T11:17:26.000Z","updated_at":"2025-02-22T23:30:49.000Z","dependencies_parsed_at":"2022-09-17T00:22:54.255Z","dependency_job_id":null,"html_url":"https://github.com/google-research/fool-me-twice","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Ffool-me-twice","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Ffool-me-twice/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Ffool-me-twice/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Ffool-me-twice/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/fool-me-twice/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250582784,"owners_count":21453913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["entailment","fever","firebase","game","nlp","verification","wikipedia"],"created_at":"2024-11-10T11:17:27.460Z","updated_at":"2026-04-02T02:05:33.844Z","avatar_url":"https://github.com/google-research.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fool Me Twice\n\nCode for the game, experiments and data introduced in the paper [Fool Me Twice: Entailment from Wikipedia Gamification](https://arxiv.org/abs/2104.04725) published at NAACL 2021.\n\nIn this work we set up a game for players to distinguish entailed from refuted statements by looking at the evidence from their Wikipedia page. Players can also write their own statements and they are rewarded by making them hard to guess but properly backed by the evidence, which they have to choose. We release a dataset of nearly 13K statements with their evidence.\n\nThis is how the game looks: the first screen allows users to choose topics of their interest. In the second screen they reveal evidence and make a guess before time runs out.\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/menu.png\" height=\"200\"\u003e\n\u003cimg src=\"images/vote.png\" height=\"200\"\u003e\n\u003c/p\u003e\n\n## Download the data\n\n| Split  | Claims | Entailed Proportion | Pages | Avg. # Tokens Claim | Avg. # Tokens Evidence |\n|--------|--------|---------------------|-------|---------------------|------------------------|\n| [Train](https://github.com/google-research/fool-me-twice/blob/main/dataset/train.jsonl) | 10,419 | 49.2%               | 1,811 | 15                  | 30                     |\n| [Dev](https://github.com/google-research/fool-me-twice/blob/main/dataset/dev.jsonl)   | 1,169  | 51.0%               | 209   | 15                  | 31                     |\n| [Test](https://github.com/google-research/fool-me-twice/blob/main/dataset/test.jsonl)  | 1,380  | 49.4%               | 234   | 15                  | 31                     |\n| Total  | 12,968 | 49.4%               | 2,254  | 15                  | 30                     |\n\n## Running analysis and training models on the date\n\n1. NLI Baselines [notebook](https://colab.research.google.com/github/google-research/fool-me-twice/blob/master/notebooks/nli_baselines.ipynb) trains BERT entailment models on FM2 and FEVER.\n\n2. The LMI Analysis [notebook](https://colab.research.google.com/github/google-research/fool-me-twice/blob/master/notebooks/lmi_analysis.ipynb) shows the most informative terms in FM2 compared to FEVER.\n\n## Running the game\n\nFor the time being we are not publishing the game on the web, but we encourage people\nto deploy it, collect and release new data and experiment with different game dynamics.\n\n### Setting-up Firebase\n\nThis project does not require a server, and uses Firebase for authentication, database, storage, serverless compute, and optionally hosting. For small volumes we don't think it should incur any significant cost,\nbut it will require setting up a Pay-As-You billing account. If you are new to Google Cloud, you can use the [$300 credit](https://cloud.google.com/free/docs/gcp-free-tier#free-trial), available at the time of writing. If someone else already set-up firebase you can copy over their config `board/src/config/firebase.json` (from step 3) and skip all the rest of the steps.\n\n1. Go to the [Firebase console](https://console.firebase.google.com/) and create a new project. You can name it in any way you like, and the name will be part of the URL if you choose to host the game through Firebase. If asked, we recommend enabling Google Analytics.\n\n2. Once the project is created, add a **Web App**, which can named \"board\", and enable hosting. No need to run any of the commands at this point.\n\n3. Click on the **Gear + Project Settings** on the top left. Set up a public-facing name as well a support email for the app. Down below in the App section, if you click on the Config radio button you can see the JSON config for your project. Create a new file in `board/src/config/firebase.json` and paste the content. Because we are using a JSON file, so no comments or no trailing commas and the keys have to be quoted. The content should look like this:\n```json\n{\n    \"apiKey\": \"API_KEY\",\n    \"authDomain\": \"PROJECT_ID.firebaseapp.com\",\n    \"databaseURL\": \"https://PROJECT_ID.firebaseio.com\",\n    \"projectId\": \"PROJECT_ID\",\n    \"storageBucket\": \"PROJECT_ID.appspot.com\",\n    \"messagingSenderId\": \"SENDER_ID\",\n    \"appId\": \"APP_ID\",\n    \"measurementId\": \"G-MEASUREMENT_ID\"\n}\n```\n\n4. Click on the **Gear + Usage and Billing** in the top left and and then in the \"Details \u0026 Settings\" tab change the plan to Blaze (Pay as you go). We recommend that you set-up a budget alert to make sure nothing unexpected happens.\n\n5. Navigate to the **Authentication Section** and add the sign-in methods allowed for the game. At least one should enabled, for example using a Google account.\n\n6. Navigate to the **Firestore Section** and create a new database. Select test mode and then pick a zone near you.\n\n7. Configure functions and firestore (replace your PROJECT_ID)\n```shh\n$ npm install -g firebase-tools\n$ cd firebase\n$ firebase login\n$ firebase use PROJECT_ID\n$ firebase deploy  # Enable the firestore API if asked\n```\n\n8. Install [Google Cloud SDK](https://cloud.google.com/sdk/install) and authenticate\n```sh\ngcloud init\ngcloud auth application-default login\ngcloud auth application-default set-quota-project PROJECT_ID\n```\n\n9. Populate database with a small number of pages and claims\n```sh\ncd ..\nsudo apt-get install npm\nnpm install -g lunr\npip install -r requirements.txt\npython -m spacy download en_core_web_sm\npython util/bootstrap.py\n```\n\n### Starting the game locally\n\n```sh\ncd board\nnpm install\nnpm run local\n```\n\n### Developing\n\nThe project is integrated with a pre-commit eslint hook, you can check the linter\nor fix in place with the `eslint src` command, or running `npm run lint`.\n\n### Deploying the game in a production setting on a server\n\nThe following snippet will package the app and serve a production build, if you wish to use\nfirebase to host as well, you can configure it following the steps [here](https://firebase.google.com/docs/hosting/quickstart)\ndirectly in `board` folder, but any number of hosting alternative will also serve.\n\n```sh\ncd board\nnpm run prestart # Only needs to run once\nnpm run build\nnpm run start\n```\n\n### Uploading more Wikipedia pages\n\nThe `categories` folder contains a list of pages in different categories that\ncan be changed in any way. The * at the beginning of the page denotes priority\nto use it when creating a workflow (see below).\n\nRunning `python util/download_wiki.py` will fetch the lazily fetch Wikipedia\npages, index them and upload them to storage for the game to use.\n\n### Creating new Workflows and downloading the data\n\nWe use the term Workflow to denote the screen a user sees when logging into the game.\nIt tracks the progress and shows tasks to write or vote on claims. Any claim that\nthe player has authored will not be shown.\n\nThe workflow is not updated dynamically at the moment; you can run a script\n(we suggest daily) to generate new tasks for players. This will allow players to\nsee new content and get points for as the claims they wrote are voted by other players.\n\nTo run the workflow generation, run the script `python util/create_workflow.py`.\nThis will create a new workflow with the name of today's date (e.g., 2020-12-25).\nYou then need to change the name of the active workflow in the Firebase remote\nconfig to reflect the current active workflow. This can be done by navigating \nto https://console.firebase.google.com/u/0/project/PROJECT_ID/config and adding a string parameter\n`kWorkflowID`. This parameter will override in realtime the `defaultConfig` of [board/src/config/server.js](https://github.com/google-research/fool-me-twice/blob/main/board/src/config/server.js).\n\nTo debug, change the name of the workflow to something not actively being used (this is possible with a flag, check options using `python util/create_workflow.py --help`).\nThe script will also do a local backup of all the claims created so far.\nAfter you run the create workflow script, you can import the resulting `csv` file into a spreadsheets to inspect the data.  Sort by the time created to see the latest claims.\n\n## \u003ca name=\"how-to-cite-tapas\"\u003e\u003c/a\u003eHow to cite FM2?\n\nPlease cite Fool Me Twice: Entailment from Wikipedia Gamification published at NAACL 2021\n```\n@inproceedings{eisenschlos-etal-2021-fool,\n    title = \"Fool Me Twice: Entailment from {W}ikipedia Gamification\",\n    author = {Eisenschlos, Julian Martin  and\n      Dhingra, Bhuwan  and\n      Bulian, Jannis  and\n      B{\\\"o}rschinger, Benjamin  and\n      Boyd-Graber, Jordan},\n    booktitle = \"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies\",\n    month = jun,\n    year = \"2021\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://www.aclweb.org/anthology/2021.naacl-main.32\",\n    pages = \"352--365\",\n    abstract = \"We release FoolMeTwice (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial examples, drastically lowering the number of examples that can be solved using {``}shortcuts{''} compared to other popular entailment datasets. Players are presented with two tasks. The first task asks the player to write a plausible claim based on the evidence from a Wikipedia page. The second one shows two plausible claims written by other players, one of which is false, and the goal is to identify it before the time runs out. Players {``}pay{''} to see clues retrieved from the evidence pool: the more evidence the player needs, the harder the claim. Game-play between motivated players leads to diverse strategies for crafting claims, such as temporal inference and diverting to unrelated evidence, and results in higher quality data for the entailment and evidence retrieval tasks. We open source the dataset and the game code.\",\n}\n```\n\n## Disclaimer\n\nThis is not an official Google product.\n\n## Contact information\n\nFor help or issues, please submit a GitHub issue.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Ffool-me-twice","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Ffool-me-twice","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Ffool-me-twice/lists"}