{"id":25126158,"url":"https://github.com/cmry/simple-queries","last_synced_at":"2026-04-16T22:31:06.641Z","repository":{"id":89446841,"uuid":"99226889","full_name":"cmry/simple-queries","owner":"cmry","description":"Repository for the experiments and dataset described in \"Simple Queries as Distant Labels for Predicting Gender on Twitter\" presented at W-NUT 2017. ","archived":false,"fork":false,"pushed_at":"2017-10-31T15:57:13.000Z","size":745,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-02T18:52:20.607Z","etag":null,"topics":["author","corpus","dataset","distant-supervision","gender","machine-learning","prediction","profiling","text-mining","twitter"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cmry.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-03T11:45:37.000Z","updated_at":"2024-08-10T20:15:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"46313bd1-2325-4e0b-8cde-e9958d22c91a","html_url":"https://github.com/cmry/simple-queries","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cmry/simple-queries","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmry%2Fsimple-queries","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmry%2Fsimple-queries/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmry%2Fsimple-queries/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmry%2Fsimple-queries/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cmry","download_url":"https://codeload.github.com/cmry/simple-queries/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmry%2Fsimple-queries/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31907424,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"ssl_error","status_checked_at":"2026-04-16T18:21:47.142Z","response_time":69,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["author","corpus","dataset","distant-supervision","gender","machine-learning","prediction","profiling","text-mining","twitter"],"created_at":"2025-02-08T09:19:19.192Z","updated_at":"2026-04-16T22:31:06.584Z","avatar_url":"https://github.com/cmry.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple Queries as Distant Labels for Predicting Gender on Twitter\n\nRepository for the work described in [Simple Queries as Distant Labels for\nPredicting Gender on Twitter](http://www.aclweb.org/anthology/W17-4407),\npresented at [W-NUT 2017](http://noisy-text.github.io/2017/index.html). Code is released under the MIT license, the data under [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). If you use anything in our repository, please cite the following work:\n\n```\n@inproceedings{emmery2017simple,\n  title={Simple Queries as Distant Labels for Predicting Gender on Twitter},\n  author={Emmery, Chris and Chrupa{\\l}a, Grzegorz and Daelemans, Walter},\n  booktitle={Proceedings of the 3rd Workshop on Noisy User-generated Text},\n  pages={50--55},\n  year={2017}\n}\n```\n\nAlso consider citing the corpora that we used to compare when using the readers\nsupplied with our work. See the [docstrings](https://github.com/cmry/simple-queries/blob/master/sec3_data.py#L410) for references.\n\n## Overview\n\n- [tl;dr](https://github.com/cmry/simple-queries#tldr)\n- [Quick Start](https://github.com/cmry/simple-queries#quick-start)\n  - [Reproduction](https://github.com/cmry/simple-queries#reproduction)\n  - [Build your Own Classifier](https://github.com/cmry/simple-queries#build-your-own-gender-classifier)\n- [Dependencies](https://github.com/cmry/simple-queries#dependencies)\n- [Paper Data](https://github.com/cmry/simple-queries#paper-data)\n\n## tl;dr\n\nThis repository offers scripts to automatically label Twitter users by gender,\nusing simple queries for self-reports (\"*I'm a {girl, boy, man, woman, ...}*\").\nUsing batches of 200 tweets from their timelines, we train a\n[fastText](https://github.com/facebookresearch/fastText) model to classify\n**other users** by gender. This distant supervision achieves\ncomparable, or better performance than state-of-the-art methods, without the\nneed of manual annotation. The collection can be repeated weekly and yields\nroughly 10k profiles.\n\n\n## Quick Start\n\nNot only do we supply code to *replicate* our experiments, we also offer a\nflexible API to use the same method for distant labeling of users by gender.\n\n\u003e **Support Disclaimer:** The code is written for MacOS and Linux systems. This\n  limitation is partly due to fastText.\n\n### Reproduction\n\nSimply run the `sec...py` files in this order: data -\u003e proc -\u003e exp -\u003e res:\n\n```shell\npython3 sec3_data.py\npython3 sec3_proc.py\nsh sec4_exp.sh\npython3 sec5_res.py\n```\n\n\u003e **Important Note**: Please make sure that your own Twitter API keys are added\n  in `misc_keys` and the correct files are in `corpora` if you'd like to\n  replicate the comparison with the other corpora. Only our own data comes with\n  the repository!\n\nIf you don't want to run the comparisons, comment out the `Plank` and `Volkova`\nparts for every `__main__` boilerplate at the end of each `.py` file (not the\n`.sh`). If you'd like to add **newly** collected data in for the comparison,\nuncomment the `twitter` parts in the same boilerplate.\n\n\u003e **Note on Score Reproduction**: The **annotation** scores differ slightly,\n  as the exact user ids that were annotated when writing the paper differ from\n  the complete set (the reproduction shows an almost exact approximation). With\n  regards to the **classification** scores: i) Twitter's Terms of Service\n  formally prohibit sharing the message IDs, and ii) a large amount of\n  accounts and tweets have been removed since the date of collection.\n\n### Build your Own Gender Classifier\n\nYou can supply a general text wrapping in `query_string`. We assumed that\npeople self-report using \"*I'm a {girl, boy, man, woman}*\" etc.\n\n```python\nquery_string = 'm a {0}'\nquery_words = {'girl': 'f', 'boy': 'm', 'man': 'm', 'woman': 'f', 'guy': 'm',\n               'dude': 'm', 'gal': 'f', 'female': 'f', 'male': 'm'}\n```\n\nThere's a list for *removing* tweets (`filters`) by certain patterns like\nretweets, or quotes (e.g. 'Random woman: \"I'm a man\"'), that might not be\nself-reports. Moreover, the gender label can be flipped if certain words\nor in the full tweet (`flip_any`) or *before* the query string (`flip_prefix`).\nExamples where this might be relevant are for example \"don't assume\nI'm a girl\".\n\n\u003e Please note that while all examples focus on a binary representation of\n  gender (as this is common in related work), the code to a certain extent also\n  allows for the inclusion of other gender identities by altering the\n  `query_words` mapping. In this case however, the label flipping logically does\n  not work.\n\n```python\nfil = ['rt ', '\"', ': ']\nflp_any = [\"according to\", \"deep down\"]\nflp_pfx = [\" feel like \", \" where \", \" as if \", \" hoping \", \" assume \",\n           \" think \", \" assumes \", \" assumed \", \" assume that \",\n           \" assumed that \", \" then \", \" expect that \", \" expect \",\n           \"that means \", \" means \", \" think \", \" implying \", \" guess \",\n           \" thinks \", \" tells me \", \" learned \", \" if \"]\n```\n\n#### Collecting Data\n\nFrom here, collection is simply setting all this information in the\n`DistantCollection` class initialization, and running the collection methods.\n\n\u003e Make sure you've added your own Twitter API keys to `misc_keys` before running!\n\n```python\nfrom sec3_data import DistantCollection\n\ndc = DistantCollection(query_string=qs, query_words=qw, filters=fil,\n                       flip_any=flp_any, flip_prefix=flp_pfx,\n                       clean_level='messages', db_id='your_db_id')\n```\n\nAnd finally fetching the users, and their timelines:\n\n```python\ndc.fetch_query_tweets()  # users\ndc.fetch_user_tweets()   # timelines\n```\n\n#### Preprocessing and Batching\n\nOnce the collection is done, you need to set up a label mapping (depending) on\nwhat you've entered as keys in `query_words`, and run that through the\npreprocessing function:\n\n```python\nfrom sec3_proc import data_to_batches\n\nlm = {'m': 0, 'f': 1, 'M': 0, 'F': 1}\ndata_to_batches(db_id='your_db_id', label_mapping=lm)\n```\n\n#### Scoring\n\nUsing the `fastText` function prints a train / test evaluation. If you'd like\nto run `MajorityBaseline` and the `LexiconGender` classifier from Sap et al.,\nuse the code below:\n\n```python\nfrom sec5_res import MajorityBaseline, LexiconGender, fastText\n\nbl = MajorityBaseline()\nlg = LexiconGender()\n\ntrain = test = 'your_db_id'\nfastText(train + '_gender', test + '_gender')  # outputs results\n\nfin = open('./data/{0}_gender.test'.format(test)).readlines()\nsscore = round(lg.lex_score(fin), 3)\nbscore = round(bl.mb_score(fin), 3)\nprint(\"Sap baseline @ test: {}\".format(sscore))\nprint(\"Maj baseline @ test: {}\".format(bscore))\n```\n\nThis should cover setting up your own classifier. If you'd like to re-use\nfastText after training is done, refer to their documentation and change\n`sec4_exp.sh` accordingly.\n\n## Dependencies\n\nThese are the packages (and their version tested with the latest version of\nthe repository):\n\n```\npandas        0.20.1\ntweepy        3.5.0\nlangdetect    1.0.7 (optional)\nscikit-learn  0.18.1\nspacy         1.9.0\n```\n\n## Paper Data\n\nThe data can be found under `./corpora/query-gender.json`. The file is\nstructured as follows:\n\n```json\n{\n    \"annotations\": {\n        \"user_id\": {\n            \"ann1\": \"m / f / o / - / 0\",\n            \"ann2\": \"m / f / o / - / 0\",\n            \"ann3\": \"m / f / o / - / 0\",\n            \"bot\": \"True / False\",\n            \"majority\": \"m / f\",\n            \"query_label\": \"m / f\",\n            \"query_label2\": \"m / f\"\n        },\n        \"...\": {\n        },\n    },\n    \"filtered\": {\n        \"tweet_id\": \"user_id\",\n        \"...\": ,\n    }\n}\n```\n\nNot all three annotators annotated every profile. The annotations from the\npaper only used 2 annotators; an approximate implementation of this evaluation\nis provided in `sec5_res.py`. Annotation field values are: `m` for male, `f`\nfor female, `o` anything else, `-` not sure, and `0` for not annotated. A\nmajority decision based on the three annotators is provided in `majority` (only\nbased on `m` / `f` annotations). There's also a label if the user was\nconsidered to be a bot (this label wasn't used in the paper), and there's two\ntags (`query_label`) provided by the Query heuristic from the paper. The\nfirst is the label **without** using the rule to flip the label, the second\n**with** the label flipped if necessary. Finally, the original Query hit\ntweets (that were removed from the final train and test data) are listed under\n`filtered`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmry%2Fsimple-queries","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcmry%2Fsimple-queries","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmry%2Fsimple-queries/lists"}