{"id":23124359,"url":"https://github.com/cconzen/dh-proj","last_synced_at":"2025-04-04T04:27:23.498Z","repository":{"id":249803041,"uuid":"604546274","full_name":"cconzen/DH-Proj","owner":"cconzen","description":"Methods in Digital Humanities seminar project","archived":false,"fork":false,"pushed_at":"2023-03-14T14:30:05.000Z","size":107748,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-09T16:09:35.659Z","etag":null,"topics":["cooccurrence","newspaper","qatar-world-cup","tf-idf","time-series","topic-modelling","webcrawler","webscraper"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cconzen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-21T09:38:22.000Z","updated_at":"2023-03-16T18:35:38.000Z","dependencies_parsed_at":"2024-07-23T13:19:26.434Z","dependency_job_id":null,"html_url":"https://github.com/cconzen/DH-Proj","commit_stats":null,"previous_names":["cconzen/dh-proj"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cconzen%2FDH-Proj","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cconzen%2FDH-Proj/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cconzen%2FDH-Proj/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cconzen%2FDH-Proj/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cconzen","download_url":"https://codeload.github.com/cconzen/DH-Proj/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247120621,"owners_count":20886939,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cooccurrence","newspaper","qatar-world-cup","tf-idf","time-series","topic-modelling","webcrawler","webscraper"],"created_at":"2024-12-17T07:40:23.842Z","updated_at":"2025-04-04T04:27:18.488Z","avatar_url":"https://github.com/cconzen.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Qatar World Cup UK Newspaper Analysis\n\n## Newspapers\n\nNewspaper | The Sun | Daily Mail | The Guardian | The Times\n--- | --- | --- | --- | ---\ntype | tabloid | tabloid | broadsheet | broadsheet\ntimespan | Sep 22 - Feb 23 | Sep 22 - Feb 23 | Sep 22 - Feb 23 | Sep 22 - Feb 23\narticles collected | 1270 | 2622 | 2199 | 805\nmean article length | 433 tokens | 678 tokens | 1304 tokens | 927 tokens \nvocabulary | 17499 lemmas | 24781 lemmas | 39283 lemmas | 22586 lemmas\ncomplete articles | yes | yes | yes | yes\ntokens complete | 549898 | 1777469 |  2608640 | 746153\nwithout stopwords | 307902 | 998000 |  1473926 | 412010\nwithout symbols | 271541 | 897411 |  1273947 | 362093\nlemmas final | 225749 | 792397 |  1138170 | 301143\n\n## Crawlers\n\nCrawlers are written to scrape articles from September 2022 including February 2023.\n\n### The Sun, Daily Mail\n\nSelenium to preload pages + Scrapy to crawl and scrape\n\n### The Times\n\nSelenium to preload and login; Scrapy and Selenium to crawl and scrape\n\nstart crawler in terminal with \n\n````\nscrapy runspider [crawler.py] -o [articles .json or .csv]\n````\n\n### The Guardian\n\nGuardian API + BeautifulSoup4 to clean body html\n\nrun script in .py file\n\n## Functions\n\nmost functions can be found in \u003cb\u003epreprocessing.py\u003c/b\u003e and \u003cb\u003emethods.py\u003c/b\u003e\n\n### preprocessing.py\n#### def preprocess(newspaper, csv=False, rare=False)\n\nPreprocesses text data from JSON files for four different newspapers (The Times, The Sun, Daily Mail and The Guardian), including tokenisation, removal of stopwords, punctuation, rare tokens and player names, part-of-speech tagging, and lemmatisation. Depending on \u003cb\u003ecsv\u003c/b\u003e It returns a Pandas DataFrame containing the preprocessed data or writes a csv file; \u003cb\u003erare\u003c/b\u003e toggles whether rare tokens (= tokens appearing less than ten times) are kept in or not.\n````\ndataframe = preprocess(\"sun\",csv=False,rare=True)\n\npreprocess(\"guardian\",csv=True)\n\npreprocess(\"times\",csv=True,rare=True)\n````\n### methods.py\n#### def df_to_dtm(df)\nConverts a pandas DataFrame of preprocessed text data (obtained using preprocessing() ) into a Document-Term Matrix (DTM) using a CountVectorizer.\n````\ndtm_dataframe = df_to_dtm(dataframe)\n````\n#### def df_to_tfidf(df)\nConverts a pandas DataFrame of preprocessed text data (obtained using preprocessing() ) into a Term Frequency-Inverse Document Frequency (TF-IDF) matrix using a CountVectorizer and a TfidfTransformer.\n````\ndf_tfidf = df_to_tfidf(dataframe)\n````\n#### def csv_to_tfidf(file_path)\nConverts a CSV file of preprocessed text data (obtained using preprocessing() ) into a Term Frequency-Inverse Document Frequency (TF-IDF) matrix using a CountVectorizer and a TfidfTransformer.\n````\ndf_tfidf_guardian = csv_to_tfidf(\"guardian.csv\")\n````\n#### def get_term_position(df_tfidf, term)\nGets the column index of a given term in a TF-IDF matrix represented by a pandas DataFrame.\n````\nget_term_position(df_tfidf_guardian, \"climate\")\n````\n#### def compare_term_position(term)\nCompares the positions of a given term in the TF-IDF matrices of four different CSV files and write the results to a new CSV file.\n````\ncompare_term_position(\"climate\")\n````\n#### def read_csv_files()\nReads the CSV files for The Guardian, Daily Mail, The Times, and The Sun and returns them as dataframes.\n\n#### def plot_tfidf(term, save=False)\nPlots the development of the normalised TF-IDF score for a given term across four UK newspapers: The Times, Daily Mail, The Sun, and The Guardian, for the period between September 2022 and February 2023.\n````\nplot_tfidf(\"lgbt\", save=True)\n````\n#### def get_vocab_from_csv(csv_file, lemma_col='lemmas')\nReads a CSV file containing lemmas and returns a set of unique lemmas (the vocabulary).\n````\ntimes_vocab = get_vocab_from_csv(\"times_rare.csv\")\n````\n#### def get_avg_token_length(vocab)\nCalculates the average length of tokens in a vocabulary (input vocabulary must be a set).\n````\nget_avg_token_length(times_vocab)\n````\n\n## Analysis\n\n### Type-token ratio \nNewspaper | The Sun | Daily Mail | The Guardian | The Times\n--- | --- | --- | --- | ---\n types | 17499 | 24781 | 39283 | 22586 \n tokens | 271541 | 754718 | 1273947 | 362093 \n type-token ratio | 0,06444 | 0,03283 | 0,03083 | 0,06238 \n \n ### Average token length\nNewspaper | The Sun | Daily Mail | The Guardian | The Times\n--- | --- | --- | --- | ---\naverage token length | 7.11898 | 7.32034 | 7.44423 | 7.43956 \n\n\n### Topic modelling\nReads CSV and calculates the Topics for each newspaper.\n\ntopics.R\n\n### Cooccurrence measures\nShow the terms that frequently occur together and the structure of the data.\n\ncooccurrence.R\n\n### Time series\nCalculates the data and analyzed it overtime.\n\ntimeseries.R\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcconzen%2Fdh-proj","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcconzen%2Fdh-proj","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcconzen%2Fdh-proj/lists"}