{"id":16994636,"url":"https://github.com/znerol/rejoha22-recommender","last_synced_at":"2025-03-22T05:43:35.738Z","repository":{"id":63784509,"uuid":"570617461","full_name":"znerol/rejoha22-recommender","owner":"znerol","description":null,"archived":false,"fork":false,"pushed_at":"2022-11-26T10:22:22.000Z","size":305,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-01-27T06:28:03.537Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/znerol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-11-25T16:21:02.000Z","updated_at":"2022-11-25T16:22:15.000Z","dependencies_parsed_at":"2023-01-22T07:00:40.691Z","dependency_job_id":null,"html_url":"https://github.com/znerol/rejoha22-recommender","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/znerol%2Frejoha22-recommender","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/znerol%2Frejoha22-recommender/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/znerol%2Frejoha22-recommender/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/znerol%2Frejoha22-recommender/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/znerol","download_url":"https://codeload.github.com/znerol/rejoha22-recommender/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244913005,"owners_count":20530769,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T03:46:07.810Z","updated_at":"2025-03-22T05:43:35.708Z","avatar_url":"https://github.com/znerol.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"REJOHA 2022 - Responsible News Recommender Systems\n==================================================\n\nThis repository contains results for the\n[Responsible News Recommender Systems](https://hack.opendata.ch/project/893)\nchallange at Rethink Journalism Hackaton 2022. Note that a responsible\nrecommender systems is an entire field of research in itself. This project\nbarely scratches the surface of it.\n\n## Project goals\n\nNews sites often present a list of most-clicked content to their users. Since\nsome categories of content are more popular than others (e.g., News and Sports),\nthere is a risk that popular content from under-represented categories just\nnever makes it onto these lists.\n\nThis project explores ways to boost the diversity of categories represented in\nmost-clicked content lists.\n\n## Data / Tools / License\n\nData is sourced from the SRG-SSR\n[mostClicked](https://developer.srgssr.ch/apis/srgssr-video/docs/mostclicked)\nendpoint for video content.\n\nA developer account at SRG-SSR developer portal is necessary to run the software\nand reproduce the results presented here. See the\n[FAQ](https://developer.srgssr.ch/faq) for more info.\n\nSource code is written in [go](https://go.dev/) and released under the\n[unlicence](http://unlicense.org/).\n\n## Step 1: Scrape data from SRG most-clicked video endpoint\n\nBased on the aggregated popularity data (available for certain content on the\nSRF APIs, for example), the first step would be to try to find out which\n\"categories\" of content are over- or under-represented in the most popular\ncontent.\n\n### Top 200 videos\n\nGenerate a list of top 200 videos:\n\n    export SRG_OAUTH_TOKEN=xyz\n    go run cmd/top200/main.go \u003e /tmp/top-200.tsv\n\nSample output:\n\n    Popularity\tCategory\tTitle\tId\n    19483\tSport\tDoppelter Richarlison schiesst Brasilien mit Traumtor zum Sieg\t516a880f-366f-43bb-9075-daccf1eddc1c\n    14244\tNews\tTagesschau vom 24.11.2022: Hauptausgabe\t60aa9e97-cfb1-48bb-951b-363107cb46a0\n    8059\tNews\t10 vor 10 vom 24.11.2022\t17c59931-8c51-4509-a3dd-6e67c3416b5f\n    7630\tSport\tSchweizerinnen kämpfen um EM-Gold\tfe0c0275-dbc6-4d3d-8441-9ebedf52df7c\n    6259\tKultur\tMythos Matter\t6702d57e-2f65-424d-8a0a-176a65b5e1f0\n    ...\n\n### Top 200 videos aggregated by category\n\nAggregate number of videos for every category:\n\n    export SRG_OAUTH_TOKEN=xyz\n    go run cmd/catpop200/main.go \u003e /tmp/catpop-200.tsv\n\nSample output:\n\n    Popularity\tCategory\n    66303\tSport\n    46909\tNews\n    28289\tUnterhaltung\n    13639\tFilm\n    10798\tDok \u0026 Reportagen\n    10666\tKultur\n    9730\tWissen \u0026 Ratgeber\n\n\n### Top 200 videos aggregated by views\n\nAggregate number of views for every category:\n\n    export SRG_OAUTH_TOKEN=xyz\n    go run cmd/catcount200/main.go \u003e /tmp/catcount-200.tsv\n\nSample output:\n\n    Sum\tCategory\n    38\tSport\n    31\tNews\n    20\tUnterhaltung\n    19\tDok \u0026 Reportagen\n    18\tFilm\n    17\tSerien\n\n## Step 2: Boost under-represented content in most-clicked video list\n\nThe top hits of the most clicked videos are often from the same category\n(sports, news). A simple way to boost content from less represented categories\nto a better rank is to normalize the views within categories. Each entry is\ngiven a score between 0 and 1, and the list is ordered by that score instead of\nby the number of clicks.\n\n### Top 200 videos with clicks normalized within each category\n\nThe most basic way to compute the score is to simply normalize the number of\nclicks within the same category. The piece with most clicks in category A is\nassigned a score of 1. The score of the remaining items in the same category is\ncomputed by scaling down the number of clicks by the same factor.\n\nThis results in a list where each category is represented with exactly one item\nat the top of the list (each of them with score 1).\n\nGenerate a list of top 200 videos (score by normalizing popularity within each category):\n\n    export SRG_OAUTH_TOKEN=xyz\n    go run cmd/topnorm200/main.go \u003e /tmp/topnorm-200.tsv\n\nSample output:\n\n    Score\tPopularity\tCategory\tTitle\tId\n    1.00\t376\tKids\tMasha und der Bär - E churzi Reis\t6b70748b-b89b-4bb9-9fec-aa6c63d291e6\n    1.00\t1755\tTalk\tBrian alias «Carlos» – Kommt er je frei?\tec3704b5-cc9b-47cf-9781-6026c61d6471\n    1.00\t323\tComedy\tStaffelfinale mit Kilian Ziegler (Staffel 14, Folge 10)\t0f4565dd-5a7f-41b9-9034-566d1e89c60c\n    1.00\t5608\tWissen \u0026 Ratgeber\tWie wir Behinderte behindern\t31f99f6e-d61c-46e1-b785-7d4c7ed9696b\n    1.00\t3604\tUnterhaltung\tAbschied mit Tränen\te01df7e2-e401-4334-84c4-6a19ed807bd4\n    1.00\t1147\tSerien\tVertrauen (Staffel 15, Folge 27)\t1b981517-5a19-4fdc-9d68-6f010eef2ff1\n    1.00\t1014\tGesellschaft\tNeuer TikTok-Star?: Sascha Ruefer geht mit WM-Kommentar viral\t529bf8ab-7955-477d-bee3-5b9d9e752dd0\n    1.00\t3402\tDok \u0026 Reportagen\tWir, die Pflegefachkräfte von morgen (Staffel 1, Folge 1)\t8bc3e03b-5088-4701-b860-f1726d9a0347\n    1.00\t2984\t\tGelingt die Einigung im Rentenstreit?\t9e8802ad-029e-4ce9-889a-ac3c9237a964\n    1.00\t11112\tSport\tAuf dem Lift mit «Gumpiball» Aline Danioth\tbd1deb25-2ddc-43bd-b863-26a7c3b4e29a\n    1.00\t11276\tNews\tTagesschau vom 25.11.2022: Hauptausgabe\t658b3c99-db7e-473b-bf17-717039cb98ac\n    1.00\t2724\tPolitik\tRentenstreit reloaded: Jetzt geht es um die zweite Säule\t5f40dc56-253f-4a30-9872-996edf9b6088\n    1.00\t1516\tKultur\tBlack Friday: Kaufrausch vs. Konsumverzicht\t86d70fa2-7951-4b34-a164-2bbff7d15edb\n    1.00\t3195\tFilm\tInga Lindström – Rasmus und Johanna\t8af0ae11-102e-4a2d-8ffb-18e5aaed0ed4\n    0.90\t10016\tSport\tOdermatt: «In Lake Louise gibt es nicht sehr viele Helfer»\tdf9dcb12-461e-4a91-918e-38704eeaaaa1\n    0.85\t977\tSerien\tHoffnungsschimmer (Staffel 15, Folge 28)\tb5b02651-d778-4596-98bf-1e69897b9517\n    0.80\t1412\tTalk\tmit Zarifa Ghafari\t169e301e-d3bd-4e45-9b54-abcfe5d2049d\n    ...\n\nHowever, it is questionable if really every category needs to be represented\nwith one top-hit at the start of the ranking. Thus, let's try a mixed approach.\n\n### Top 200 videos with clicks normalized using a mixed approach\n\nA straightforward way to refine the score is to include more factors in the\nequation. In addition to the score derived from normalizing clicks within a\ncategory, a second factor is derived from normalizing clicks over all items.\n\nTo refine the contribution of one or the other factor to the total score, they\ncan be raised to the power of some exponent.\n\nGenerate a list of top 200 videos (score with mixed approach):\n\nBy default this is using the following exponents:\n\n  * Exponent used to weight the score derived from clicks normalized over all items:\n    `expGeneral=1`\n  * Exponent used to weight the score derived from clicks normalized within each category:\n    `expCategory=2`\n\nChange the source code in order to adapt the exponents.\n\n    export SRG_OAUTH_TOKEN=xyz\n    go run cmd/topmixnorm200/main.go \u003e /tmp/topmixnorm-200.tsv\n\nSample output:\n\n    Score\tPopularity\tCategory\tTitle\tId\n    1.00\t11276\tNews\tTagesschau vom 25.11.2022: Hauptausgabe\t658b3c99-db7e-473b-bf17-717039cb98ac\n    0.99\t11112\tSport\tAuf dem Lift mit «Gumpiball» Aline Danioth\tbd1deb25-2ddc-43bd-b863-26a7c3b4e29a\n    0.72\t10016\tSport\tOdermatt: «In Lake Louise gibt es nicht sehr viele Helfer»\tdf9dcb12-461e-4a91-918e-38704eeaaaa1\n    0.50\t5608\tWissen \u0026 Ratgeber\tWie wir Behinderte behindern\t31f99f6e-d61c-46e1-b785-7d4c7ed9696b\n    0.32\t3604\tUnterhaltung\tAbschied mit Tränen\te01df7e2-e401-4334-84c4-6a19ed807bd4\n    0.30\t3402\tDok \u0026 Reportagen\tWir, die Pflegefachkräfte von morgen (Staffel 1, Folge 1)\t8bc3e03b-5088-4701-b860-f1726d9a0347\n    0.28\t3195\tFilm\tInga Lindström – Rasmus und Johanna\t8af0ae11-102e-4a2d-8ffb-18e5aaed0ed4\n    0.26\t2984\t\tGelingt die Einigung im Rentenstreit?\t9e8802ad-029e-4ce9-889a-ac3c9237a964\n    0.24\t2724\tPolitik\tRentenstreit reloaded: Jetzt geht es um die zweite Säule\t5f40dc56-253f-4a30-9872-996edf9b6088\n    0.16\t1755\tTalk\tBrian alias «Carlos» – Kommt er je frei?\tec3704b5-cc9b-47cf-9781-6026c61d6471\n    0.13\t1516\tKultur\tBlack Friday: Kaufrausch vs. Konsumverzicht\t86d70fa2-7951-4b34-a164-2bbff7d15edb\n    0.11\t2317\tFilm\tInga Lindström – Hochzeit in Hardingsholm\t123563f0-ec53-4c53-91f0-35cd19eee7ba\n    0.10\t1147\tSerien\tVertrauen (Staffel 15, Folge 27)\t1b981517-5a19-4fdc-9d68-6f010eef2ff1\n    0.10\t5179\tSport\tOdermatt: «Hat mich selber etwas überrascht»\tfd74ec5b-612e-4fb8-84ef-f0eb9a1ff9f5\n    0.09\t1014\tGesellschaft\tNeuer TikTok-Star?: Sascha Ruefer geht mit WM-Kommentar viral\t529bf8ab-7955-477d-bee3-5b9d9e752dd0\n    0.08\t1412\tTalk\tmit Zarifa Ghafari\t169e301e-d3bd-4e45-9b54-abcfe5d2049d\n    0.07\t1191\tKultur\tMythos Matter\t6702d57e-2f65-424d-8a0a-176a65b5e1f0\n    0.06\t977\tSerien\tHoffnungsschimmer (Staffel 15, Folge 28)\tb5b02651-d778-4596-98bf-1e69897b9517\n    0.06\t1983\tDok \u0026 Reportagen\tKatar – Perlen im Sand\tc00cafb8-8a9b-47cb-9053-c294bd2815cb\n    0.06\t4302\tNews\t10 vor 10 vom 25.11.2022\t8fb42def-a546-4c6a-b416-0d7d6e1bf959\n    0.05\t1644\tPolitik\tMänner ausgeschlossen, Millionen für die Papst-Garde, Transfrau im Männergefängnis\td20389af-8ee3-43a7-be9b-fd95ba74fb38\n    0.05\t1943\tUnterhaltung\tDarum sind die Lieder von Mani Matter zeitlos | Making-of | Neu aufgelegt: Mani Matter 2022\tdfb3389b-52a9-4467-92a3-87e471f7e68e\n    0.05\t1700\t\tOksana Mirinenko findet nur langsam zurück ins Leben\t3851ea9f-83de-40e7-8096-459ded72f1e9\n    0.04\t3735\tSport\tFeuz und Co. reut's, den Iltis freut's\ta745ab17-f042-4db1-b340-f7a717078124\n\n### Effects of applying the mixed approach on the ranking\n\nEvaluating the resultis it becomes clear that a higher value of `expCategory`\nleads to more content of distinct categories among the top ranked entris.\n\nResults for `expCategory=1`:\n\n![Screenshot of Spreadsheet with Results for `expCategory=1`](./img/ranking-effect-mixed-exp-category-1.png)\n\nResults for `expCategory=2`:\n\n![Screenshot of Spreadsheet with Results for `expCategory=2`](./img/ranking-effect-mixed-exp-category-2.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fznerol%2Frejoha22-recommender","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fznerol%2Frejoha22-recommender","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fznerol%2Frejoha22-recommender/lists"}