{"id":22357218,"url":"https://github.com/mdh266/wikimedia_challenge","last_synced_at":"2026-05-09T01:11:31.631Z","repository":{"id":93022939,"uuid":"84988907","full_name":"mdh266/Wikimedia_Challenge","owner":"mdh266","description":"Analyzing click-through rates from Wikimedia","archived":false,"fork":false,"pushed_at":"2019-02-24T23:25:19.000Z","size":18802,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-31T14:43:11.157Z","etag":null,"topics":["data-analysis","data-challenge","exploratory-data-analysis","matplotlib","pandas","python"],"latest_commit_sha":null,"homepage":"http://michael-harmon.com/blog/Wikimedia_Challenge.html","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mdh266.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-03-14T19:27:26.000Z","updated_at":"2024-12-11T22:59:27.000Z","dependencies_parsed_at":"2023-03-08T11:00:45.338Z","dependency_job_id":null,"html_url":"https://github.com/mdh266/Wikimedia_Challenge","commit_stats":{"total_commits":13,"total_committers":2,"mean_commits":6.5,"dds":0.07692307692307687,"last_synced_commit":"23ba78dded93a874454df5f68e78d9330c47f841"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdh266%2FWikimedia_Challenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdh266%2FWikimedia_Challenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdh266%2FWikimedia_Challenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdh266%2FWikimedia_Challenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mdh266","download_url":"https://codeload.github.com/mdh266/Wikimedia_Challenge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245659160,"owners_count":20651544,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","data-challenge","exploratory-data-analysis","matplotlib","pandas","python"],"created_at":"2024-12-04T14:13:36.862Z","updated_at":"2026-05-09T01:11:31.596Z","avatar_url":"https://github.com/mdh266.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wikimedia Data Challenge\n\n## Introduction\n\nThis Project was put out as data challenge by the \u003ca href=\"https://wikimediafoundation.org/wiki/Home\"\u003eWikimedia Foundation\u003c/a\u003e for a data analyst position. I thought it would be a great opportunity to brush up on some data analysis tools in Pandas and decided to complete the challenge.\n\n\nThe goal of the project was to analyze data from *event logging* (EL) to track a variety of performance and usage metrics to help the company make decisions. Specifically, they were interested in:\n\n- *clickthrough rate*: the proportion of search sessions where the user clicked on one of the results displayed\n- *zero results rate*: the proportion of searches that yielded 0 results\n\nEL uses JavaScript to asynchronously send messages (events) to their servers when the user has performed specific actions. \n\n\n## Data\n\nThe dataset comes from a [tracking schema](3) that the Wikimedia Foundation uses for assessing user satisfaction. Desktop users are randomly sampled to be anonymously tracked by this schema which uses a \"I'm alive\" pinging system that we can use to estimate how long our users stay on the pages they visit. The dataset contains just a little more than a week of EL data.\n\n| Column          | Value   | Description                                                                       |\n|:----------------|:--------|:----------------------------------------------------------------------------------|\n| uuid            | string  | Universally unique identifier (UUID) for backend event handling.                  |\n| timestamp       | integer | The date and time (UTC) of the event, formatted as YYYYMMDDhhmmss.                |\n| session_id      | string  | A unique ID identifying individual sessions.                                      |\n| group           | string  | A label (\"a\" or \"b\").                                     |\n| action          | string  | Identifies in which the event was created. See below.                             |\n| checkin         | integer | How many seconds the page has been open for.                                      |\n| page_id         | string  | A unique identifier for correlating page visits and check-ins.                    |\n| n_results       | integer | Number of hits returned to the user. Only shown for searchResultPage events.      |\n| result_position | integer | The position of the visited page's link on the search engine results page (SERP). |\n\nThe following are possible values for an event's action field:\n\n- **searchResultPage**: when a new search is performed and the user is shown a SERP.\n- **visitPage**: when the user clicks a link in the results.\n- **checkin**: when the user has remained on the page for a pre-specified amount of time.\n\n### Example Session\n\n|uuid                             |      timestamp|session_id       |group |action           | checkin|page_id          | n_results| result_position|\n|:--------------------------------|:--------------|:----------------|:-----|:----------------|-------:|:----------------|---------:|---------------:|\n|4f699f344515554a9371fe4ecb5b9ebc | 20160305195246|001e61b5477f5efc |b     |searchResultPage |      NA|1b341d0ab80eb77e |         7|              NA|\n|759d1dc9966353c2a36846a61125f286 | 20160305195302|001e61b5477f5efc |b     |visitPage        |      NA|5a6a1f75124cbf03 |        NA|               1|\n|77efd5a00a5053c4a713fbe5a48dbac4 | 20160305195312|001e61b5477f5efc |b     |checkin          |      10|5a6a1f75124cbf03 |        NA|               1|\n|42420284ad895ec4bcb1f000b949dd5e | 20160305195322|001e61b5477f5efc |b     |checkin          |      20|5a6a1f75124cbf03 |        NA|               1|\n|8ffd82c27a355a56882b5860993bd308 | 20160305195332|001e61b5477f5efc |b     |checkin          |      30|5a6a1f75124cbf03 |        NA|               1|\n|2988d11968b25b29add3a851bec2fe02 | 20160305195342|001e61b5477f5efc |b     |checkin          |      40|5a6a1f75124cbf03 |        NA|               1|\n\nThis user's search query returned 7 results, they clicked on the first result, and stayed on the page between 40 and 50 seconds. (The next check-in would have happened at 50s.)\n\n[1]: https://www.mediawiki.org/wiki/Wikimedia_Discovery\n[2]: https://wikimediafoundation.org/wiki/Home\n[3]: https://meta.wikimedia.org/wiki/Schema:TestSearchSatisfaction2\n\n\n## Objectives\n\nThe objectives of this project are to find answers to the following questions:\n\n1. What is their daily overall clickthrough rate? How does it vary between the groups?\n\n2. Which results do people tend to try first? How does it change day-to-day?\n\n3. What is their daily overall zero results rate? How does it vary between the groups?\n\n4. Let *session length* be approximately the time between the first event and the last event in a session. Choose a variable from the dataset and describe its relationship to session length. Visualize the relationship.\n\n\n\n## Requirements \n1. \u003ca href=\"https://www.python.org/\"\u003e Python\u003c/a\u003e (3.x\n2. \u003ca href=\"http://jupyter.org/\"\u003eJupyter Notebook\u003c/a\u003e\n3. \u003ca href=\"http://www.numpy.org/\"\u003eNumPy\u003c/a\u003e\n4. \u003ca href=\"http://matplotlib.org/\"\u003ematplotlib\u003c/a\u003e\n5. \u003ca href=\"http://pandas.pydata.org\"\u003ePandas\u003c/a\u003e\n\n\nTo install the requirements with pip (except for Python), type in the main directory:\n\n\u003ccode\u003e pip install -r requirements.txt \u003c/code\u003e\n\n\nOr you can install the dependencies and access the notebook using \u003ca href=\"https://www.docker.com/\"\u003eDocker\u003c/a\u003e by building the Docker image with the following:\n\n\n\tdocker built -t wikimedia .\n\nFollowed by running the command container:\n\n\tdocker run -p 8888:8888 -t wikimedia\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdh266%2Fwikimedia_challenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmdh266%2Fwikimedia_challenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdh266%2Fwikimedia_challenge/lists"}