{"id":13653826,"url":"https://github.com/dbeley/senscritiquescraper","last_synced_at":"2025-04-23T06:32:04.235Z","repository":{"id":40973365,"uuid":"192126896","full_name":"dbeley/senscritiquescraper","owner":"dbeley","description":"Python library to extract data from senscritique.com.","archived":true,"fork":false,"pushed_at":"2024-06-26T21:38:16.000Z","size":190,"stargazers_count":14,"open_issues_count":7,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-11-10T04:36:33.794Z","etag":null,"topics":["python","scraper","senscritique","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dbeley.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-15T21:52:51.000Z","updated_at":"2024-06-26T21:45:41.000Z","dependencies_parsed_at":"2024-01-14T14:29:33.438Z","dependency_job_id":"eada6768-5dbb-4931-8690-ba952b6227d7","html_url":"https://github.com/dbeley/senscritiquescraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbeley%2Fsenscritiquescraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbeley%2Fsenscritiquescraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbeley%2Fsenscritiquescraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbeley%2Fsenscritiquescraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dbeley","download_url":"https://codeload.github.com/dbeley/senscritiquescraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250385271,"owners_count":21421886,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","scraper","senscritique","web-scraping"],"created_at":"2024-08-02T02:01:18.503Z","updated_at":"2025-04-23T06:31:59.218Z","avatar_url":"https://github.com/dbeley.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# senscritiquescraper\n\n![Build Status](https://github.com/dbeley/senscritiquescraper/workflows/CI/badge.svg)\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/e95f1fcf5d2e47b480a3ef9c98ce1b1d)](https://www.codacy.com/app/dbeley/senscritiquescraper?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=dbeley/senscritiquescraper\u0026amp;utm_campaign=Badge_Grade)\n\n\u003e :warning: Senscritique closed their old website previously available at old.senscritique.com. It means **senscritiquescraper** won't properly work anymore.\n\nAPI to extract data from senscritique.com.\n\nExample of what can be extracted:\n- topchart: \u003chttps://old.senscritique.com/films/tops/top111\u003e\n- collection: \u003chttps://old.senscritique.com/34nUBqnQvCSkt/collection/all/all/all/all/all/all/all/all/all/page-1\u003e\n- survey: \u003chttps://old.senscritique.com/top/resultats/Les_meilleurs_films_de_2019/2301802\u003e\n- list_work: \u003chttps://old.senscritique.com/films/oeuvres\u003e\n- work: \u003chttps://old.senscritique.com/film/J_ai_perdu_mon_corps/39006031\u003e\n\nType of work:\n- film: \u003chttps://old.senscritique.com/films\u003e\n- serie: \u003chttps://old.senscritique.com/series\u003e\n- jeu: \u003chttps://old.senscritique.com/jeuxvideo\u003e\n- livre: \u003chttps://old.senscritique.com/livres\u003e\n- bd: \u003chttps://old.senscritique.com/bd\u003e\n- album: \u003chttps://old.senscritique.com/musique\u003e\n- morceau: individual tracks from an album\n\nThe \"list_work\" urls are difficult to find, here are the one for each genre where the subgenres can be accessed:\n- \u003chttps://old.senscritique.com/films/oeuvres\u003e\n- \u003chttps://old.senscritique.com/series/oeuvres\u003e\n- \u003chttps://old.senscritique.com/jeuxvideo/oeuvres\u003e\n- \u003chttps://old.senscritique.com/livres/oeuvres\u003e\n- \u003chttps://old.senscritique.com/bd/oeuvres\u003e\n- \u003chttps://old.senscritique.com/musique/oeuvres\u003e\n\nSome examples of the API usage are included in the examples folder:\n- `scr_get_collection.py`: Extract the collection of an user.\n- `scr_get_get_list_work.py`: Extract a list of work.\n- `scr_get_get_survey.py`: Extract the results of a survey.\n- `scr_get_get_topchart.py`: Extract the results of a topchart.\n- `scr_get_get_work_details.py`: Extract infos about a list of Senscritique urls.\n- `scr_get_get_work_urls.py`: From a list of search terms in a text file, extract the urls of the first result on Senscritique for each term.\n\n## Requirements\n\n- requests\n- beautifulsoup4\n- lxml\n- pandas\n\n## Installation\n\n### First method (installing the senscritiquescraper package)\n\n```bash\ngit clone https://github.com/dbeley/senscritiquescraper\ncd senscritiquescraper\npython setup.py install\npython examples/scr_get_work_urls.py -h\n```\n\n### Second method (installing with pipenv)\n\n```bash\ngit clone https://github.com/dbeley/senscritiquescraper\ncd senscritiquescraper\npipenv install '-e .'\npipenv run python examples/scr_get_work_urls.py -h\n```\n\n## Example\n\n```python\n\u003e\u003e\u003e import pandas as pd\n\u003e\u003e\u003e from senscritiquescraper import Senscritique\n```\n\n### Work/oeuvre\n\nExtract the URL of the first result for the query \"Star Wars\" with a filter on movies, then extract infos about it:\n\n```python\n\u003e\u003e\u003e # Possible genres are Films, Livres, Séries, BD, Jeux, Morceaux, Albums (setting the genre is optional).\n\u003e\u003e\u003e sw_url = Senscritique.get_url(\"Star Wars\", rank=1, genre=\"Films\")\n\u003e\u003e\u003e sw_infos = Senscritique.get_work_details(sw_url)\n\u003e\u003e\u003e import json\n\u003e\u003e\u003e json.dumps(sw_infos, indent=4, ensure_ascii=False)\n```\n\n```text\n{\n    \"Title\": \"La Guerre des Étoiles\",\n    \"URL\": \"https://www.senscritique.com/film/La_Guerre_des_Etoiles/365132\",\n    \"Rating\": \"7.8\",\n    \"Rating Details\": {\n        \"1\": 187,\n        \"2\": 143,\n        \"3\": 294,\n        \"4\": 629,\n        \"5\": 1940,\n        \"6\": 5722,\n        \"7\": 17573,\n        \"8\": 23914,\n        \"9\": 12138,\n        \"10\": 6868\n    },\n    \"Year\": \"1977\",\n    \"Cover URL\": \"https://media.senscritique.com/media/000012797534/160/La_Guerre_des_Etoiles.png\",\n    \"Review Count\": \"299\",\n    \"Vote Count\": \"69408\",\n    \"Favorite Count\": \"3800\",\n    \"Wishlist Count\": \"2500\",\n    \"In Progress Count\": null,\n    \"Description\": \"Dans une galaxie lointaine, l'univers est dominé par l'Empire galactique avec à sa tête le sombre et impitoyable Dark Vador. Mais la révolte gronde, l'Alliance rebelle menée par la princesse Leia, s'empare des plans de l'Etoile Noire, la base secrète de l'Empire. Avant d'être capturée par les...\",\n    \"Producer\": \"George Lucas\",\n    \"Genre\": \"Aventure, science-fiction et action\",\n    \"Length\": \"2 h 01 min\",\n    \"Release Date\": \"19 octobre 1977 (France)\",\n    \"Category\": \"Movie\"\n}\n```\n\n### User Collection\n\nExtract the user collection of the user 34nUBqnQvCSkt:\n\n```python\n\u003e\u003e\u003e user_collection = Senscritique.get_user_collection(\"34nUBqnQvCSkt\")\n\u003e\u003e\u003e df_user_collection = pd.DataFrame(user_collection)\n\u003e\u003e\u003e df_user_collection[[\"Title\", \"Year\", \"Category\", \"User Rating\", \"Number of Ratings\", \"Average Rating\"]]\n```\n\n```text\n                                      Title  Year    Category User Rating Number of Ratings Average Rating\n0                                   Walk On  1974       Track          10                54            7.5\n1                           Symphonie no. 9  1984       Music          10              1068            9.1\n2                          Berserk, tome 13  1997      Comics          10               922            9.1\n3  The Legend of Zelda : Breath of the Wild  2017  Video Game          10             11728            8.8\n4                                Sur écoute  2002      Series          10             21979            9.1\n5                       12 hommes en colère  1957       Movie          10             45414            8.7\n6                                      1984  1949        Book          10             81697            8.4\n```\n\n### Top Chart\n\nExtract infos about the Top 111 Musique topchart:\n\n```python\n\u003e\u003e\u003e topchart = Senscritique.get_topchart(\"https://www.senscritique.com/musique/tops/top111\")\n\u003e\u003e\u003e df_topchart = pd.DataFrame(topchart)\n\u003e\u003e\u003e df_topchart[['Title', 'Year', 'Average Rating', 'Number of Ratings']]\n```\n```text\n                                         Title  Year Average Rating Number of Ratings\n0    Dvořák: Symphonie No. 9 / Smetana: Moldau  1985            9.1               428\n1                                      Requiem  1971            9.1              1548\n2                              Symphonie no. 9  1984            9.1              1068\n3                                      Requiem  2011            8.9              2353\n4                                    Nocturnes  1992            8.9              1293\n..                                         ...   ...            ...               ...\n106                                Long Season  1996            8.2              1430\n107                                   Murmuüre  2010            8.2               800\n108                         Saxophone Colossus  1956            8.2               555\n109                                 Solo Piano  1989            8.2               452\n110                        The Main Ingredient  1994            8.2               320\n```\n\n### Work List\n\nExtract infos about all the RPG video games listed on Senscritique:\n\n```python\n\u003e\u003e\u003e list_work = Senscritique.get_list_work(\"https://www.senscritique.com/jeuxvideo/oeuvres/RPG--3094\")\n\u003e\u003e\u003e df_list_work = pd.DataFrame(list_work)\n\u003e\u003e\u003e df_list_work[[\"Title\", \"Year\", \"Genre\", \"Average Rating\"]]\n```\n```text\n                                     Title  Year                                              Genre Average Rating\n0                        Madô Monogatari I  None                                                                 -\n1                           Arc The Lad II  2012                                                RPG            7.1\n2                        Kingdom Hearts II  2006                                      RPG et action            8.1\n3     Might and Magic III : Isles of Terra  1991                                                RPG            7.8\n4                          Siege Of Avalon  2000                                                RPG              -\n...                                    ...   ...                                                ...            ...\n6675                          Silver Ghost  1988                                                                 -\n6676                          Memetown USA  2018                            Aventure et jeu de rôle              -\n6677                        Lords of Chaos  1990  Stratégie tour par tour, tactique, wargame et ...              -\n6678                       Marginalia Hero  2019                                        Jeu de rôle              -\n6679                    The Eternal Dagger  1987                                                                 -\n```\n\n### Survey\n\nExtract infos about the best movies of 2021 survey:\n\n```python\n\u003e\u003e\u003e survey = Senscritique.get_survey(\"https://www.senscritique.com/top/resultats/Les_meilleurs_films_de_2021/2917616\")\n\u003e\u003e\u003e df_survey = pd.DataFrame(survey)\n\u003e\u003e\u003e df_survey[[\"Title\", \"Year\", \"Producer\"]]\n```\n```text\n                          Title  Year                                     Producer\n0               Malcolm \u0026 Marie  2021                                 Sam Levinson\n1  Zack Snyder's Justice League  2021                                  Zack Snyder\n2             Pieces of a Woman  2021                             Kornél Mundruczó\n3                  I Care a Lot  2021                                   J Blakeson\n4                       The Dig  2021                                  Simon Stone\n5                    La Mission  2020                              Paul Greengrass\n6         Promising Young Woman  2020                              Emerald Fennell\n7     Raya et le dernier dragon  2021  Don Hall, Carlos López Estrada, Paul Briggs\n8                     Nomadland  2020                                   Chloé Zhao\n9                        Minari  2021                              Lee Isaac Chung\n```\n\n### Example Scripts\n\nSome scripts using the API are provided in the examples folder.\n\nFor each scripts, you can see the help and the available options with the `-h` argument.\n\n```bash\npython scr_get_topchart.py -h\n```\n\n#### scr_get_topchart\n\n```text\nusage: scr_get_topchart.py [-h] [--debug] [-u URL] [main_argument]\n\nSenscritique scraper for a top list/chart.\n\npositional arguments:\n  main_argument      URL to parse\n\noptional arguments:\n  -h, --help         show this help message and exit\n  --debug            Display debugging information\n  -u URL, --url URL  URL to parse (same as without argument)\n```\n\n#### scr_get_collection\n\n```bash\npython scr_get_collection.py -h\n```\n\n```text\nusage: scr_get_collection.py [-h] [--debug] [-u USER] [main_argument]\n\nSenscritique scraper for an user collection.\n\npositional arguments:\n  main_argument         Name of the user\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --debug               Display debugging information\n  -u USER, --user USER  Name of the user (same as without argument)\n```\n\n#### scr_get_survey\n\n```bash\npython scr_get_survey.py -h\n```\n\n```text\nusage: scr_get_survey.py [-h] [--debug] [-u URL] [main_argument]\n\nSenscritique scraper for a survey.\n\npositional arguments:\n  main_argument      URL to parse\n\noptional arguments:\n  -h, --help         show this help message and exit\n  --debug            Display debugging information\n  -u URL, --url URL  URL to parse (same as without argument)\n```\n\n#### scr_get_list_work\n\n```bash\npython scr_get_list_work.py -h\n```\n\n```text\nusage: scr_get_list_work.py [-h] [--debug] [-u URL] [main_argument]\n\nSenscritique scraper for a list_work.\n\npositional arguments:\n  main_argument      URL to parse.\n\noptional arguments:\n  -h, --help         show this help message and exit\n  --debug            Display debugging information\n  -u URL, --url URL  URL to parse (same as without argument)\n```\n\n#### scr_get_work_details\n\n```bash\npython scr_get_work_details.py -h\n```\n\n```text\nusage: scr_get_work_details.py [-h] [--debug] [-f FILE] [main_argument]\n\nThis script will export in a csv file one or several works from senscritique contained in a csv file (it will use the 'URL' field).\n\npositional arguments:\n  main_argument         File to parse.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --debug               Display debugging information\n  -f FILE, --file FILE  File to parse.\n```\n\n#### scr_get_work_urls\n\nRunning `scr_get_work_url -f FILE` will extract the first search result on Senscritique for each line of a text file and export it in a new file.\n\nIt's useful to be used with the `scr_get_work_details.py` script.\n\n```text\nusage: scr_get_work_urls.py [-h] [--debug] [-f FILE] [-g GENRE] [main_argument]\n\nThis script will export in a file the URLs for the first Senscritique search result of each search terms contained in another file.\n\npositional arguments:\n  main_argument         File to parse.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --debug               Display debugging information\n  -f FILE, --file FILE  File to parse.\n  -g GENRE, --genre GENRE\n                        Genre to filter (available choices : Morceaux, Albums, Films, Livres, Séries, BD, Jeux).\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdbeley%2Fsenscritiquescraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdbeley%2Fsenscritiquescraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdbeley%2Fsenscritiquescraper/lists"}