{"id":20453072,"url":"https://github.com/duyndh98/mangaproject","last_synced_at":"2025-04-13T03:33:41.605Z","repository":{"id":116964198,"uuid":"113472918","full_name":"duyndh98/MangaProject","owner":"duyndh98","description":"A Manga Crawler and Viewer Project (truyentranh.net)","archived":false,"fork":false,"pushed_at":"2018-04-29T17:05:46.000Z","size":28875,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-26T20:49:33.962Z","etag":null,"topics":["html-css","python","web-crawler","web-scraping","web-view"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/duyndh98.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-07T16:18:51.000Z","updated_at":"2024-04-02T17:42:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"0bf6b087-0155-441f-833c-6e3c14025e80","html_url":"https://github.com/duyndh98/MangaProject","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyndh98%2FMangaProject","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyndh98%2FMangaProject/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyndh98%2FMangaProject/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyndh98%2FMangaProject/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/duyndh98","download_url":"https://codeload.github.com/duyndh98/MangaProject/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248659795,"owners_count":21141183,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["html-css","python","web-crawler","web-scraping","web-view"],"created_at":"2024-11-15T11:11:23.019Z","updated_at":"2025-04-13T03:33:41.599Z","avatar_url":"https://github.com/duyndh98.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Manga Reading\n\nProject Task: Manga Crawler and Manga Viewer\nThis is project, we will build a simple system for manga reading. The system contains two separated modules: a Manga Crawler and a Manga Viewer.\nProgramming languages requirements: Python. You can use TCP, UDP for communication\n\n## 1. Manga Crawler\nFor this module, you can choose one or many of the following sites to get data: truyentranh.net, or truyentranhtuan.com. You may suggest any other manga site.\nThe crawl module needs to get info of at least 100 manga\nFor each manga, the following need to be stored:\n-\tManga name\n-\tAuthor\n-\tDescription\n-\tCategories\n-\tLast update\n-\tChapter name and list of pages\nThe info must be stored in JSON file manga_info.json. A sample file is provided\n\n### Result\nSite to get data: truyentranh.net\n128 manga crawled and stored in manga_info.json\n\nInstructions:\n\nProgramming language: Python 3\n\nFirstly, I declare Spider class and create a spider to perform the crawl and extract structured data from the site. A spider should have behaviours such as initialize, crawl, gather_urls, …\n\nThe manga_info.json instruction:\n\n![](https://github.com/duyndh/MangaProject/blob/master/crawler.png)\n\nAfter trying and failing, I found the most effective method to crawl more than 100 manga from this site. \n\nThe crawlers can be divided into 3 modules: \n-\tPreprocessing: from http://truyentranh.net/danh-sach.tall.html, the spider gather manga urls in alphabetical order, with 5 manga urls whose title start with each letter.\n-\tManga crawling: the spider visits each manga url gathered in above step, gathers manga informations (name, thumbnail, author, description, categories, last update) and all the chapter urls on that page.\n-\tChapter crawling: the spider visits each chapter url gathered in above step, gather chapter name and all the contents (images) on that page.\n\nEach step splited into 3 step:\n-\tExtract: HTTP request, fetch the HTML (and resolve the domain).\n-\tTransform: take features out of the HTML (title, images, content…) + run NLP algorithms.\n-\tStore: save to the JSON.\n\n## 2. Manga Viewer\nBuild a simple webserver to feed data for web browser to read manga base on the info constructed in the above module\nThe webserver read info from manga_info.json and construct the html:\n-\t`/allmanga.html`: list of all manga, name, author and author of each manga are shown\n-\t`/manga_info.html?id=\u003cmanga_id\u003e`: show detailed info of a manga with manga_id include manga name, description, author, categories, chapter list….\n-\t`/chapter.html?id=\u003cchapter_id\u003e`: view all the page of a chapter with chapter_id\n\n![](https://github.com/duyndh/MangaProject/blob/master/viewer0.png)\n\n![](https://github.com/duyndh/MangaProject/blob/master/viewer1.png)\n\n![](https://github.com/duyndh/MangaProject/blob/master/viewer2.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduyndh98%2Fmangaproject","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fduyndh98%2Fmangaproject","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduyndh98%2Fmangaproject/lists"}