{"id":27957604,"url":"https://github.com/laion-ai/interesting-text-datasets","last_synced_at":"2026-01-24T16:06:23.121Z","repository":{"id":63235179,"uuid":"566252615","full_name":"LAION-AI/interesting-text-datasets","owner":"LAION-AI","description":null,"archived":false,"fork":false,"pushed_at":"2022-12-28T20:40:50.000Z","size":23,"stargazers_count":43,"open_issues_count":1,"forks_count":3,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-07T18:13:48.078Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LAION-AI.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-11-15T09:33:24.000Z","updated_at":"2025-02-23T08:56:38.000Z","dependencies_parsed_at":"2023-01-31T07:30:55.436Z","dependency_job_id":null,"html_url":"https://github.com/LAION-AI/interesting-text-datasets","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/LAION-AI/interesting-text-datasets","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LAION-AI%2Finteresting-text-datasets","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LAION-AI%2Finteresting-text-datasets/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LAION-AI%2Finteresting-text-datasets/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LAION-AI%2Finteresting-text-datasets/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LAION-AI","download_url":"https://codeload.github.com/LAION-AI/interesting-text-datasets/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LAION-AI%2Finteresting-text-datasets/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28730904,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-24T10:24:43.181Z","status":"ssl_error","status_checked_at":"2026-01-24T10:24:36.112Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T18:13:44.431Z","updated_at":"2026-01-24T16:06:23.103Z","avatar_url":"https://github.com/LAION-AI.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\nBooks \u0026 Documents:\n\n\nhttps://huggingface.co/datasets/the_pile_books3\n\nDescription:\nThis dataset is Shawn Presser's work and is part of EleutherAi/The Pile dataset.\n\nThis dataset contains all of bibliotik in plain .txt form, aka 197,000 books processed in exactly the same way as did for bookcorpusopen (a.k.a. books1). \n\nOn s3: Not yet.\n\nConverted to training format: not yet\n\n\n\nhttps://the-eye.eu/libraries.html\n\nDescription:\nlibgen \u0026 zlib\n\nOn s3: yes\nConverted to training format: not yet\n\n\nhttps://archive.org/details/fanfictiondotnet_repack\n\nhttps://archive.org/details/Fanfictiondotnet1011dump\n\nfanfiction.net ID 11M+ should get scraped\n\nDescription:\ndump of fanfiction.net\nMany short stories, books, ... \n\nOn s3: Yes\n\nConverted to training format: not yet\n\n\n\n\nhttps://the-eye.eu/public/Random/torrents/archiveorg_DjVuTXT_Part1.torrent\n\nDescription:\n16 M ebooks from IA\n\nOn s3: Not Yet\n\nConverted to training format: not yet\n\n\n\n\nhttps://the-eye.eu/public/Books/\nDescription:\n5+M ebooks from different domains\n\nOn s3: Not Yet\n\nConverted to training format: not yet\n\n\nall ebook torrents from piratebay: https://pirate-bays.net/search?q=ebooks\nDescription:\nmany differentr ebook torrents\n\nOn s3: Not Yet\n\nConverted to training format: not yet\n\n\n\nhttps://the-eye.eu/public/Site-Dumps/campdivision.com/camp/Text%20Files/Miscellaneous%20Texts/\n\nhttps://the-eye.eu/public/Site-Dumps/campdivision.com/camp/Text%20Files/PDF/\n\n\nDescription:\nmany TV captions / subtitles - need to be checked\n\nOn s3: Yes.\n\nConverted to training format: not yet\n\n\n\nhttps://huggingface.co/datasets/bookcorpusopen\n\nhttps://huggingface.co/datasets/demelin/moral_stories\n\nSubs:\nhttps://the-eye.eu/public/Random/archive.org_dumps/archive.org_tvarchive_CaptionProject_December1st2022.tar.zst\n\nDescription:\nmany TV captions / subtitles - need to be checked\n\nOn s3: Yes.\n\nConverted to training format: not yet\n\n\n\nLargescale Webtext:\n\nhttps://huggingface.co/datasets/oscar\n\nhttps://huggingface.co/datasets/mc4\n\nhttps://huggingface.co/datasets/the_pile\n\nhttps://huggingface.co/datasets/spanish_billion_words\n\nhttps://huggingface.co/datasets/arabic_billion_words\n\nhttps://huggingface.co/datasets/olm/wikipedia\n\nhttps://huggingface.co/datasets/cc100\n\n\nhttps://files.pushshift.io/reddit/comments/ \nhttps://arxiv.org/abs/2001.08435\n\nDescription:\nReddit comments dumps\n\nOn s3: Not yet\n\nConverted to training format: not yet\n\n\n\nhttps://the-eye.eu/public/social/twitter/\n\n\n\nCode:\nhttps://huggingface.co/datasets/bigcode/the-stack-dedup\n\nhttps://huggingface.co/datasets/code_search_net\n\nhttps://huggingface.co/datasets/codeparrot/github-code\n\nLaw:\nhttps://openreview.net/forum?id=3HCT3xfNm9r\nhttps://huggingface.co/datasets/pile-of-law/pile-of-law\n\n\n\nScientific papers:\n\nTranslation:\nhttps://huggingface.co/datasets/opus100\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaion-ai%2Finteresting-text-datasets","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flaion-ai%2Finteresting-text-datasets","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaion-ai%2Finteresting-text-datasets/lists"}