{"id":15576403,"url":"https://github.com/njsmith/zscontrib","last_synced_at":"2026-04-29T03:31:45.563Z","repository":{"id":16252989,"uuid":"19000916","full_name":"njsmith/zscontrib","owner":"njsmith","description":"Extra misc ZS-related stuff that doesn't belong in the zs repository","archived":false,"fork":false,"pushed_at":"2014-05-14T14:18:18.000Z","size":200,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-12T16:44:43.571Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/njsmith.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-04-21T17:44:59.000Z","updated_at":"2019-04-21T11:32:46.000Z","dependencies_parsed_at":"2022-09-10T15:01:09.748Z","dependency_job_id":null,"html_url":"https://github.com/njsmith/zscontrib","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/njsmith/zscontrib","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fzscontrib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fzscontrib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fzscontrib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fzscontrib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/njsmith","download_url":"https://codeload.github.com/njsmith/zscontrib/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fzscontrib/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32409082,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T02:37:21.628Z","status":"ssl_error","status_checked_at":"2026-04-29T02:36:50.947Z","response_time":110,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T18:48:42.441Z","updated_at":"2026-04-29T03:31:45.548Z","avatar_url":"https://github.com/njsmith.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"zscontrib\n=========\n\nExtra misc ZS-related stuff that doesn't belong in the zs repository\n\nZS is here: https://github.com/njsmith/zs\n\nThese scripts are much hackier than the stuff in the zs repository\nproper. They certainly assume UNIX, and almost certainly will only\nwork under Python 2.7, not anything earlier or later.\n\n\nHow to build .zs files from the Google Books v2 files\n-----------------------------------------------------\n\nRequirements: curl, python 2, gnu make, gnu sort\n\nNice to have: lzop, `pv \u003chttp://www.ivarch.com/programs/pv.shtml\u003e`_\n\nInstall this package::\n\n    pip install git+https://github.com/njsmith/zscontrib.git\n\nThen generate a makefile for the subcorpus you want::\n\n    python -m zscontrib.gbooks2 make-makefile LANGUAGE\n\nwhere LANGUAGE is \"eng-us-all\" or whatever. This only takes a few\nseconds, and creates a makefile called::\n\n    google-books-LANGUAGE-20120701.make\n\nThis Makefile has targets like ``sorted-3gram``, which download and\nsorts all the individual 3gram files, ``size-check-3gram`` which\ndouble-checks that the downloaded files are similar in size to the\noriginal ones (to make sure you didn't have a download failure that\nwill end up corrupting your .zs file!), and\n``google-books-eng-us-all-20120701-3gram.zs``, which depends on those\nfirst two, and creates the actual .zs file.\n\nYou probably want to start by doing something like::\n\n  make -f google-books-LANGUAGE-20120701.make -j \u003csomething\u003e sorted-3gram\n\nwhere \u003csomething\u003e can be determined empirically by trying a few values\nand observing which ones crash your computer/cluster. (Each parallel\nmake will spawn a curl | gunzip | sort | gzip pipeline, and sort is\nmultithreaded, so you can get a ton of CPU/disk/network usage going\nthis way.)\n\nLater, once you have the sorted files, you can run::\n\n  make -f google-books-LANGUAGE-20120701.make google-books-LANGUAGE-20120701-3gram.zs\n\nBy default ``zs make`` will try to use all your CPUs, so it's probably\nnot a great idea to run this command at the same time as you're\nrunning a big ``make -j`` on the same box.\n\nNote that we use \"0gram\" to refer to what Google calls \"totalcounts\",\nand \"dependency\" to refer to what Google calls \"0gram\". This is\nbecause Google's terminology is bad and ours is better. We also\nnormalize the totalcounts file into the same format as the other\nn-grams, with an empty string in the \"ngram\" column.\n\nIf you have multiple machines then you can work on the 1grams on one,\nthe 2grams on another, etc.\n\nThere are some variables at the top of the makefile that you might\nwell want to edit or otherwise override:\n\n* ``ZS_OPTS`` are options to be passed to ``zs make``. You can\n  pass --codec or -j here, for example.\n\n* ``TMPDIR`` should probably point to a directory on *fast, local* disk,\n  since it's where ``sort`` will scribble temporary files while\n  working. Make sure it's big enough to handle the data that will be\n  scribbled into it. (Basically this will be the size of a single\n  Google-distributed ngram file, when compressed with lzop or whatever\n  (see next item), times the number of parallel jobs you have running\n  at once.)\n\n* ``TMP_COMPRESS`` is the program that ``sort`` will use to\n  compress/decompress these temporary files. ``lzop`` is a good\n  choice. If you don't have lzop then you might want to make a little\n  script like::\n\n      #!/bin/sh\n      exec gzip -1 \"$@\"\n\n  and then point to that script here. Or if you have tons of space or\n  are building a corpus for a small language then you can disable\n  temporary compression altogether by editing the definition of\n  ``SORT``.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjsmith%2Fzscontrib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnjsmith%2Fzscontrib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjsmith%2Fzscontrib/lists"}