{"id":15576434,"url":"https://github.com/njsmith/cctext","last_synced_at":"2026-07-15T17:06:30.421Z","repository":{"id":145901682,"uuid":"21014505","full_name":"njsmith/cctext","owner":"njsmith","description":null,"archived":false,"fork":false,"pushed_at":"2014-06-25T16:33:32.000Z","size":36020,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-19T13:11:10.948Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/njsmith.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-06-19T20:27:52.000Z","updated_at":"2014-06-25T15:34:15.000Z","dependencies_parsed_at":"2023-06-06T19:51:21.376Z","dependency_job_id":null,"html_url":"https://github.com/njsmith/cctext","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/njsmith/cctext","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fcctext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fcctext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fcctext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fcctext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/njsmith","download_url":"https://codeload.github.com/njsmith/cctext/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fcctext/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35513512,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-15T02:00:06.706Z","response_time":131,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T18:49:14.193Z","updated_at":"2026-07-15T17:06:30.396Z","avatar_url":"https://github.com/njsmith.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"cctext\n======\n\nThese are some work-in-progress scripts for extracting natural\nlanguage text from the Common Crawl corpus.\n\n\nBuilding and running\n--------------------\n\nThis is complicated, because we need to get access to the CLD2\nlanguage detection library we need to get java and c++ libraries to\nplay along with each other, and also because I have no idea what I'm\ndoing with java build systems::\n\nStep 0: get a recent version of gradle. As of 2014-06-25, Debian\nships gradle 1.5, which **will not** work. I'm using gradle **1.12**.\n\nThen::\n\n    # Get and build CLD2. You must use 'cld2' as the checkout directory,\n    # because that's where build.gradle expects to find it.\n    svn checkout http://cld2.googlecode.com/svn/trunk cld2\n    (cd cld2/internal \u0026\u0026 ./compile_libs.sh)\n\n    # Build the CLD2 C++ wrapper shim.\n    gradle CLD2wrapSharedLibrary\n\n    # Build the Java code, fetch dependencies, and create an\n    # invocation script:\n    gradle installApp\n\nNow you should be able to do::\n\n    ./run.sh\n\nwhich is a little script that sets up the rest of the stuff that the\nabove incantations didn't set up (``LD_LIBRARY_PATH``), and starts a\nJVM. (It uses rlwrap to get non-terrible command-line editing, so you\nmight want to install that too.)\n\nRight now we just drop into a Jython REPL from which you can play\naround with the actual code, because I haven't written a main()\nfunction yet.\n\n\nThe CLD2 wrapper\n----------------\n\nCLD2 is pretty amazing, but it's written in C++, so using it from Java\nis really painful. There are four pieces involved in making it work.\n\n1. ``src/main/java/org/vorpus/cctext/LanguageDetection.java`` defines the\n   high-level interface used by the main driver program.\n\n2. It works by calling the auto-generated Bridj wrapper in\n   ``lib/CLD2wrap.jar``. This jar file was created by running::\n\n      java -jar jnaerator-0.11-shaded.jar -library CLD2wrap -mode Jar -o lib -package org.vorpus.cld2wrap -f src/CLD2wrap/headers/cld2wrap.h\n\n   You'll have to re-run this if you ever modify ``cld2wrap.h``.\n\n3. The auto-generated wrapper does FFI calls into the hand-written C++\n   wrapper in ``src/CLD2wrap/``. This is necessary because CLD2's\n   native API involves bizarre and exotic C++ features\n   like... ``std::vector`` and ordinary function calls, which Bridj\n   apparently cannot handle directly.\n\n4. The hand-written C++ wrapper links against ``libcld2_full.so``, and\n   uses CLD's native API directly.\n\n\nA cry for help\n--------------\n\nThis is terrible. If you know how to make it not so terrible please help.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjsmith%2Fcctext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnjsmith%2Fcctext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjsmith%2Fcctext/lists"}