{"id":23072513,"url":"https://github.com/doches/childes","last_synced_at":"2025-04-03T10:43:48.011Z","repository":{"id":1383558,"uuid":"1338303","full_name":"doches/childes","owner":"doches","description":"Code for wrangling the CHILDES corpus into something useful","archived":false,"fork":false,"pushed_at":"2011-03-02T17:34:50.000Z","size":124,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-09T00:17:52.150Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://texasexpat.net/childes","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/doches.png","metadata":{"files":{"readme":"README.mdown","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-02-07T16:02:58.000Z","updated_at":"2017-01-20T00:42:41.000Z","dependencies_parsed_at":"2022-07-17T12:46:25.114Z","dependency_job_id":null,"html_url":"https://github.com/doches/childes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doches%2Fchildes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doches%2Fchildes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doches%2Fchildes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doches%2Fchildes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/doches","download_url":"https://codeload.github.com/doches/childes/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246989512,"owners_count":20865305,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-16T07:20:00.486Z","updated_at":"2025-04-03T10:43:47.996Z","avatar_url":"https://github.com/doches.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scripts for CHILDES\n\nIf you've ever tried to work with the [CHILDES][] corpus before, you know that dealing with it can be...difficult. It's\norganized into a tremendous number of flat text files, one per session; the file format used to encode the session transcript (not to\nmention the metadata!) is, to put it mildly, highly irregular. \n\nIn short, it's a mess.\n\nThis repository is a central place for various CHILDES-related tools, scripts, and hacks. The first of these are a set of scripts to wrangle the `.cha` format (the 'highly irregular' thing above) into sanity-inducing XML, adding in POS tags and dependency parses along the way.\n\n### Tasks\n\n#### Produce an XML version of CHILDES in xml/ (do this first!)\n\n    ruby tools/xmlize.rb Eng-USA/ \u0026\u0026 ruby tools/centralize_xml.rb Eng-USA/ xml/\n\n#### Get a list of nouns (and filter by WordNet)\n\n    ruby tools/extract_nouns.rb xml/ | ruby tools/filter_nouns.rb \u003e childes.nouns\n\n#### Build a basic clustering using WordNet hypernyms\n\n    cat childes.nouns | ruby tools/cluster_nouns.rb \u003e childes.wordnet_cluster\n\n### Tools\n\n## xml2document\n\nTakes an XML file, extracts all of the utterances, and prints them to \nstandard output for later processing.\nTakes an optional key to ignore (e.g. CHI)\n\n#### Usage\n\n    ruby tools/xml2document.rb path/to/childes.xml \u003cignore_key\u003e\n\n\n\n\n\n## cluster\\_wordnet\\_nouns\n\nReads a list of noun/count pairs from STDIN (like the output of\nextract_nouns or filter_nouns) and prints a yaml clustering based\non their first-order WordNet synsets. \nRequires [doches/rwordnet][].\n\n#### Usage\n\n    [cat noun.list] | ruby tools/cluster_wordnet_nouns.rb\n\n\n\n\n\n## corpusize\n\nTakes a path to a directory containing XML (as ouput by **cha2xml**),\nfinds all of the utterances therein, and writes a target corpus of\nall of them to standard out. Basically a wrapper around **xml2corpus**,\nand probably the sort of thing you're only interested in if you're also\nusing [doches/corncob][].\n\n#### Usage\n\n    ruby tools/corpusize.rb path/to/xml/root\n\n\n\n\n\n## centralize\\_xml\n\nReplicates the directory structure of *input* in *output*, copying\nonly xml files over into the new structure. Use this after **xmlize**, \nto build a version of CHILDES containing only XML. You don't have to \ndo this (other tools will silently ignore `.cha` files, preferring XML), \nbut it satisfies my housekeeping urges.\n\n#### Usage\n\n    ruby tools/centralize_xml.rb path/to/CHILDES/input path/to/xml/output\n\n\n\n\n\n## build\\_agemap\n\nTakes a directory containing CHILDES XML and prints a \nmapping (tab-delimited) of child age to filename.\n\n#### Usage\n\n    ruby tools/build_agemap.rb path/to/childes\n\n\n\n\n\n## xml2corpus\n\nTakes an XML file, extracts all of the utterances, and prints a TargetCorpus\n(see [doches/corncob][]) using nouns from the POStag list as target words.\nLike **cha2xml**, you probably want to call this automatically from some \nother script.\nTakes an optional key to ignore (e.g. CHI)\n\n#### Usage\n\n    ruby tools/xml2corpus.rb path/to/childes.xml \u003cignore_key\u003e\n\n\n\n\n\n## filter\\_corpus\n\nFilters a target corpus (from standard input) to include only lines involving target\nwords from a list, printing the result to standard out. Used to clean up the output \nof **xml2corpus** according to the output of **filter_nouns**.\n\n#### Usage\n\n    [cat file.target_corpus] | ruby tools/filter_corpus.rb path/to/nouns.filtered\n\n\n\n\n\n## cha2xml\n\nReads a [CHILDES][] .cha file from STDIN and outputs XML file containing cleaned\ndialog to STDOUT\n\n#### Usage\n\n    [cat thing.cha] | ruby cha2xml.rb \u003coptions\u003e\n\n\n#### Options\n\n**xmlize** calls cha2xml with *all* of these options on by default\n\n   + **--braces** Strip out experimenter annotations (\"foo [this is a note] bar\") from utterances.\n   + **--clean** Remove words containing nonsensical (i.e. non-word) characters.\n   + **--minipar** Run utterances through MINIPAR, including the result in the `\u003cparse\u003e` tag. Looks for `./vendor/pdemo/pdemo`, with data files in `./vendor/data`.\n   + **--tag** Run utterances through a [pure Ruby implementation of the Brill tagger][rbtagger], including the result in the `\u003ctags\u003e` tag.\n\n\n\n\n## agemap2documents\n\nReads an agemap from standard input and creates a set of nlda-friendly \ncorpora in \u003coutput\u003e, binned into six-month periods\n\n#### Usage\n\n    cat [agemap] | ruby tools/agemap2nldacorpora.rb path/to/output\n\n\n\n\n\n## compute\\_reading\\_levels\n\nCompute reading levels for each document in a directory, and produce\na data file ready for plotting with GnuPlot.\n\n#### Usage\n\n    ruby tools/compute_reading_levels.rb path/to/dir \u003e file.dat\n\n\n\n\n\n## xmlize\n\nScans a diretory for .cha files, converting any it finds into dialog XML\n\n#### Usage\n\n    ruby xmlize.rb \u003cpath/to/CHILDES/root\u003e\n\n\n\n\n\n## filter\\_nouns\n\nReads a list of noun/counts (as output by extract_nouns) from STDIN,\nfiltering the list to include only nouns appearing in WordNet (as nouns).\nRequires [doches/rwordnet][].\n\n\n#### Usage\n\n    [cat noun.txt] | ruby tools/filter_nouns.rb\n\n\n\n\n\n## agemap2corpora\n\nReads an agemap from standard input and creates a set of corpora in \u003coutput\u003e,\none per each six-month period\n\n#### Usage\n\n    cat [agemap] | ruby tools/agemap2corpora.rb path/to/output\n\n\n\n\n\n## reading\\_level\n\nCompute reading levels (e.g. [Coleman-Liau Index][], [Automated Readability Index][], [Flesh-Kincaid][] Readability Test) for a given target_corpus\n\n#### Input\n\nOne sentence per line.\n\n\n#### Usage\n\n    ruby tools/reading_level.rb path/to/file.target_corpus \u003coptions\u003e\n\n\n\n#### Output\n\nPrints a tab-delimited list of metric names as a comment (e.g. \"# coleman  ari words\"), followed\nby a tab-delimited list of computed metrics (e.g. \"4.3  4.1 6.0 132.8\")\n\n#### Options\n\n   + **--ari** [Automated Readability Index][]\n   + **--coleman** [Coleman-Liau Index][]\n   + **--fkre** [Flesh-Kincaid][] Readability Test\n   + **--words** Average number of words per sentence\n   + **--syllables** Average number of syllables per sentence\n   + **--characters** Average number of characters per sentence\n\n\n\n\n## extract\\_nouns\n\nLooks recursively in a directory for xml, scanning each file found\nfor nouns in the POS tag list and outputting a list of all nouns\nfound (plus counts)\n\n#### Usage\n\n    ruby tools/extract_nouns.rb path/to/xml/root\n\n\n\n\n\n  [doches/corncob]: http://github.com/doches/corncob\n  [doches/rwordnet]: http://github.com/doches/rwordnet\n  [rbtagger]: http://rubygems.org/gems/rbtagger\n  [CHILDES]: http://childes.psy.cmu.edu/\n  [Automated Readability Index]: http://en.wikipedia.org/wiki/Automated_Readability_Index\n  [Flesh-Kincaid]: http://en.wikipedia.org/wiki/Flesch-Kincaid_Readability_Test\n  [Coleman-Liau Index]: http://en.wikipedia.org/wiki/Coleman-Liau_Index\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoches%2Fchildes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoches%2Fchildes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoches%2Fchildes/lists"}