{"id":15520723,"url":"https://github.com/proycon/ssam","last_synced_at":"2025-04-23T04:43:28.846Z","repository":{"id":57668410,"uuid":"292930307","full_name":"proycon/ssam","owner":"proycon","description":"split sampler: split your data into multiple sets (e.g. train/test/development)","archived":false,"fork":false,"pushed_at":"2021-03-03T13:11:18.000Z","size":29,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-22T10:21:16.733Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proycon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-04T19:27:17.000Z","updated_at":"2023-03-19T02:27:07.000Z","dependencies_parsed_at":"2022-09-07T15:50:43.051Z","dependency_job_id":null,"html_url":"https://github.com/proycon/ssam","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fssam","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fssam/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fssam/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fssam/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proycon","download_url":"https://codeload.github.com/proycon/ssam/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250372920,"owners_count":21419722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T10:29:04.746Z","updated_at":"2025-04-23T04:43:28.825Z","avatar_url":"https://github.com/proycon.png","language":"Rust","funding_links":[],"categories":["\u003ca name=\"text-processing\"\u003e\u003c/a\u003eText processing"],"sub_categories":[],"readme":"[![Crate](https://img.shields.io/crates/v/ssam.svg)](https://crates.io/crates/ssam)\n\n# Ssam\n\nSsam, short for split sampler, splits one or more text-based input files into multiple\nsets using random sampling. This is useful for splitting data into a training, test and\ndevelopment sets, or whatever sets you desire.\n\n## Features\n\n* Split input into multiple sets, user can specify the size of each set in either absolute numbers or a relative fraction.\n* Supports both sampling without replacement (default) or with replacement.\n* Defaults to line-based sampling, but a custom delimiter can be configured to sample larger blocks of variable size.\n* Can handle multiple input files that will be considered **dependent**. Useful for splitting and sampling for instance parallel corpora.\n* By default ordering is preserved, use ``--shuffle`` for more randomness.\n* Specify a seed for the random number generator to create **reproducible samples**.\n\n## Installation\n\nInstall it using Rust's package manager:\n\n```\ncargo install ssam\n```\n\nNo cargo/rust on your system yet? Do ``sudo apt install cargo`` on Debian/ubuntu based systems, ``brew install rust`` on mac, or use [rustup](https://rustup.rs/).\n\n## Usage\n\nSee ``ssam --help`` for extensive usage information.\n\nSuppose you have a text file ``sentences.txt`` with one sentence per line, and you want to sample the sentences into a test, development and\ntrain set using respectively 10% (`0.1`), 10%  (`0.1`) and the remainder (`*`) of the sentences:\n\n```\n$ ssam --sizes \"0.1,0.1,*\" --names \"test,dev,train\" sentences.txt\n```\n\nThis will output three files: `sentences.train.txt`, `sentences.test.txt` and `sentences.dev.txt`. If you don't specify\nany names explicitly the infix will simply be ``set1``,``set2``,``set3``, etc..\n\nSuppose you have the same sentences in German in a file called `sätze.txt` and the sentences are aligned up nicely with\nthe ones in `sentences.txt` (i.e. the same line numbers correspond and contain translations). You can now make a\ndependent split as follows:\n\n```\n$ ssam --shuffle --sizes \"0.1,0.1,*\" --names \"test,dev,train\" sentences.txt sätze.txt\n```\n\nThe sentences will still correspond in each of the output sets. We also added ``--shuffle`` for more randomness in the\noutput order, as by default ssam preserves order.\n\nSsam can also read from stdin (provided you want to supply only one input document). If you're only doing one sample\n(rather than three as shown above), then it will simply output to stdout.\n\nRather than using lines as units, you can specify a delimiter manually. For example, set ``--delimiter \"\"`` (empty\ndelimiter) if you want empty lines to be the delimiter, such as for instance those often used to separate paragraphs.\nAlternative you can set it to an explicit marker in your input, like ``--delimiter \"\u003cutt\u003e\"`` for example.\n\n## In Memoriam\n\nIn loving memory of our cat [Sam](https://proycon.anaproy.nl/img/photos/2020-01-04:-In-Memoriam-2009-2019.jpg), 2009-2019.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Fssam","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproycon%2Fssam","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Fssam/lists"}