{"id":20040839,"url":"https://github.com/vadimkantorov/fasttsv","last_synced_at":"2026-05-09T20:02:45.091Z","repository":{"id":146466482,"uuid":"190565719","full_name":"vadimkantorov/fasttsv","owner":"vadimkantorov","description":"TSV parser for Python in pure vectorized NumPy code","archived":false,"fork":false,"pushed_at":"2019-06-15T12:25:01.000Z","size":25,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-12T19:21:59.702Z","etag":null,"topics":["numpy","parsing","simd","tsv","vectorized-code"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vadimkantorov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-06T10:53:02.000Z","updated_at":"2021-11-25T12:49:34.000Z","dependencies_parsed_at":"2023-07-17T16:34:52.318Z","dependency_job_id":null,"html_url":"https://github.com/vadimkantorov/fasttsv","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vadimkantorov%2Ffasttsv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vadimkantorov%2Ffasttsv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vadimkantorov%2Ffasttsv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vadimkantorov%2Ffasttsv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vadimkantorov","download_url":"https://codeload.github.com/vadimkantorov/fasttsv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241470364,"owners_count":19968041,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["numpy","parsing","simd","tsv","vectorized-code"],"created_at":"2024-11-13T10:44:10.099Z","updated_at":"2026-05-09T20:02:45.002Z","avatar_url":"https://github.com/vadimkantorov.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fasttsv\nTSV parser for Python in using only NumPy ops.\n\nThis is not production-ready code, just a primer on a branchless parsing technique using vectorized code.\n\n#### TODO\n- support strings\n- support negative integers and floats\n\n# Approach\n1. Read the whole file into a byte array in memory\n2. Find positions of tabs and decimal points\n3. Compute digit count for every field\n3. For the integer case, given the maximum number of digits in the file, precompute the parsed integers finishing on a given positions for all possible digit counts\n4. For every field, use the computed digit count to index into the precomputed parsed integers array\n5. Assemble values for the real-valued columns: the integral and remainder parts are neighboring parsed integers\n\n# Features, scope and limitations\n1. Supports integer, real (only decimal point notation) and utf-8 string columns (quotes not supported)\n2. Uses only NumPy methods, and can be extended to GPU using Google Jax or PyTorch. It can also run on Pyodide.\n\n# Further reading\nFor some truly fascinating vectorized parsing check out [simdjson](https://github.com/lemire/simdjson) and [csvmonkey](https://github.com/dw/csvmonkey).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvadimkantorov%2Ffasttsv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvadimkantorov%2Ffasttsv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvadimkantorov%2Ffasttsv/lists"}