{"id":16499443,"url":"https://github.com/tpapp/largecolumns.jl","last_synced_at":"2025-07-29T22:13:43.592Z","repository":{"id":55588545,"uuid":"108100007","full_name":"tpapp/LargeColumns.jl","owner":"tpapp","description":"Handle large columns (vectors of equal length) with bits types in Julia using mmap.","archived":false,"fork":false,"pushed_at":"2020-12-20T09:15:59.000Z","size":28,"stargazers_count":4,"open_issues_count":4,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-28T16:20:11.785Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tpapp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-24T08:40:59.000Z","updated_at":"2021-10-10T03:50:40.000Z","dependencies_parsed_at":"2022-08-15T03:50:21.407Z","dependency_job_id":null,"html_url":"https://github.com/tpapp/LargeColumns.jl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tpapp%2FLargeColumns.jl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tpapp%2FLargeColumns.jl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tpapp%2FLargeColumns.jl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tpapp%2FLargeColumns.jl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tpapp","download_url":"https://codeload.github.com/tpapp/LargeColumns.jl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241405077,"owners_count":19957764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T14:52:54.167Z","updated_at":"2025-03-01T18:21:05.494Z","avatar_url":"https://github.com/tpapp.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LargeColumns\n\n[![Project Status: WIP – Initial development is in progress, but there has not yet been a stable, usable release suitable for the public.](http://www.repostatus.org/badges/latest/wip.svg)](http://www.repostatus.org/#wip)\n[![Build Status](https://travis-ci.org/tpapp/LargeColumns.jl.svg?branch=master)](https://travis-ci.org/tpapp/LargeColumns.jl)\n[![Coverage Status](https://coveralls.io/repos/github/tpapp/LargeColumns.jl/badge.svg?branch=master)](https://coveralls.io/github/tpapp/LargeColumns.jl?branch=master)\n[![codecov.io](http://codecov.io/github/tpapp/LargeColumns.jl/coverage.svg?branch=master)](http://codecov.io/github/tpapp/LargeColumns.jl?branch=master)\n\nManage large vectors of bits types in Julia. A thin wrapper for\nmmapped binary data, with a few sanity checks and convenience\nfunctions.\n\n## Specification\n\nFor each dataset, the columns (vectors of equal length) and metadata\nare stored in a directory like this:\n\n```\ndir/\n├── layout.jld2\n├── meta/\n│   └ ...\n├── 1.bin\n├── 2.bin\n├── ...\n├── ...\n└── ...\n```\n\nThe file `layout.jld2` specifies the number and types of columns (using\n[JLD2.jl](https://github.com/simonster/JLD2.jl), and the total number of\nelements. The `$i.bin` files contain the data for each column, which\ncan be [memory mapped](https://en.wikipedia.org/wiki/Memory-mapped_file).\n\nAdditional metadata can be saved as in files in the directory\n`meta`. This is ignored by this library; use the function `meta_path`\nto calculate paths relative to `dir/meta`.\n\n## Interfaces\n\nTwo interfaces are provided. Use `SinkColumns` for an *ex ante*\nunknown number of elements, written sequentially. This is useful for\ningesting data.\n\n`MmappedColumns` is useful when the number of records is known and\nfixed.\n\nTypes for the columns are specified as `Tuple`s. See the docstrings\nfor both interfaces and the unit tests for examples.\n\n# Acknowledgments\n\nWork on this library was supported by the Austrian National Bank\nJubiläumsfonds grant #17378.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftpapp%2Flargecolumns.jl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftpapp%2Flargecolumns.jl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftpapp%2Flargecolumns.jl/lists"}