{"id":23714392,"url":"https://github.com/gpakosz/packedarray","last_synced_at":"2025-09-03T18:32:07.737Z","repository":{"id":1554380,"uuid":"11863240","full_name":"gpakosz/PackedArray","owner":"gpakosz","description":"Random access array of tightly packed unsigned integers","archived":false,"fork":false,"pushed_at":"2022-06-11T09:35:56.000Z","size":240,"stargazers_count":153,"open_issues_count":7,"forks_count":30,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-07-11T09:46:47.291Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gpakosz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-08-03T13:08:07.000Z","updated_at":"2024-05-29T03:13:36.000Z","dependencies_parsed_at":"2022-08-06T10:16:38.422Z","dependency_job_id":null,"html_url":"https://github.com/gpakosz/PackedArray","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpakosz%2FPackedArray","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpakosz%2FPackedArray/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpakosz%2FPackedArray/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpakosz%2FPackedArray/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gpakosz","download_url":"https://codeload.github.com/gpakosz/PackedArray/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231908215,"owners_count":18444268,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-30T20:19:25.042Z","updated_at":"2024-12-30T20:19:25.465Z","avatar_url":"https://github.com/gpakosz.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PackedArray: random access array of tightly packed unsigned integers\n[![Build Status](https://travis-ci.org/gpakosz/PackedArray.png?branch=master)](https://travis-ci.org/gpakosz/PackedArray)\n## TLDR\n\n*PackedArray comes to the rescue when you're in a desperate need for an uint9_t\nor uint17_t array.*\n\n## What?\n\nWhen you want to hold an unordered sequence of unsigned integers into memory,\nthe C programming language lets you choose among 4 data types:\n\n- `uint8_t`\n- `uint16_t`\n- `uint32_t`\n- `uint64_t`\n\nIf your numbers are within the [0, 100000] range, only 17 bits per integer are\nneeded since 2\u003csup\u003e17\u003c/sup\u003e = 131072. However, you can't use an array of\n`uint16_t` because 16 bits are not enough to store numbers between 65536 and\n100000. When you use the next available type, `uint32_t`, you're wasting 15 bits\nper integer which represents a 47% overhead in terms of storage requirements.\n                                                         \n`PackedArray` saves memory by packing integers/items together at the bit-level:\n\n\u003ctable class=\"monospace\"\u003e\n  \u003ctr\u003e\n    \u003ctd colspan=\"32\"\u003eb0\u003c/td\u003e\n    \u003ctd colspan=\"32\"\u003eb1\u003c/td\u003e\n    \u003ctd colspan=\"32\"\u003eb2\u003c/td\u003e\n    \u003ctd style=\"border-style: dashed; border-right: none;\"\u003e...\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd colspan=\"9\"\u003ei0\u003c/td\u003e\n    \u003ctd colspan=\"9\"\u003ei1\u003c/td\u003e\n    \u003ctd colspan=\"9\"\u003ei2\u003c/td\u003e\n    \u003ctd colspan=\"9\"\u003ei3\u003c/td\u003e\n    \u003ctd colspan=\"9\"\u003ei4\u003c/td\u003e\n    \u003ctd colspan=\"9\"\u003ei5\u003c/td\u003e\n    \u003ctd colspan=\"9\"\u003ei6\u003c/td\u003e\n    \u003ctd colspan=\"9\"\u003ei7\u003c/td\u003e\n    \u003ctd colspan=\"9\"\u003ei8\u003c/td\u003e\n    \u003ctd colspan=\"9\"\u003ei9\u003c/td\u003e\n    \u003ctd style=\"border-style: dashed; border-right: none;\"\u003e...\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\nA `PackedArray` is backed by an `uint32_t` buffer. Several items end up being\nstored inside the same buffer cell, e.g. i0, i1, and i2. Some items span two\nbuffer cells, e.g. i3, and i7. `PackedArray` is responsible for\nencoding/decoding items into/from the storage buffer.\n\n`PackedArraySIMD` is a `PackedArray` variant that makes use of SSE2 or NEON\ninstructions.\n\nGoing SIMD processes integers 4 by 4 but imposes an interleaved layout in the\nstorage buffer.\n \n`PackedArraySIMD` interleaved layout, 13 bits per item:\n\n\u003ctable class=\"monospace\"\u003e\n  \u003ctr\u003e\n    \u003ctd colspan=\"32\"\u003eb0\u003c/td\u003e\n    \u003ctd colspan=\"32\"\u003eb1\u003c/td\u003e\n    \u003ctd colspan=\"32\"\u003eb2\u003c/td\u003e\n    \u003ctd colspan=\"32\"\u003eb3\u003c/td\u003e\n    \u003ctd style=\"border-style: dashed; border-right: none;\"\u003e...\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd colspan=\"13\"\u003ei0\u003c/td\u003e\n    \u003ctd colspan=\"13\"\u003ei4\u003c/td\u003e\n    \u003ctd colspan=\"6\"\u003ei8a\u003c/td\u003e\n    \u003ctd colspan=\"13\"\u003ei1\u003c/td\u003e\n    \u003ctd colspan=\"13\"\u003ei5\u003c/td\u003e\n    \u003ctd colspan=\"6\"\u003ei9a\u003c/td\u003e\n    \u003ctd colspan=\"13\"\u003ei2\u003c/td\u003e\n    \u003ctd colspan=\"13\"\u003ei6\u003c/td\u003e\n    \u003ctd colspan=\"6\"\u003ei10a\u003c/td\u003e\n    \u003ctd colspan=\"13\"\u003ei3\u003c/td\u003e\n    \u003ctd colspan=\"13\"\u003ei7\u003c/td\u003e\n    \u003ctd colspan=\"6\"\u003ei11a\u003c/td\u003e\n    \u003ctd style=\"border-style: dashed;\"\u003ei8b\u003c/td\u003e\n    \u003ctd style=\"border-style: dashed; border-right: none;\"\u003e...\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\nAs a consequence, the data layout of `PackedArraySIMD` isn't compatible with its\nnon SIMD counterpart. In other words, you cannot use `PackedArray` to unpack\ndata packed with `PackedArraySIMD` or the other way around.\n\nIt is also worth noting the implementations of `PackedArraySIMD_pack` and\n`PackedArraySIMD_unpack` require more plumbing than their non-SIMD counterparts.\nAdditional computations are needed to find out and adjust a data window that can\nbe processed 4 by 4 with SIMD instructions.\n\n`PackedArray` and `PackedArraySIMD` are released under the WTFPL v2 license.\n\nFor more information, see the [PackedArray announcement on my personal website].\n\n[PackedArray announcement on my personal website]: http://pempek.net/articles/2013/08/03/packedarray-random-access-array-tightly-packed-unsigned-integers/\n\n## Why?\n\n`PackedArray` is designed as a drop-in replacement for an unsigned integer\narray. I couldn't find such a data structure in the wild, so I implemented one.\n\nInstead of writing:\n\n    uint32_t* a = (uint32_t*)malloc(sizeof(uint32_t) * count);\n    ...\n    value = a[i];\n    ...\n    a[j] = value;\n\nYou write:\n\n    PackedArray* a = PackedArray_create(bitsPerItem, count);\n    ...\n    value = PackedArray_get(a, i);\n    ...\n    PackedArray_set(a, j, value);\n\nThe `PackedArray_computeBitsPerItem` helper scans a `uint32_t` array and returns\nthe number of bits needed to create a `PackedArray` capable of holding its\ncontent.\n\nThere are also `PackedArray_pack` and `PackedArray_unpack` that operate on\nseveral items in a row. Those two could really have been named\n`PackedArray_write` and `PackedArray_read` but I decided \"pack\" / \"unpack\"\nconveys better something is happening under the hood.\n\n    // bulk packing / unpacking\n    PackedArray_pack(a, j, in, count);\n    PackedArray_unpack(a, j, out, count);\n\n    // the following are semantically equivalent\n    PackedArray_set(a, j, value);\n    PackedArray_pack(a, j, \u0026value, 1);\n\n    value = PackedArray_get(a, i);\n    PackedArray_unpack(a, i, \u0026value, 1);\n\n--------------------------------------------------------------------------------\n\n## Compiling\n\nIn order to use `PackedArray` or `PackedArraySIMD` in your own project, you just\nhave to bring in the two `PackedArray.h` and `PackedArray.c` (or\n`PackedArraySIMD.c`) files. It's that simple.\n\nYou can customize `PackedArray.c`'s behavior by defining the following macros:\n\n- `PACKEDARRAY_ASSERT`\n- `PACKEDARRAY_MALLOC`\n- `PACKEDARARY_FREE`\n\nYou can customize `PackedArraySIMD.c`'s behavior by defining the following\nmacros:\n\n- `PACKEDARRAY_ASSERT`\n- `PACKEDARRAY_ALIGNED_MALLOC`\n- `PACKEDARARY_FREE`\n\n`PackedArray.c` and `PackedArraySIMD.c` can compile themselves into either a\ntest program or a micro-benchmark. For that, you have to use one of the\nfollowing preprocessor directives:\n\n- `PACKEDARRAY_SELF_TEST`\n- `PACKEDARRAY_SELF_BENCH`\n\nFor example, from command line:\n\n    $ cc -o PackedArraySelfTest -DPACKEDARRAY_SELF_TEST -O2 -g PackedArray.c\n    $ cc -o PackedArraySelfBench -DPACKEDARRAY_SELF_BENCH -DNDEBUG -O2 -g PackedArray.c\n\n    $ cc -o PackedArraySIMDSelfTest -DPACKEDARRAY_SELF_TEST -O2 -g PackedArraySIMD.c\n    $ cc -o PackedArraySIMDSelfBench -DPACKEDARRAY_SELF_BENCH -DNDEBUG -O2 -g PackedArraySIMD.c\n\n### Compiling for Windows\n\nThere is a Visual Studio 2012 solution in the `_win-vs11/` folder.\n\n### Compiling for Linux or Mac\n\nThere is a GNU Make 3.81 `MakeFile` in the `_gnu-make/` folder:\n\n    $ make -C _gnu-make/\n\n### Compiling for Mac\n\nSee above if you want to compile from command line. Otherwise there is an Xcode\nproject located in the `_mac-xcode/` folder.\n\n### Compiling for iOS\n\nThere is an Xcode project located in the `_ios-xcode/` folder.\n\nIf you prefer compiling from command line and deploying to a jailbroken device\nthrough SSH, use:\n\n    $ make -C _gnu-make/ binsubdir=ios CC=\"$(xcrun --sdk iphoneos --find clang) -isysroot $(xcrun --sdk iphoneos --show-sdk-path) -arch armv7 -arch armv7s -arch arm64\" postbuild=\"codesign -s 'iPhone Developer'\"\n\n### Compiling for Android\n\nYou will have to install the Android NDK, and point the `$NDK_ROOT` environment\nvariable to the NDK path: e.g. `export NDK_ROOT=/opt/android-ndk` (without a\ntrailing `/` character).\n\nNext, the easy way is to make a standalone Android toolchain with the following\ncommand:\n\n    $ $NDK_ROOT/build/tools/make-standalone-toolchain.sh --system=$(uname -s | tr [A-Z] [a-z])-$(uname -m) --platform=android-3 --toolchain=arm-linux-androideabi-clang3.3 --install-dir=/tmp/android-clang\n\nNow you can compile the self test and self benchmark programs by running:\n\n    $ make -C _gnu-make/ binsubdir=android CC=/tmp/android-clang/bin/clang CFLAGS='-march=armv7-a -mfloat-abi=softfp -mfpu=neon -O2'\n\n--------------------------------------------------------------------------------\n\n## Implementation details, what the hell is going on?\n\nFirst, in `PackedArray.c` or `PackedArraySIMD.c`, everything that comes below\nthe `- 8\u003c ----` marker is the code for the self test and self micro-benchmark\nprograms and can be discarded if you really want to:\n\nIf you want to cut down your anxiety, you can use the provided GNU Makefile and\ninvoke:\n\n    $ make -C _gnu-make/ cut\n\nThis produces the `PackedArray.cut.c` and `PackedArraySIMD.cut.c` files.\n\nYou may also be troubled by `PackedArray.c` and `PackedArraySIMD.c` including\nthemselves with `#include PACKEDARRAY_SELF`. By combining preprocessing tricks\nand including themselves, `PackedArray.c` and `PackedArraySIMD.c`\n\"generate the code\" for the unrolled pack and unpack implementations.\n\nBy default `PACKEDARRAY_SELF` is defined to `\"PackedArray.c\"` which assumes the\ncompiler is going to look for the file in the same directory as the file from\nwhich the `#include` statement is being evaluated. This helps compiling when the\nbuild system refers to the source files with relative paths. Depending on your\ncompiler/build system combination you may want to override `PACKEDARRAY_SELF` to\n`__FILE__`.\n\nIf you want to see the generated code, you can use the provided GNU Makefile and\ninvoke:\n\n    $ make -C _gnu-make/ preprocess\n\nThis produces the `PackedArray.pp.c` and `PackedArraySIMD.pp.c` files.\n\n\n--------------------------------------------------------------------------------\n\nIf you find `PackedArray` or `PackedArraySIMD` useful and decide to use it in\nyour own projects please drop me a line [@gpakosz].\n\nIf you use it in a commercial project, consider using [Gittip].\n\n[@gpakosz]: https://twitter.com/gpakosz\n[Gittip]: https://www.gittip.com/gpakosz/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpakosz%2Fpackedarray","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgpakosz%2Fpackedarray","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpakosz%2Fpackedarray/lists"}