{"id":32556600,"url":"https://github.com/tatami-inc/tatami","last_synced_at":"2025-10-28T22:57:54.066Z","repository":{"id":38317276,"uuid":"369964413","full_name":"tatami-inc/tatami","owner":"tatami-inc","description":"C++ API for various matrix types.","archived":false,"fork":false,"pushed_at":"2025-09-15T00:46:13.000Z","size":29853,"stargazers_count":14,"open_issues_count":3,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-09-15T01:13:45.583Z","etag":null,"topics":["cpp"],"latest_commit_sha":null,"homepage":"https://tatami-inc.github.io/tatami","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tatami-inc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-05-23T04:59:46.000Z","updated_at":"2025-08-17T21:07:43.000Z","dependencies_parsed_at":"2024-05-22T19:30:29.665Z","dependency_job_id":"1782d5d8-29fc-4b72-a207-9cea7830815a","html_url":"https://github.com/tatami-inc/tatami","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/tatami-inc/tatami","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tatami-inc%2Ftatami","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tatami-inc%2Ftatami/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tatami-inc%2Ftatami/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tatami-inc%2Ftatami/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tatami-inc","download_url":"https://codeload.github.com/tatami-inc/tatami/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tatami-inc%2Ftatami/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281527380,"owners_count":26516845,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-28T02:00:06.022Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp"],"created_at":"2025-10-28T22:57:25.544Z","updated_at":"2025-10-28T22:57:54.056Z","avatar_url":"https://github.com/tatami-inc.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A C++ API for all sorts of matrices \n\n![Unit tests](https://github.com/tatami-inc/tatami/actions/workflows/run-tests.yaml/badge.svg)\n![Documentation](https://github.com/tatami-inc/tatami/actions/workflows/doxygenate.yaml/badge.svg)\n[![Codecov](https://codecov.io/gh/tatami-inc/tatami/branch/master/graph/badge.svg?token=Z189ORCLLR)](https://codecov.io/gh/tatami-inc/tatami)\n\n## Overview\n\n**tatami** is a spiritual successor to the [**beachmat** C++ API](https://github.com/tatami-inc/beachmat) that provides read access to different matrix representations.\nSpecifically, applications can use **tatami** to read rows and/or columns of a matrix without any knowledge of the specific matrix representation.\nThis allows application developers to write a single piece of code that will work seamlessly with different inputs, even if the underlying representation varies at run-time.\nIn particular, **tatami** is motivated by analyses of processed genomics data, where matrices are often interpreted as a collection of row- or column-wise vectors.\nMany applications involve looping over rows or columns to compute some statistic or summary - for example, testing for differential expression within each row of the matrix.\n**tatami** aims to optimize this access pattern across a variety of different matrix representations, depending on how the data is provided to the application.\n\n## Quick start\n\n**tatami** is a header-only library, so it can be easily used by just `#include`ing the relevant source files:\n\n```cpp\n#include \"tatami/tatami.hpp\"\n\nint nrows = 10;\nint ncols = 20;\nstd::vector\u003cdouble\u003e vals(nrows * ncols);\n\n// 'double' is the data type, 'int' is the row/column index type.\nstd::shared_ptr\u003ctatami::Matrix\u003cdouble, int\u003e \u003e mat(\n    new tatami::DenseMatrix\u003cdouble, int, std::vector\u003cdouble\u003e \u003e(\n        nrows,\n        ncols,\n        vals,\n        /* row_major = */ true\n    )\n);\n\n// Get the dimensions:\nint NR = mat-\u003enrow(), NC = mat-\u003encol();\n\n// Extract the 'i'-th column.\nauto extractor = mat-\u003edense_column();\nstd::vector\u003cdouble\u003e buffer(NR);\nauto ptr = extractor-\u003efetch(i, buffer.data());\nptr[0]; \n\n// Extract the [5, 12) rows of the 'i'-th column.\nauto sliced_extractor = mat-\u003edense_column(5, 7)\nauto sliced_ptr = sliced_extractor-\u003efetch(i, buffer.data());\n```\n\nThe key idea here is that, once `mat` is created, the application does not need to worry about the exact format of the matrix referenced by the pointer.\nApplication developers can write code that works interchangeably with a variety of different matrix representations.\n\n## Instructions\n\n### Creating a `tatami::Matrix`\n\nUsers can create an instance of a concrete `tatami::Matrix` subclass by using one of the constructors or the equivalent `make_*` utility:\n\n| Description                                | Class                             |\n|--------------------------------------------|-----------------------------------|\n| Dense matrix                               | `DenseMatrix`                     |\n| Compressed sparse matrix                   | `CompressedSparseMatrix`          |\n| List of lists sparse matrix                | `FragmentedSparseMatrix`          |\n| Delayed isometric unary operation          | `DelayedUnaryIsometricOperation`  |\n| Delayed isometric binary operation         | `DelayedBinaryIsometricOperation` |\n| Delayed combination                        | `DelayedBind`                     |\n| Delayed subset                             | `DelayedSubset`                   |\n| Delayed transpose                          | `DelayedTranspose`                |\n| Delayed cast                               | `DelayedCast`                     |\n| Delayed cast                               | `DelayedCast`                     |\n| Constant matrix                            | `ConstantMatrix`                  |\n\nFor example, to create a compressed sparse matrix from sparse triplet data, we could do:\n\n```cpp\n#include \"tatami/tatami.hpp\"\n\nint NROW = 10, NCOL = 20;\nstd::vector\u003cdouble\u003e x; // vector of non-zero values.\nstd::vector\u003cint\u003e i; // row indices of length equal to 'x'.\nstd::vector\u003cint\u003e j; // column indices of length equal to 'x'.\n\nauto indptrs = tatami::compress_sparse_triplets(NROW, NCOL, x, i, j, /* csr = */ false);\nstd::shared_ptr\u003ctatami::Matrix\u003cdouble, int\u003e \u003e smat(\n    new tatami::CompressedSparseMatrix\u003c\n        double, // data type\n        int, // index type\n        std::vector\u003cdouble\u003e, // type of 'x'\n        std::vector\u003cint\u003e, // type of the row indices 'i'\n        std::vector\u003csize_t\u003e // type of the pointers 'indptrs'\n    \u003e(\n        NR, \n        NC, \n        std::move(x), \n        std::move(i), \n        std::move(indptrs),\n        /* csr = */ false\n    )\n);\n```\n\nWe typically create a `shared_ptr` to a `tatami::Matrix` to leverage run-time polymorphism.\nThis enables downstream applications to accept many different matrix representations by compiling against the `tatami::Matrix` interface.\nAlternatively, applications may use templating to achieve compile-time polymorphism on the different **tatami** subclasses,\nbut this is rather restrictive without providing obvious performance benefits. \n\nWe use templating to define the type of values returned by the interface.\nThis includes the type of the data (most typically `double`) as well as the type of row/column indices (default `int`, but one could imagine using, e.g., `size_t`).\nIt is worth noting that the storage type does not need to be the same as the interface type.\nFor example, developers could store a matrix of small counts as `uint16_t` while returning `double`s for compatibility with downstream mathematical code.\n\nThe delayed operations are ~stolen from~ inspired by those in the [**DelayedArray**](https://github.com/Bioconductor/DelayedArray) package.\nIsometric operations are particularly useful as they accommodate matrix-scalar/vector arithmetic and various mathematical operations.\nFor example, we could apply a sparsity-breaking delayed operation to our sparse matrix `smat` without actually creating a dense matrix:\n\n```cpp\nstd::shared_ptr\u003ctatami::Matrix\u003cdouble, int\u003e \u003e mat2(\n    new tatami::DelayedUnaryIsometricOperation\u003c\n        double, // type of the result of the operation \n        double, // type of the original matrix\n        int // row/column index type\n    \u003e(\n        smat, \n        std::make_shared\u003c\n            tatami::DelayedUnaryIsometricAddScalarHelper\u003c\n                double, // type of the result of the operation\n                double, // type of the original matrix\n                int, // row/column index type\n                double // type of the scalar\n            \u003e\n        \u003e(2.0)\n    )\n);\n```\n\nSome libraries in the [**@tatami-inc**](https://github.com/tatami-inc) organization implement further extensions of **tatami**'s interface, \ne.g., for [HDF5-backed matrices](https://github.com/tatami-inc/tatami_hdf5) and [R-based matrices](https://github.com/tatami-inc/tatami_r).\n\n### Extracting matrix contents\n\nGiven an abstract `tatami::Matrix`, we create an `Extractor` instance to actually extract the matrix data.\nEach `Extractor` object can store intermediate data for re-use during iteration through the matrix, which is helpful for some matrix implementations that do not easily support random access.\nFor example, to perform extract dense rows from our `mat`:\n\n```cpp\nint NR = mat-\u003enrow();\nint NC = mat-\u003encol();\n\nauto ext = mat-\u003edense_row();\nstd::vector\u003cdouble\u003e buffer(NC);\n\nfor (int r = 0; r \u003c NR; ++r) {\n    auto current = ext-\u003efetch(r, buffer.data());\n    // Do something with the 'current' pointer.\n}\n```\n\nThe `tatami::MyopicDenseExtractor::fetch()` method returns a pointer to the row's contents.\nIn some matrix representations (e.g., `DenseMatrix`), the returned pointer directly refers to the matrix's internal data store.\nHowever, this is not the case in general so we need to allocate a buffer of appropriate length (`buffer`) in which the dense contents can be stored;\nif this buffer is used, the returned pointer refers to the start of the buffer.\n\nAlternatively, we could extract sparse columns via `tatami::MyopicSparseExtractor::fetch()`, \nwhich returns a `tatami::SparseRange` containing pointers to arrays of (structurally non-zero) values and their row indices.\nThis provides some opportunities for optimization in algorithms that only need to operate on non-zero values.\nThe `fetch()` call requires buffers for both arrays - again, this may not be used for matrix subclasses with contiguous internal storage of the values/indices.\n\n```cpp\nauto sext = mat-\u003esparse_column();\nstd::vector\u003cdouble\u003e vbuffer(NR);\nstd::vector\u003cint\u003e ibuffer(NR);\n\nfor (int c = 0; c \u003c NC; ++c) {\n    auto current = sext-\u003efetch(c, vbuffer.data(), ibuffer.data());\n    current.number; // Number of structural non-zeros\n    current.value; // Pointer to the value array\n    current.index; // Pointer to the index array\n}\n```\n\nIn both the dense and sparse cases, we can restrict the values that are extracted by `fetch()`.\nThis provides some opportunities for optimization by avoiding the unnecessary extraction of uninteresting data.\nTo do so, we specify the start and length of a contiguous block of interest, or we supply a vector containing the indices of the elements of interest:\n\n```cpp\n// Get rows [5, 17) from each column.\nauto bext = mat-\u003edense_column(5, 12); \n\n// Get these columns from each row.\nauto iext = mat-\u003esparse_row(std::vector\u003cint\u003e{ 1, 3, 5, 7 });\n```\n\n### Handling different access patterns\n\nIn performance-critical sections, it may be desirable to customize the extraction based on properties of the matrix.\nThis is supported with the following methods:\n\n- `tatami::Matrix::is_sparse()` indicates whether a matrix is sparse.\n- `tatami::Matrix::prefer_rows()` indicates whether a matrix is more efficiently accessed along its rows (e.g., row-major dense matrices).\n\nUsers can then write dedicated code paths to take advantage of these properties.\nFor example, we might use different algorithms for dense data, where we don't have to look up indices; and for sparse data, if we can avoid the uninteresting zero values.\nSimilarly, if we want to compute a row-wise statistic, but the matrix is more efficiently accessed by column according to `prefer_rows()`,\nwe could iterate on the columns and attempt to compute the statistic in a \"running\" manner \n(see [`colsums.cpp`](https://github.com/tatami-inc/gallery/tree/master/src/colsums.cpp) for an example).\nIn the most complex cases, this leads to code like:\n\n```cpp\nif (mat-\u003eis_sparse()) {\n    if (mat-\u003eprefer_rows()) {\n        auto sext = mat-\u003esparse_row();\n        // Do compute along sparse rows.\n    } else {\n        auto sext = mat-\u003esparse_column();\n        // Do compute along sparse columns.\n    }\n} else {\n    if (mat-\u003eprefer_rows()) {\n        auto sext = mat-\u003edense_row();\n        // Do compute along dense rows.\n    } else {\n        auto sext = mat-\u003edense_column();\n        // Do compute along dense columns.\n    }\n}\n```\n\nOf course, this assumes that it is possible to provide sparse-specific optimizations as well as running calculations for the statistic of interest.\nIn most cases, only a subset of the extraction patterns are actually feasible so special code paths would not be beneficial.\n\n### Supporting parallelization\n\nThe mutable nature of an `Extractor` instance means that the `fetch()` calls themselves are not `const`.\nThis means that the same extractor cannot be safely re-used across different threads as each call to `fetch()` will modify the extractor's contents.\nFortunately, the solution is simple - just create a separate `Extractor` (and the associated buffers) for each thread.\nWith OpenMP, this looks like:\n\n```cpp\n#pragma omp parallel num_threads(nthreads);\n{\n    auto wrk = mat-\u003edense_row();\n    std::vector\u003cdouble\u003e buffer(NC);\n\n    #pragma omp for\n    for (int r = 0; r \u003c NR; ++r) {\n        auto ptr = wrk-\u003efetch(r, buffer.data());\n        // Do something in each thread.\n    }\n}\n```\n\nUsers may also consider using the `tatami::parallelize()` function, which accepts a function with the range of jobs (in this case, rows) to be processed in each thread.\nThis automatically falls back to the standard `\u003cthread\u003e` library if OpenMP is not available.\nApplications can also set the `TATAMI_CUSTOM_PARALLEL` macro to override the parallelization scheme in all `tatami::parallelize()` calls.\n\n```cpp\ntatami::parallelize([\u0026](int thread, int start, int length) -\u003e void {\n    auto wrk = mat-\u003edense_row();\n    std::vector\u003cdouble\u003e buffer(NC);\n    for (int r = 0; r \u003c length; ++r) {\n        auto ptr = wrk-\u003efetch(r + start, buffer.data());\n        // Do something in each thread.\n    }\n}, NR, nthreads);\n```\n\n### Defining an oracle\n\nWhen constructing an `Extractor`, users can supply an `Oracle` that specifies the sequence of rows/columns to be accessed.\nKnowledge of the future access pattern enables optimizations in some `Matrix` implementations, \ne.g., file-backed matrices can reduce the number of disk reads by pre-fetching the right data for future accesses.\nThe most obvious use case involves accessing consecutive rows/columns:\n\n```cpp\nauto o = std::make_shared\u003ctatami::ConsecutiveOracle\u003cint\u003e \u003e(0, NR));\nauto wrk = mat-\u003edense_row(o);\nfor (int r = 0; r \u003c NR; ++r) {\n    // No need to specify the index to fetch, as 'wrk' already knows the\n    // sequence of indices as prediced by the oracle.\n    auto ptr = wrk-\u003efetch(buffer.data());\n}\n```\n\nIn fact, this use case is so common that we can just use the `tatami::consecutive_extractor()` wrapper to construct the oracle and pass it to `tatami::Matrix`.\nThis will return a `tatami::OracularDenseExtractor` instance that contains the oracle's predictions.\n\n```cpp\n// Same as 'wrk' above.\nauto cwrk = tatami::consecutive_extractor\u003cfalse\u003e(mat.get(), row, 0, NR);\n```\n\nAlternatively, we can use the `FixedOracle` class with an array of row/column indices that are known ahead of time.\nAdvanced users can also define their own `Oracle` subclasses to generate predictions on the fly.\n\n## Comments on other operations\n\nAs previously mentioned, **tatami** is primarily designed to pull out rows or columns of a matrix.\nSome support is provided for computing basic statistics via the [**tatami_stats**](https://github.com/tatami-inc/tatami_stats) library,\nin the same vein as the [**matrixStats**](https://github.com/HenrikBengtsson/matrixStats) R package.\nMatrix multiplication is similarly implemented via the [**tatami_mult**](https://github.com/tatami-inc/tatami_mult) library.\n\n**tatami** does not currently support more sophisticated matrix operations like decompositions. \nIf these are required, we suggest copying data from **tatami** into other frameworks like [**Eigen**](https://eigen.tuxfamily.org/), \neffectively trading the diversity of representations for a more comprehensive suite of operations.\nFor example, we often use **tatami** to represent the large input datasets in a custom memory-efficient format;\nprocess it into a much smaller submatrix, e.g., by selecting features of interest in a genome-scale analysis;\nand then copy the data into a `Eigen::MatrixXd` or `Eigen::SparseMatrix` for the desired operations.\nIn practice, many standard decompositions do not scale well for large matrices,\nso our applications end up using approximate methods like [**ILRBA**](https://github.com/LTLA/CppIrlba) that only require a multiplication operator.\n\n\u003c!---\n(At this point, it is worth noting that **Eigen** also supports delayed operations via its expression templates.\nThese are determined at compile-time and are more efficient than **tatami**'s operations.\nHowever, it is also much harder to pass expression templates around an application while preserving the delayed operations.\nEach expression is its own type so supporting multiple expressions would require a potentially-combinatorial increase in the number of realized template functions.\nAdditionally, the expressions do not extend the lifetime of the matrices on which they operate, so if the matrix is destructed before the expression is evaluated, a segfault will occur.\n**tatami** avoids these problems and makes it easier to pass around a matrix with delayed operations.)\n--\u003e\n\n**tatami** does not directly support modification of the matrix contents.\nInstead, \"modifications\" are performed by adding delayed operations on top of an immutable matrix.\nThis avoids difficult bugs where the hypothetical modification of matrix contents via one `shared_ptr` affects all references to the same matrix across the application.\nDelayed operations are also more appropriate for matrices referring to read-only data sources, e.g., remote stores or files.\nThat said, if delayed operations are undesirable, we can use functions like `tatami::convert_to_dense()` to realize our modifications into a new matrix instance. \n\n## Building projects \n\n### CMake with `FetchContent`\n\nIf you're using CMake, you just need to add something like this to your `CMakeLists.txt`:\n\n```cmake\ninclude(FetchContent)\n\nFetchContent_Declare(\n  tatami\n  GIT_REPOSITORY https://github.com/tatami-inc/tatami\n  GIT_TAG master # or any version of interest \n)\n\nFetchContent_MakeAvailable(tatami)\n```\n\nThen you can link to **tatami** to make the headers available during compilation:\n\n```cmake\n# For executables:\ntarget_link_libraries(myexe tatami)\n\n# For libaries\ntarget_link_libraries(mylib INTERFACE tatami)\n```\n\nBy default, this will use `FetchContent` to fetch all external dependencies.\nApplications are advised to pin the versions of these dependencies themselves - see [`extern/CMakeLists.txt`](extern/CMakeLists.txt) for suggested (minimum) versions.\nIf you want to install them manually, use `-DTATAMI_FETCH_EXTERN=OFF`.\n\n### CMake with `find_package()`\n\nYou can install the library by cloning a suitable version of this repository and running the following commands:\n\n```sh\nmkdir build \u0026\u0026 cd build\ncmake .. -DTATAMI_TESTS=OFF\ncmake --build . --target install\n```\n\nThen you can use `find_package()` as usual:\n\n```cmake\nfind_package(tatami_tatami CONFIG REQUIRED)\ntarget_link_libraries(mylib INTERFACE tatami::tatami)\n```\n\nAgain, this will use `FetchContent` to fetch all external dependencies, so see advice above.\n\n### Manual\n\nIf you're not using CMake, the simple approach is to just copy the files the `include/` subdirectory - \neither directly or with Git submodules - and include their path during compilation with, e.g., GCC's `-I`.\nThis also requires the external dependencies listed in [`extern/CMakeLists.txt`](extern/CMakeLists.txt). \n\n## Links\n\nCheck out the [reference documentation](https://tatami-inc.github.io/tatami) for more details on each function and class.\n\nThe [gallery](https://github.com/tatami-inc/gallery) contains worked examples for common operations based on row/column traversals.\n\nThe [**tatami_stats**](https://github.com/tatami-inc/tatami_stats) repository computes some common statistics on **tatami** matrices.\n\nThe [**tatami_hdf5**](https://github.com/tatami-inc/tatami_hdf5) repository contains **tatami** bindings for HDF5-backed matrices.\n\nThe [**tatami_r**](https://github.com/tatami-inc/tatami_r) repository contains **tatami** bindings for matrix-like objects in R.\n\nThe [**beachmat**](https://github.com/tatami-inc/beachmat) package vendors the **tatami** headers for easy use by other R packages.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftatami-inc%2Ftatami","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftatami-inc%2Ftatami","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftatami-inc%2Ftatami/lists"}