{"id":27892610,"url":"https://github.com/itamarst/numba-arrow-research","last_synced_at":"2025-05-05T13:28:48.219Z","repository":{"id":278211052,"uuid":"934881030","full_name":"itamarst/numba-arrow-research","owner":"itamarst","description":"Research and maybe experiments in adding Arrow support to Numba","archived":false,"fork":false,"pushed_at":"2025-03-11T15:56:30.000Z","size":24,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-11T16:40:35.256Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/itamarst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-18T14:57:18.000Z","updated_at":"2025-03-11T15:56:33.000Z","dependencies_parsed_at":"2025-02-18T16:21:28.388Z","dependency_job_id":"569280fb-d300-4662-a1db-c51822ce12dc","html_url":"https://github.com/itamarst/numba-arrow-research","commit_stats":null,"previous_names":["itamarst/numba-arrow-research"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itamarst%2Fnumba-arrow-research","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itamarst%2Fnumba-arrow-research/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itamarst%2Fnumba-arrow-research/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itamarst%2Fnumba-arrow-research/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/itamarst","download_url":"https://codeload.github.com/itamarst/numba-arrow-research/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252506067,"owners_count":21758989,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-05T13:28:47.449Z","updated_at":"2025-05-05T13:28:48.214Z","avatar_url":"https://github.com/itamarst.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Research and maybe experiments in adding Arrow support to Numba\n\n## Why?\n\n* Numba only supports NumPy arrays out of the box.\n* Lots of projects now use Arrow (Polars, Pandas, if used PyArrow directly, no doubt others).\n* Numba is a nice way to write _fast_ extensions without switching languages.\n* Current Numba usage involves converting to NumPy arrays and then back, which is a problem because it loses information about missing data.\n\n## Existing attempts/conversations\n\n[Apparently](https://numba.discourse.group/t/feature-request-about-supporting-arrow-in-numba/1668/2) the Awkward Array library uses the same data representation as Arrow for its columns, and can therefore convert to/from Arrow with zero-copy.\nAnd Awkward Array has a Numba integration provided.\nSo this may just be a matter of documentation rather than coding.\n\nNext step, then:\n\n1. Validate that awkward array is indeed zero-copy from Arrow.\n2. Play around with the Numba integration and see if it works.\n3. In particular, do a proof-of-concept with Polars.\n\nDocs: https://awkward-array.org/doc/main/user-guide/how-to-use-in-numba-features.html\n\n## What I've learned so far\n\n* Pandas is a required dependency for the Awkard Arrow integration to work.\n  Annoying but not the end of the world.\n* Not necessarily much in the way of APIs to access?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitamarst%2Fnumba-arrow-research","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fitamarst%2Fnumba-arrow-research","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitamarst%2Fnumba-arrow-research/lists"}