{"id":15294898,"url":"https://github.com/poopoothegorilla/fastframe","last_synced_at":"2025-06-12T05:31:55.326Z","repository":{"id":74271379,"uuid":"259106357","full_name":"poopoothegorilla/fastframe","owner":"poopoothegorilla","description":"DataFrame project that utilizes Apache Arrow","archived":false,"fork":false,"pushed_at":"2020-07-08T03:09:36.000Z","size":223,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-05-07T05:12:29.299Z","etag":null,"topics":["apache-arrow","data-science","dataframe","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/poopoothegorilla.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-26T18:40:44.000Z","updated_at":"2024-11-16T05:31:46.000Z","dependencies_parsed_at":"2023-04-16T09:05:39.064Z","dependency_job_id":null,"html_url":"https://github.com/poopoothegorilla/fastframe","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/poopoothegorilla/fastframe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poopoothegorilla%2Ffastframe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poopoothegorilla%2Ffastframe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poopoothegorilla%2Ffastframe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poopoothegorilla%2Ffastframe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/poopoothegorilla","download_url":"https://codeload.github.com/poopoothegorilla/fastframe/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poopoothegorilla%2Ffastframe/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259404210,"owners_count":22852140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-arrow","data-science","dataframe","golang"],"created_at":"2024-09-30T17:07:49.138Z","updated_at":"2025-06-12T05:31:55.306Z","avatar_url":"https://github.com/poopoothegorilla.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FastFrame\n[![Go Report Card](https://goreportcard.com/badge/github.com/poopoothegorilla/fastframe)](https://goreportcard.com/report/github.com/poopoothegorilla/fastframe)\n\nWARNING: DO NOT USE IN PRODUCTION EVERYTHING IS EXPERIMENTAL\n\nFastFrame is a DataFrame project which utilizes Apache Arrow\n(https://github.com/apache/arrow/tree/master/go) for the underlying data\nstorage.\n\n## Benchmarks\n\n### Series\n\nSome of the benchmarks show where columnar / arrow based computation are beneficial. \n\nhttps://github.com/poopoothegorilla/fastframe/wiki/Benchmarks:-Series\n\n### DataFrame\n\nCurrently, the code for the joins could be improved.\nhttps://github.com/poopoothegorilla/fastframe/wiki/Benchmarks:-Datafame\n\n## Example\n\nThis is an example of calculating cosine similarity\n\nsee dataframe/example... files\n\n```go\nfunc generateMovieRatingsDF(pool memory.Allocator) dataframe.DataFrame {\n\tf := strings.NewReader(`\"user_name\",\"user_id\",\"movie_id\",\"rating\"\njim,1,2,3.5\njim,1,29,3.5\njim,1,32,3.5\njim,1,47,3.5\njim,1,50,3.5\njim,1,112,3.5\njoe,2,3,4\njoe,2,29,5\njoe,2,62,5\njoe,2,70,5\njoe,2,47,4\njoe,2,110,4\njoe,2,242,3\njoe,2,260,5\nfig,3,62,4\nfig,3,50,3.5\nfig,3,70,2\nfig,3,112,5`)\n\n\treturn dataframe.NewFromCSV(pool, csv.NewReader(f), -1, nil)\n}\n\nfunc Example() {\n\tpool := memory.NewGoAllocator()\n\n\trawMovieRatingsDF := generateMovieRatingsDF(pool)\n\tcastList := map[string]arrow.DataType{\n\t\t\"user_id\":  arrow.PrimitiveTypes.Int32,\n\t\t\"movie_id\": arrow.PrimitiveTypes.Int64,\n\t\t\"rating\":   arrow.PrimitiveTypes.Float64,\n\t}\n\tmovieRatingsDF := rawMovieRatingsDF.Cast(castList)\n\trawMovieRatingsDF.Release()\n\n\tfmt.Println(\"MOVIE RATINGS:\")\n\ttable := array.NewTableReader(movieRatingsDF, 5)\n\tn := 0\n\tfor table.Next() {\n\t\trec := table.Record()\n\t\tfor i, col := range rec.Columns() {\n\t\t\tfmt.Printf(\"rec[%d][%q]: %v\\n\", n, rec.ColumnName(i), col)\n\t\t}\n\t\tn++\n\t}\n\ttable.Release()\n\n\t// CREATE pivot table\n\tpivot := movieRatingsDF.Pivot(\"user_id\", \"movie_id\", \"rating\")\n\tfmt.Println(\"PIVOT:\")\n\ttable = array.NewTableReader(pivot, 3)\n\tn = 0\n\tfor table.Next() {\n\t\trec := table.Record()\n\t\tfor i, col := range rec.Columns() {\n\t\t\tfmt.Printf(\"rec[%d][%q]: %v\\n\", n, rec.ColumnName(i), col)\n\t\t}\n\t\tn++\n\t}\n\ttable.Release()\n\n\tmatrix := pivot.CosineSimilarity()\n\tpivot.Release()\n\n\tfmt.Println(\"MATRIX:\")\n\ttable = array.NewTableReader(matrix, 2)\n\tn = 0\n\tfor table.Next() {\n\t\trec := table.Record()\n\t\tfor i, col := range rec.Columns() {\n\t\t\tfmt.Printf(\"rec[%d][%q]: %v\\n\", n, rec.ColumnName(i), col)\n\t\t}\n\t\tn++\n\t}\n\ttable.Release()\n\tmatrix.Release()\n\t// Output:\n\t// MOVIE RATINGS:\n\t// rec[0][\"user_name\"]: [\"jim\" \"jim\" \"jim\" \"jim\" \"jim\"]\n\t// rec[0][\"user_id\"]: [1 1 1 1 1]\n\t// rec[0][\"movie_id\"]: [2 29 32 47 50]\n\t// rec[0][\"rating\"]: [3.5 3.5 3.5 3.5 3.5]\n\t// rec[1][\"user_name\"]: [\"jim\" \"joe\" \"joe\" \"joe\" \"joe\"]\n\t// rec[1][\"user_id\"]: [1 2 2 2 2]\n\t// rec[1][\"movie_id\"]: [112 3 29 62 70]\n\t// rec[1][\"rating\"]: [3.5 4 5 5 5]\n\t// rec[2][\"user_name\"]: [\"joe\" \"joe\" \"joe\" \"joe\" \"fig\"]\n\t// rec[2][\"user_id\"]: [2 2 2 2 3]\n\t// rec[2][\"movie_id\"]: [47 110 242 260 62]\n\t// rec[2][\"rating\"]: [4 4 3 5 4]\n\t// rec[3][\"user_name\"]: [\"fig\" \"fig\" \"fig\"]\n\t// rec[3][\"user_id\"]: [3 3 3]\n\t// rec[3][\"movie_id\"]: [50 70 112]\n\t// rec[3][\"rating\"]: [3.5 2 5]\n\t// PIVOT:\n\t// rec[0][\"user_id\"]: [1 2 3]\n\t// rec[0][\"2\"]: [3.5 0 0]\n\t// rec[0][\"29\"]: [3.5 5 0]\n\t// rec[0][\"32\"]: [3.5 0 0]\n\t// rec[0][\"47\"]: [3.5 4 0]\n\t// rec[0][\"50\"]: [3.5 0 3.5]\n\t// rec[0][\"112\"]: [3.5 0 5]\n\t// rec[0][\"3\"]: [0 4 0]\n\t// rec[0][\"62\"]: [0 5 4]\n\t// rec[0][\"70\"]: [0 5 2]\n\t// rec[0][\"110\"]: [0 4 0]\n\t// rec[0][\"242\"]: [0 3 0]\n\t// rec[0][\"260\"]: [0 5 0]\n\t// MATRIX:\n\t// rec[0][\"0\"]: [1 0.30588186723552135]\n\t// rec[0][\"1\"]: [0.30588186723552135 1]\n\t// rec[0][\"2\"]: [0.46616560472322743 0.3485753093366648]\n\t// rec[1][\"0\"]: [0.46616560472322743]\n\t// rec[1][\"1\"]: [0.3485753093366648]\n\t// rec[1][\"2\"]: [1]\n}\n```\n\n## Similar Projects\n\nBullseye (https://github.com/go-bullseye/bullseye): Another DataFrame project backed by Arrow. This project might merge with this project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpoopoothegorilla%2Ffastframe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpoopoothegorilla%2Ffastframe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpoopoothegorilla%2Ffastframe/lists"}