{"id":33179882,"url":"https://github.com/vmzakharov/dataframe-ec","last_synced_at":"2026-01-14T02:55:04.678Z","repository":{"id":40779111,"uuid":"271114804","full_name":"vmzakharov/dataframe-ec","owner":"vmzakharov","description":"A tabular data structure (aka a data frame) based on the Eclipse Collections framework","archived":false,"fork":false,"pushed_at":"2025-10-31T18:23:24.000Z","size":1223,"stargazers_count":66,"open_issues_count":0,"forks_count":10,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-10-31T20:23:57.682Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vmzakharov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-06-09T21:35:55.000Z","updated_at":"2025-10-31T18:23:28.000Z","dependencies_parsed_at":"2024-04-22T20:59:14.972Z","dependency_job_id":"58dbbe4e-3083-41a4-a6fc-225ea7b327bd","html_url":"https://github.com/vmzakharov/dataframe-ec","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/vmzakharov/dataframe-ec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmzakharov%2Fdataframe-ec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmzakharov%2Fdataframe-ec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmzakharov%2Fdataframe-ec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmzakharov%2Fdataframe-ec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vmzakharov","download_url":"https://codeload.github.com/vmzakharov/dataframe-ec/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmzakharov%2Fdataframe-ec/sbom","scorecard":{"id":1239389,"data":{"date":"2025-10-20","repo":{"name":"github.com/vmzakharov/dataframe-ec","commit":"b011553da79343ae24fdd8a9c6d73be96cf31c2e"},"scorecard":{"version":"v5.3.1-0.20251020080748-9e702f4c9dc6","commit":"9e702f4c9dc6992745395f4c8ac89942d0cf87ea"},"score":3.4,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 1/29 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/maven-verify.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/maven-verify.yml:14: update your workflow using https://app.stepsecurity.io/secureworkflow/vmzakharov/dataframe-ec/maven-verify.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/maven-verify.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/vmzakharov/dataframe-ec/maven-verify.yml/master?enable=pin","Info:   0 out of   2 GitHub-owned GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 2 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/9e702f4c9dc6992745395f4c8ac89942d0cf87ea/docs/checks.md#sast"}}]},"last_synced_at":"2025-10-31T20:23:56.612Z","repository_id":40779111,"created_at":"2025-10-31T20:23:56.618Z","updated_at":"2025-10-31T20:23:56.618Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28408798,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T01:52:23.358Z","status":"online","status_checked_at":"2026-01-14T02:00:06.678Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-16T03:00:36.844Z","updated_at":"2026-01-14T02:55:04.669Z","avatar_url":"https://github.com/vmzakharov.png","language":"Java","funding_links":[],"categories":["数据科学"],"sub_categories":[],"readme":"# dataframe-ec\nA tabular data structure (aka a data frame) based on the Eclipse Collections framework. The underlying data frame structure is columnar, with focus on memory optimization achieved by using efficient Eclipse Collections data structures and APIs.\n\nFor more on Eclipse Collections see: https://www.eclipse.org/collections/.\n\n## Where to Get It\n\nGet the latest release of `dataframe-ec` here:\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eio.github.vmzakharov\u003c/groupId\u003e\n  \u003cartifactId\u003edataframe-ec\u003c/artifactId\u003e\n  \u003cversion\u003e1.4.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nNote that starting with the release `0.20.0`, `dataframe-ec` is compatible with JDK 17+. The last release of `dataframe-ec` to support Java 8+ is `0.19.8` which can be found at these coordinates:\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eio.github.vmzakharov\u003c/groupId\u003e\n  \u003cartifactId\u003edataframe-ec\u003c/artifactId\u003e\n  \u003cversion\u003e0.19.8\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## Code Kata\n\nLearn dataframe-ec with a Kata! Check out [dataframe-ec kata](https://github.com/vmzakharov/dataframe-ec-kata), an easy and fun way to learn the basics of data frame APIs and common usage patterns through hands-on exercises.\n\n## Data Frame Operations\n\n- **create** a data frame programmatically or load from a csv file\n- **add a column** to a data frame, columns can be\n    - stored or computed\n    - of type: string, long, int, double, float, date, date/time\n- **drop** one or more **columns**\n- **select** a subset of rows based on a criteria\n- **sort** by one or more columns or by an expression\n- select **distinct** rows based on all column values or a subset of columns\n- **union** - concatenating data frames with the same schemas\n- **join** with another data frame, based on the specified column values, inner and outer joins are supported\n- **join with complements**, a single operation that returns three data frames - a complement of this data frame in another one, an inner join of the data frames, and a complement of the other data frame in this one\n- **lookup join** - enrich a data frame by adding one or more columns based on value lookup in another data frame\n- **pivot** - similar to pivot in a spreadsheet where the distinct values in a column become columns in the pivot table and the row values in the table are grouped and aggregated based on the aggregation functions provided\n- find relative complements (**set differences**) of two data frames based on the specified column values\n- **aggregation** - aggregating the entire data frame or grouping by the specified column values and aggregating within a group\n- **flag rows** - individually or matching a criteria\n\u003cbr\u003e...and more!\n\n## Additional Reading\n\n* [CSV File Support](docs/CSV_FILE_SUPPORT.md) - reading and writing CSV (and CSV-like) files\n* [Handling Null Values](docs/NULL_SUPPORT.md) - dealing with real life data requires flexibility in handling nulls\n* [Object Pooling](docs/POOLING.md) - the framework uses pooling to save memory\n* [JSON Support](https://github.com/vmzakharov/dataframe-ec-json-support) - a companion project that support reading and writing JSON formatted text\n\n## Code Examples\n\n### Creating a Data Frame\nA data frame can be **loaded from a CSV file**. For more details on reading and writing CSV (and CSV-like) files see the **[Csv File Support](docs/CSV_FILE_SUPPORT.md)** document.\n\nLet's say there is a file called \"donut_orders.csv\" with the following contents:\n\n```\nCustomer,  Count,  Price,  Date\n\"Archibald\", 5.0, 23.45, 2020-10-15\n\"Bridget\", 10.0, 40.34, 2020-11-10\n\"Clyde\", 4.0, 19.5, 2020-10-19\n```\n\nThen a data frame can be loaded from this file as shown below. The file schema can be inferred, like in this example, or specified explicitly.\n```Java\nDataFrame ordersFromFile  = new CsvDataSet(\"donut_orders.csv\", \"Donut Orders\").loadAsDataFrame();\n```\n\n`ordersFromFile`\n\n|Customer |   Count |   Price |   Date|\n|---|---:|---:|---|\n|\"Archibald\" | 5.0000 | 23.4500 | 2020-10-15|\n|\"Bridget\" | 10.0000 | 40.3400 | 2020-11-10|\n|\"Clyde\" | 4.0000 | 19.5000 | 2020-10-19|\n\nThe `loadAsDataFrame()` method can take a numeric parameter, which specifies how many rows of data to load. For example:\n```Java\nDataFrame firstTwoOrders = new CsvDataSet(\"donut_orders.csv\", \"Donut Orders\").loadAsDataFrame(2);\n```\n`firstTwoOrders`\n\n|Customer |   Count |   Price |   Date|\n|---|---:|---:|---|\n|\"Archibald\" | 5.0000 | 23.4500 | 2020-10-15|\n|\"Bridget\" | 10.0000 | 40.3400 | 2020-11-10|\n\n`CsvDataSet` can also accept a **schema** object, which explicitly defines column types and also supports a number of options such as the value separator and the null marker in a source file. For example, let's consider a file \"donut_orders_complicated.csv\" that looks like\n\n```\nCustomer|Count|Price|Date\n\"Archibald\"|5|23.45|2020-10-15\n\"Bridget\"|10|*null*|2020-11-10\n\"Clyde\"|4|19.5|*null*\n```\n\nA data frame can be loaded from this file by providing a schema\n\n```Java\nCsvSchema donutSchema = new CsvSchema()\n        .separator('|')\n        .nullMarker(\"*null*\");\ndonutSchema.addColumn(\"Customer\", STRING);\ndonutSchema.addColumn(\"Count\", LONG);\ndonutSchema.addColumn(\"Price\", DOUBLE);\ndonutSchema.addColumn(\"Date\", DATE);\n\nCsvDataSet dataSet = new CsvDataSet(\"donut_orders_complicated.csv\", \"Donut Orders\", donutSchema);\nDataFrame schemingDonuts = dataSet.loadAsDataFrame();\n```\n\n`schemingDonuts`\n\n|Customer | Count | Price | Date|\n|---|---:|---:|---|\n|\"Archibald\" | 5 | 23.4500 | 2020-10-15|\n|\"Bridget\" | 10 | null | 2020-11-10|\n|\"Clyde\" | 4 | 19.5000 | null|\n\nA data frame can be created **programmatically** by providing values for individual rows or columns. Here is a sample constructing a data frame row by row:\n```Java\nDataFrame orders = new DataFrame(\"Donut Orders\")\n    .addStringColumn(\"Customer\").addLongColumn(\"Count\").addDoubleColumn(\"Price\").addDateColumn(\"Date\")\n    .addRow(\"Alice\",  5, 23.45, LocalDate.of(2020, 10, 15))\n    .addRow(\"Bob\",   10, 40.34, LocalDate.of(2020, 11, 10))\n    .addRow(\"Alice\",  4, 19.50, LocalDate.of(2020, 10, 19))\n    .addRow(\"Carl\",  11, 44.78, LocalDate.of(2020, 12, 25))\n    .addRow(\"Doris\",  1,  5.00, LocalDate.of(2020,  9,  1));\n```\n`orders`\n\n|Customer | Count | Price | Date|\n|---|---:|---:|---|\n|\"Alice\" | 5 | 23.4500 | 2020-10-15|\n|\"Bob\" | 10 | 40.3400 | 2020-11-10|\n|\"Alice\" | 4 | 19.5000 | 2020-10-19|\n|\"Carl\" | 11 | 44.7800 | 2020-12-25|\n|\"Doris\" | 1 | 5.0000 | 2020-09-01|\n\nThis way of creating a data frame is more useful for contexts like unit tests, where readability matters. In your applications you probably want to load a data frame from a file or populate individual columns with strongly typed values as in the following example, which produces a data frame with the same exact contents as the one in the example above:\n\n```Java\nDataFrame ordersByCol = new DataFrame(\"Donut Orders\")\n        .addStringColumn(\"Customer\", Lists.immutable.of(\"Alice\", \"Bob\", \"Alice\", \"Carl\", \"Doris\"))\n        .addLongColumn(\"Count\", LongLists.immutable.of(5, 10, 4, 11, 1))\n        .addDoubleColumn(\"Price\", DoubleLists.immutable.of(23.45, 40.34, 19.50, 44.78, 5.00))\n        .addDateColumn(\"Date\", Lists.immutable.of(LocalDate.of(2020, 10, 15), LocalDate.of(2020, 11, 10),\n                LocalDate.of(2020, 10, 19), LocalDate.of(2020, 12, 25), LocalDate.of(2020, 9, 1)));\nordersByCol.seal(); // finished constructing a data frame\n```\n\n`ordersByCol`\n\n|Customer | Count | Price | Date|\n|---|---:|---:|---|\n|\"Alice\" | 5 | 23.4500 | 2020-10-15|\n|\"Bob\" | 10 | 40.3400 | 2020-11-10|\n|\"Alice\" | 4 | 19.5000 | 2020-10-19|\n|\"Carl\" | 11 | 44.7800 | 2020-12-25|\n|\"Doris\" | 1 | 5.0000 | 2020-09-01|\n\nA data frame can also be created from a hierarchical data set via a **projection operator** supported by the DSL. Let's say we have two record types: `Person` and `Address`. Then we can use the projection operator to turn a list of `Person` objects into a data frame as follows.\n\n```Java\nvar visitor = new InMemoryEvaluationVisitor();\nvisitor.getContext().addDataSet(\n        new ObjectListDataSet(\"Person\", Lists.immutable.of(\n            new Person(\"Alice\", 30, new Address(\"Brooklyn\", \"North Dakota\")),\n            new Person(\"Bob\", 40, new Address(\"Lexington\", \"Nebraska\")),\n            new Person(\"Carl\", 50, new Address(\"Northwood\", \"Idaho\"))\n        )));\n\nvar script = ExpressionParserHelper.DEFAULT.toScript(\"\"\"\n        project {\n            Name : Person.name,\n            ${Lucky Number} : Person.luckyNumber,\n            State : Person.address.state\n        }\n        \"\"\");\n\nDataFrame projectionValue = ((DataFrameValue) script.evaluate(visitor)).dataFrame();\n```\n\n`projectionValue`\n\n|Name | Lucky Number | State|\n|---|---:|---|\n|\"Alice\" | 30 | \"North Dakota\"|\n|\"Bob\" | 40 | \"Nebraska\"|\n|\"Carl\" | 50 | \"Idaho\"|\n\n### Aggregation Functions\n\n#### Built-in Aggregation Functions\n\nThe following aggregation functions are supported out of the box.\n- `sum`\n- `min`\n- `max`\n- `avg` - average, the return value is of the same type as the input data\n- `avg2d` - average, the return value is of type double for any primitive input type\n- `count`\n- `same` - the result is `null` if the aggregated values are not the equal to each other, otherwise it equals to that value\n\nIt is possible to add custom aggregation functions at runtime (see unit tests for examples).\n\nFor the following examples we will use the `orders` data frame we created earlier:\n\n`orders`\n\n|Customer | Count | Price | Date|\n|---|---:|---:|---|\n|\"Alice\" | 5 | 23.4500 | 2020-10-15|\n|\"Bob\" | 10 | 40.3400 | 2020-11-10|\n|\"Alice\" | 4 | 19.5000 | 2020-10-19|\n|\"Carl\" | 11 | 44.7800 | 2020-12-25|\n|\"Doris\" | 1 | 5.0000 | 2020-09-01|\n\n#### Aggregation Example\n\n```Java\nDataFrame orderStats = orders.aggregate(Lists.immutable.of(max(\"Price\", \"MaxPrice\"), min(\"Price\", \"MinPrice\"), sum(\"Price\", \"Total\")));\n```\n\n`orderStats`\n\n|MaxPrice | MinPrice | Total|\n|---:|---:|---:|\n|44.7800 | 5.0000 | 133.0700|\n\n#### Sum of Columns\n```Java\nDataFrame totalOrdered = orders.sum(Lists.immutable.of(\"Count\", \"Price\"));\n```\n\n`totalOrdered`\n\n|Count | Price|\n|---:|---:|\n|31 | 133.0700|\n\n#### Sum With Group By\n```Java\nDataFrame totalsByCustomer = orders.sumBy(Lists.immutable.of(\"Count\", \"Price\"), Lists.immutable.of(\"Customer\"));\n```\n\n`totalsByCustomer`\n\n|Customer | Count | Price|\n|---|---:|---:|\n|\"Alice\" | 9 | 42.9500|\n|\"Bob\" | 10 | 40.3400|\n|\"Carl\" | 11 | 44.7800|\n|\"Doris\" | 1 | 5.0000|\n\n### Add a Calculated Column\n```Java\norders.addColumn(\"AvgDonutPrice\", \"Price / Count\");\n```\n`orders`\n\n|Customer | Count | Price | Date | AvgDonutPrice|\n|---|---:|---:|---|---:|\n|\"Alice\" | 5 | 23.4500 | 2020-10-15 | 4.6900|\n|\"Bob\" | 10 | 40.3400 | 2020-11-10 | 4.0340|\n|\"Alice\" | 4 | 19.5000 | 2020-10-19 | 4.8750|\n|\"Carl\" | 11 | 44.7800 | 2020-12-25 | 4.0709|\n|\"Doris\" | 1 | 5.0000 | 2020-09-01 | 5.0000|\n\n### Filtering\nSelection of a dataframe containing the rows matching the filter condition\n```Java\nDataFrame bigOrders = orders.selectBy(\"Count \u003e= 10\");\n```\n`bigOrders`\n\n|Customer | Count | Price | Date | AvgDonutPrice|\n|---|---:|---:|---|---:|\n|\"Bob\" | 10 | 40.3400 | 2020-11-10 | 4.0340|\n|\"Carl\" | 11 | 44.7800 | 2020-12-25 | 4.0709|\n\nSelection of a dataframe *without* the rows matching the filter condition\n```Java\nDataFrame smallOrders = orders.rejectBy(\"Count \u003e= 10\");\n```\n`smallOrders`\n\n|Customer | Count | Price | Date | AvgDonutPrice|\n|---|---:|---:|---|---:|\n|\"Alice\" | 5 | 23.4500 | 2020-10-15 | 4.6900|\n|\"Alice\" | 4 | 19.5000 | 2020-10-19 | 4.8750|\n|\"Doris\" | 1 | 5.0000 | 2020-09-01 | 5.0000|\n\nSplit the rows of a data frame into two subsets both matching and not matching the filter condition respectively\n```Java\nTwin\u003cDataFrame\u003e highAndLow = orders.partition(\"Count \u003e= 10\");\n```\nResult - a pair of data frames:\n\n`highAndLow.getOne()`\n\n|Customer | Count | Price | Date | AvgDonutPrice|\n|---|---:|---:|---|---:|\n|\"Bob\" | 10 | 40.3400 | 2020-11-10 | 4.0340|\n|\"Carl\" | 11 | 44.7800 | 2020-12-25 | 4.0709|\n\n`highAndLow.getTwo()`\n\n|Customer | Count | Price | Date | AvgDonutPrice|\n|---|---:|---:|---|---:|\n|\"Alice\" | 5 | 23.4500 | 2020-10-15 | 4.6900|\n|\"Alice\" | 4 | 19.5000 | 2020-10-19 | 4.8750|\n|\"Doris\" | 1 | 5.0000 | 2020-09-01 | 5.0000|\n\n### Drop Column\n```Java\norders.dropColumn(\"AvgDonutPrice\");\n```\n`orders`\n\n|Customer | Count | Price | Date|\n|---|---:|---:|---|\n|\"Alice\" | 5 | 23.4500 | 2020-10-15|\n|\"Bob\" | 10 | 40.3400 | 2020-11-10|\n|\"Alice\" | 4 | 19.5000 | 2020-10-19|\n|\"Carl\" | 11 | 44.7800 | 2020-12-25|\n|\"Doris\" | 1 | 5.0000 | 2020-09-01|\n\n### Sort\nSort by the order date:\n```Java\norders.sortBy(Lists.immutable.of(\"Date\"));\n```\n`orders`\n\n|Customer | Count | Price | Date|\n|---|---:|---:|---|\n|\"Doris\" | 1 | 5.0000 | 2020-09-01|\n|\"Alice\" | 5 | 23.4500 | 2020-10-15|\n|\"Alice\" | 4 | 19.5000 | 2020-10-19|\n|\"Bob\" | 10 | 40.3400 | 2020-11-10|\n|\"Carl\" | 11 | 44.7800 | 2020-12-25|\n\nSort by Customer ignoring the first letter of their name\n```Java\norders.sortByExpression(\"substr(Customer, 1)\");\n```\n`orders`\n\n|Customer | Count | Price | Date|\n|---|---:|---:|---|\n|\"Carl\" | 11 | 44.7800 | 2020-12-25|\n|\"Alice\" | 5 | 23.4500 | 2020-10-15|\n|\"Alice\" | 4 | 19.5000 | 2020-10-19|\n|\"Bob\" | 10 | 40.3400 | 2020-11-10|\n|\"Doris\" | 1 | 5.0000 | 2020-09-01|\n\n### Union\n\nThe union operation concatenates two data frames with the same schema. Note that it does not remove duplicate rows.\n```Java\nDataFrame otherOrders = new DataFrame(\"Other Donut Orders\")\n    .addStringColumn(\"Customer\").addLongColumn(\"Count\").addDoubleColumn(\"Price\").addDateColumn(\"Date\")\n    .addRow(\"Eve\",  2, 9.80, LocalDate.of(2020, 12, 5));\n\nDataFrame combinedOrders = orders.union(otherOrders);\n```\n`combinedOrders`\n\n|Customer | Count | Price | Date|\n|---|---:|---:|---|\n|\"Alice\" | 5 | 23.4500 | 2020-10-15|\n|\"Bob\" | 10 | 40.3400 | 2020-11-10|\n|\"Alice\" | 4 | 19.5000 | 2020-10-19|\n|\"Carl\" | 11 | 44.7800 | 2020-12-25|\n|\"Doris\" | 1 | 5.0000 | 2020-09-01|\n|\"Eve\" | 2 | 9.8000 | 2020-12-05|\n\n### Distinct\nSay we want to get a list of all clients who placed orders, which are listed in the `orders` data frame above. We can\nuse the `distinct()` method for that:\n\n```Java\nDataFrame distinctCustomers = orders.distinct(Lists.immutable.of(\"Customer\"));\n```\n\n`distinctCustomers`\n\n|Customer|\n|---|\n|\"Alice\"|\n|\"Bob\"|\n|\"Carl\"|\n|\"Doris\"|\n\n### Joins\n\n#### Inner and Outer Joins\n```Java\nDataFrame joining1 = new DataFrame(\"df1\")\n        .addStringColumn(\"Foo\").addStringColumn(\"Bar\").addStringColumn(\"Letter\").addLongColumn(\"Baz\")\n        .addRow(\"Pinky\", \"pink\", \"B\", 8)\n        .addRow(\"Inky\", \"cyan\", \"C\", 9)\n        .addRow(\"Clyde\", \"orange\", \"D\", 10);\n\nDataFrame joining2 = new DataFrame(\"df2\")\n        .addStringColumn(\"Name\").addStringColumn(\"Color\").addStringColumn(\"Code\").addLongColumn(\"Number\")\n        .addRow(\"Grapefruit\", \"pink\", \"B\", 2)\n        .addRow(\"Orange\", \"orange\", \"D\", 4)\n        .addRow(\"Apple\", \"red\", \"A\", 1);\n\nDataFrame joined = joining1.outerJoin(joining2, Lists.immutable.of(\"Bar\", \"Letter\"), Lists.immutable.of(\"Color\", \"Code\"));\n```\n\n`joining1`\n\n|Foo | Bar | Letter | Baz|\n|---|---|---|---:|\n|\"Pinky\" | \"pink\" | \"B\" | 8|\n|\"Inky\" | \"cyan\" | \"C\" | 9|\n|\"Clyde\" | \"orange\" | \"D\" | 10|\n\n`joining2`\n\n|Name | Color | Code | Number|\n|---|---|---|---:|\n|\"Grapefruit\" | \"pink\" | \"B\" | 2|\n|\"Orange\" | \"orange\" | \"D\" | 4|\n|\"Apple\" | \"red\" | \"A\" | 1|\n\n`joined`\n\n|Foo | Bar | Letter | Baz | Name | Number|\n|---|---|---|---:|---|---:|\n|\"Inky\" | \"cyan\" | \"C\" | 9 | null | null|\n|\"Clyde\" | \"orange\" | \"D\" | 10 | \"Orange\" | 4|\n|\"Pinky\" | \"pink\" | \"B\" | 8 | \"Grapefruit\" | 2|\n|null | \"red\" | \"A\" | null | \"Apple\" | 1|\n\n#### Join With Complements\n```Java\nDataFrame sideA = new DataFrame(\"Side A\")\n        .addStringColumn(\"Key\").addLongColumn(\"Value\")\n        .addRow(\"A\", 1)\n        .addRow(\"B\", 2)\n        .addRow(\"X\", 3)\n        ;\n\nDataFrame sideB = new DataFrame(\"Side B\")\n        .addStringColumn(\"Id\").addLongColumn(\"Count\")\n        .addRow(\"X\", 30)\n        .addRow(\"B\", 10)\n        .addRow(\"C\", 20)\n        ;\n\nTriplet\u003cDataFrame\u003e result = sideA.joinWithComplements(sideB, Lists.immutable.of(\"Key\"), Lists.immutable.of(\"Id\"));\n```\n`sideA`\n\n|Key | Value|\n|---|---:|\n|\"A\" | 1|\n|\"B\" | 2|\n|\"X\" | 3|\n\n`sideB`\n\n|Id | Count|\n|---|---:|\n|\"X\" | 30|\n|\"B\" | 10|\n|\"C\" | 20|\n\nComplement of B in A:\n\n`result.getOne()`\n\n|Key | Value|\n|---|---:|\n|\"A\" | 1|\n\nIntersection (inner join) of A and B based on the key columns:\n\n`result.getTwo()`\n\n|Key | Value | Count|\n|---|---:|---:|\n|\"B\" | 2 | 10|\n|\"X\" | 3 | 30|\n\nComplement of A in B:\n\n`result.getThree()`\n\n|Id | Count|\n|---|---:|\n|\"C\" | 20|\n\n#### Lookup Join\n\n```Java\nDataFrame pets = new DataFrame(\"Pets\")\n        .addStringColumn(\"Name\").addLongColumn(\"Pet Kind Code\")\n        .addRow(\"Sweet Pea\", 1)\n        .addRow(\"Mittens\",   2)\n        .addRow(\"Spot\",      1)\n        .addRow(\"Eagly\",     5)\n        .addRow(\"Grzgxxch\", 99);\n\nDataFrame codes = new DataFrame(\"Pet Kinds\")\n        .addLongColumn(\"Code\").addStringColumn(\"Description\")\n        .addRow(1, \"Dog\")\n        .addRow(2, \"Cat\")\n        .addRow(5, \"Eagle\")\n        .addRow(7, \"Snake\");\n\npets.lookup(DfJoin.to(codes)\n        .match(\"Pet Kind Code\", \"Code\")\n        .select(\"Description\")\n        .ifAbsent(\"Unclear\"));\n```\n\n`pets`\n\n|Name | Pet Kind Code | Description|\n|---|---:|---|\n|\"Sweet Pea\" | 1 | \"Dog\"|\n|\"Mittens\" | 2 | \"Cat\"|\n|\"Spot\" | 1 | \"Dog\"|\n|\"Eagly\" | 5 | \"Eagle\"|\n|\"Grzgxxch\" | 99 | \"Unclear\"|\n\nThere is another option to do lookup using a more fluent API directly on the `DataFrame` class:\n\n```Java\npets.lookupIn(codes)\n    .match(\"Pet Kind Code\", \"Code\")\n    .select(\"Description\")\n    .ifAbsent(\"Unclear\")\n    .resolveLookup();\n```\n\nNote that if using this approach, once all the lookup parameters are specified you will need to call the `resolveLookup()` method to actually execute the lookup.\n\n### Pivot\n\nSay we have a data frame of individual donut sales like this:\n\n```Java\n  DataFrame donutSales = new DataFrame(\"Donut Shop Purchases\")\n          .addStringColumn(\"Customer\").addStringColumn(\"Month\").addStringColumn(\"Donut Type\")\n          .addLongColumn(\"Qty\").addDoubleColumn(\"Amount\")\n          .addRow(\"Alice\", \"Jan\", \"Blueberry\", 10, 10.00)\n          .addRow(\"Alice\", \"Feb\", \"Glazed\", 10, 12.00)\n          .addRow(\"Alice\", \"Feb\", \"Old Fashioned\", 10, 8.00)\n          .addRow(\"Alice\", \"Jan\", \"Blueberry\", 10, 10.00)\n          .addRow(\"Bob\", \"Jan\", \"Blueberry\", 5, 5.00)\n          .addRow(\"Bob\", \"Jan\", \"Pumpkin Spice\", 5, 10.00)\n          .addRow(\"Bob\", \"Jan\", \"Apple Cider\", 4, 4.40)\n          .addRow(\"Bob\", \"Mar\", \"Apple Cider\", 8, 8.80)\n          .addRow(\"Dave\", \"Jan\", \"Blueberry\", 10, 10.00)\n          .addRow(\"Dave\", \"Jan\", \"Old Fashioned\", 20, 16.00)\n          .addRow(\"Carol\", \"Jan\", \"Blueberry\", 6, 6.00)\n          .addRow(\"Carol\", \"Feb\", \"Old Fashioned\", 12, 9.60)\n          .addRow(\"Carol\", \"Mar\", \"Jelly\", 10, 15.00)\n          .addRow(\"Carol\", \"Jan\", \"Apple Cider\", 12, 13.20)\n          ;\n```\n\n`donutSales`\n\n|Customer | Month | Donut Type | Qty | Amount|\n|---|---|---|---:|---:|\n|\"Alice\" | \"Jan\" | \"Blueberry\" | 10 | 10.0000|\n|\"Alice\" | \"Feb\" | \"Glazed\" | 10 | 12.0000|\n|\"Alice\" | \"Feb\" | \"Old Fashioned\" | 10 | 8.0000|\n|\"Alice\" | \"Jan\" | \"Blueberry\" | 10 | 10.0000|\n|\"Bob\" | \"Jan\" | \"Blueberry\" | 5 | 5.0000|\n|\"Bob\" | \"Jan\" | \"Pumpkin Spice\" | 5 | 10.0000|\n|\"Bob\" | \"Jan\" | \"Apple Cider\" | 4 | 4.4000|\n|\"Bob\" | \"Mar\" | \"Apple Cider\" | 8 | 8.8000|\n|\"Dave\" | \"Jan\" | \"Blueberry\" | 10 | 10.0000|\n|\"Dave\" | \"Jan\" | \"Old Fashioned\" | 20 | 16.0000|\n|\"Carol\" | \"Jan\" | \"Blueberry\" | 6 | 6.0000|\n|\"Carol\" | \"Feb\" | \"Old Fashioned\" | 12 | 9.6000|\n|\"Carol\" | \"Mar\" | \"Jelly\" | 10 | 15.0000|\n|\"Carol\" | \"Jan\" | \"Apple Cider\" | 12 | 13.2000|\n\nNow we want to see total number of donuts purchased by each customer in each month. So the month when the sale happen becomes a column in the pivot table, the customer is the grouping criteria and the aggregate value is the number of donuts sold to this customer in this month.\n\n```Java\nDataFrame qtyByCustomerAndMonth = donutSales.pivot(\n        Lists.immutable.of(\"Customer\"),\n        \"Month\",\n        Lists.immutable.of(sum(\"Qty\")));\n```\n`qtyByCustomerAndMonth`\n\n|Customer | Jan | Feb | Mar|\n|---|---:|---:|---:|\n|\"Alice\" | 20 | 20 | 0|\n|\"Bob\" | 14 | 0 | 8|\n|\"Dave\" | 30 | 0 | 0|\n|\"Carol\" | 18 | 12 | 10|\n\nIt is possible to aggregate more than one value in a pivot table, then the pivot table will contain as many columns per each pivot column value as there are aggregations (in this case for each month there will be two columns):\n```Java\nDataFrame qtyAndAmountByCustomerAndMonth = donutSales.pivot(\n        Lists.immutable.of(\"Customer\"),\n        \"Month\",\n        Lists.immutable.of(sum(\"Qty\"), sum(\"Amount\")));\n```\n\n`qtyAndAmountByCustomerAndMonth`\n\n|Customer | Jan:Qty | Jan:Amount | Feb:Qty | Feb:Amount | Mar:Qty | Mar:Amount|\n|---|---:|---:|---:|---:|---:|---:|\n|\"Alice\" | 20 | 20.0000 | 20 | 20.0000 | 0 | 0.0000|\n|\"Bob\" | 14 | 19.4000 | 0 | 0.0000 | 8 | 8.8000|\n|\"Dave\" | 30 | 26.0000 | 0 | 0.0000 | 0 | 0.0000|\n|\"Carol\" | 18 | 19.2000 | 12 | 9.6000 | 10 | 15.0000|\n\n## Domain Specific Language\n\nThe framework supports a simple Domain Specific Language (DSL) for computed column expression and operations on data frames such as filtering.\n\n### Script\n\nA DSL script is a sequence of one or more statements (see below for the kinds of statements supported). The result of executing a script is the value of the last statement (or expression) that was executed in the script.\n\n### Value Types\n\nThe language supports variables and literal values of the following types:\n\n| DSL Type    | Corresponding\u003cbr\u003eJava Type | Notes                                                          |\n|-------------|----------------------------|----------------------------------------------------------------|\n| `STRING`    | `String`                   |                                                                |\n| `INT`       | `int`                      | 32-bit integer values (up to  2^31 – 1)                        |\n| `LONG`      | `long`                     | 64-bit integer values (up to 2^63 - 1)                         |\n| `FLOAT`     | `float`                    | 32-bit floating point values                                   |\n| `DOUBLE`    | `double`                   | 64-bit floating point values                                   |\n| `DECIMAL`   | `BigDecimal`               | arbitrary precision (for all practical reasons) numbers        |\n| `DATE`      | `LocalDate`                |                                                                |\n| `DATE_TIME` | `LocalDateTime`            |                                                                |\n| `BOOLEAN`   | `boolean`                  |                                                                |\n| `VECTOR`    | n/a                        | a list of values, a tuple (values could be of different types) |\n| `VOID`      | n/a                        | a singleton instance of this type represents `null` value      |\n\nThere is no implicit type conversion of values and variables to avoid errors, to minimize surprising results, and to fail early.\n\n### Literals\n\nThe following are examples of literals\n\n| Type        | Example                                                                                                                                                             |\n|-------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `STRING`    | `\"Hello\"`\u003cbr\u003e`'Abracadabra'`\u003cbr\u003e(both single and double quotes are supported)                                                                                       |\n| `LONG`      | `123`                                                                                                                                                               |\n| `DOUBLE`    | `123.456`                                                                                                                                                           |\n| `DECIMAL`   | There is no decimal literal per se, however there is a built-in function `toDecimal()` that lets specify decimal constants, e.g. `toDecimal(1234, 3)`               |\n| `DATE`      | There is no date literal per se, however there is a built-in function `toDate()` that lets specify date constants, e.g. `toDate(2021, 11, 25)`                      |\n| `DATE_TIME` | There is no date literal per se, however there is a built-in function `toDateTime()` that lets specify date/time constants, e.g. `toDate(2021, 11, 25, 12, 13, 14)` |\n| `VECTOR`    | `(1, 2, 3)` \u003cbr\u003e`('A', 'B', 'C')` \u003cbr\u003e`(x, x + 1, x + 2)`                                                                                                           |\n| `BOOLEAN`   | there are no literal of boolean type as there was no scenario where they would be required, however boolean variables and expressions are fully supported           |\n\n### Variables\n\nThe variables in the DSL are immutable - once assigned, the value of the variable cannot change. This helps avoid errors arising from reusing variables.\n\nA variable type is inferred at runtime and doesn't need to be declared.\n\nExamples:\n```\nx = 123\n123 in (x - 1, x, x + 1) ? 'in' : 'out'\na = \"Hello, \"\nb = \"there\"\nsubstr(a + b, 3)\n```\n\n### Expressions\n\n| Category          | Expression                   | Example                                                                                                                      |\n|-------------------|------------------------------|------------------------------------------------------------------------------------------------------------------------------|\n| Unary             | `-`\u003cbr\u003e`not`                 | `-123`\u003cbr\u003e`not (a \u003e b)`                                                                                                      |\n| Binary Arithmetic | `+` `-` `*` `/`              | `1 + 2`\u003cbr\u003e`unit_price * quantity`\u003cbr\u003estring concatenation:\u003cbr\u003e `\"Hello, \" + \"world!\"`                                       |\n| Comparison        | `\u003e` `\u003e=` `\u003c` `\u003c=` `==` `!=`  |                                                                                                                              |\n| Boolean           | `and`\u003cbr\u003e`or`\u003cbr\u003e`xor`       |                                                                                                                              |\n| Containment       | `in`\u003cbr\u003e`not in`             | vectors: \u003cbr\u003e`\"a\" in (\"a\", \"b\", \"c\")`\u003cbr\u003e`x not in (1, 2, 3)`\u003cbr\u003estrings:\u003cbr\u003e`'ello' in 'Hello!'`\u003cbr\u003e`\"bye\" not in \"Hello!\"` |\n| Empty             | `is empty`\u003cbr\u003e`is not empty` | `\"\" is empty`\u003cbr\u003e`'Hello' is not empty`\u003cbr\u003evectors:\u003cbr\u003e`(1, 2, 3) is not empty`\u003cbr\u003e`() is empty`                             |\n| Null check        | `is null`\u003cbr\u003e`is not null`   | `\"\" is null`\u003cbr\u003e`'Hello' is not null`\u003cbr\u003e`x is null ? 0.0 : abs(x)`                                                          |\n\n### Statements\n\nThe following statements are available:\n- assignment\n- conditional\n- a freestanding expression\n\n### Functions\n\nThere are two types of functions—intrinsic (built-in) and explicitly declared using the DSL `function` declaration.\n\nRecursion (direct or indirect) is not supported.\n\n#### Built-in functions\n\n| Function   | Usage                                                           |\n|------------|-----------------------------------------------------------------|\n| abs        | abs(number)                                                     |\n| contains   | contains(string, substring)                                     |\n| print      | print(value)                                                    |\n| println    | println(value)                                                  |\n| startsWith | startsWith(string, prefix)                                      |\n| substr     | substr(string, beginIndex[, endIndex])                          |\n| toDate     | toDate(string in the yyyy-mm-dd format)\u003cbr\u003etoDate(yyyy, mm, dd) |\n| toDateTime | toDateTime(yyyy, mm, dd, hh, mm, ss)                            |\n| toDouble   | toDouble(string)                                                |\n| toLong     | toLong(string)                                                  |\n| toDecimal  | toUpper(unscaledValue, scale)                                   |\n| toString   | toString(number)                                                |\n| toUpper    | toUpper(string)                                                 |\n| withinDays | withinDays(date1, date2, numberOfDays)                          |\n\n#### User Declared Function Example 1\n```\nfunction abs(x)\n{\n  if x \u003e 0 then\n    x\n  else\n    -x\n  endif\n}\nabs(-123)\n```\n\n#### User Declared Function Example 2\n```\nfunction hello()\n{\n  'Hello'\n}\n\nhello() + ' world!'\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmzakharov%2Fdataframe-ec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvmzakharov%2Fdataframe-ec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmzakharov%2Fdataframe-ec/lists"}