{"id":18261666,"url":"https://github.com/cwensel/cascading","last_synced_at":"2025-10-04T23:06:42.265Z","repository":{"id":495059,"uuid":"121672","full_name":"cwensel/cascading","owner":"cwensel","description":"Cascading is a feature rich API for defining and executing complex and fault tolerant data processing flows locally or on a cluster.","archived":false,"fork":false,"pushed_at":"2025-04-08T23:08:11.000Z","size":33673,"stargazers_count":349,"open_issues_count":9,"forks_count":221,"subscribers_count":33,"default_branch":"wip-4.6","last_synced_at":"2025-04-13T23:53:45.053Z","etag":null,"topics":["hadoop","java","mapreduce","tez"],"latest_commit_sha":null,"homepage":"http://cascading.wensel.net/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cwensel.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.txt","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.txt","dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["cwensel"]}},"created_at":"2009-02-04T17:52:06.000Z","updated_at":"2025-04-08T23:08:15.000Z","dependencies_parsed_at":"2023-07-07T01:16:22.702Z","dependency_job_id":"53570fc2-666a-4244-99ab-d307ad70d44e","html_url":"https://github.com/cwensel/cascading","commit_stats":{"total_commits":2414,"total_committers":31,"mean_commits":77.87096774193549,"dds":0.06752278376139187,"last_synced_commit":"246692bcd4a68bed73f64e5549d3c436abc23c2e"},"previous_names":[],"tags_count":834,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cwensel%2Fcascading","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cwensel%2Fcascading/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cwensel%2Fcascading/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cwensel%2Fcascading/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cwensel","download_url":"https://codeload.github.com/cwensel/cascading/tar.gz/refs/heads/wip-4.6","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254254042,"owners_count":22039792,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hadoop","java","mapreduce","tez"],"created_at":"2024-11-05T11:04:37.459Z","updated_at":"2025-10-04T23:06:37.227Z","avatar_url":"https://github.com/cwensel.png","language":"Java","funding_links":["https://github.com/sponsors/cwensel"],"categories":["大数据"],"sub_categories":[],"readme":"# Cascading\n\nThanks for using [Cascading](https://cascading.wensel.net/).\n\n## Cascading 4.6\n\nCascading 4.6 includes a few major changes and additions from prior major releases:\n\n* Support for Apache Hadoop 3. and Apache Tez 0.10.x\n* Removed Apache Hadoop 2.x and Apache Tez 0.8.x support\n\nCascading 4 includes a few major changes and additions from prior major releases:\n\n* Local mode support improved for production use\n* Moved the website to https://cascading.wensel.net/\n* Changed the Maven group name to `net.wensel` (from `cascading`)\n* Moved to GitHub Packages (WIPs) and Maven Central (releases)\n* Added native JSON support via the `cascading-nested-json` sub-project\n* Removed `cascading-xml` sub-project\n* Removed Apache Hadoop 1.x support\n* Imported deprecated Apache Parquet classes to retain support and provide enhancements\n\n## General Information:\n\nFor current WIP releases, go to: https://github.com/cwensel?tab=packages\u0026repo_name=cascading\n\nFor project documentation and community support, visit: [cascading.wensel.net](https://cascading.wensel.net/)\n\nThe project includes nine Cascading jar files:\n\n* `cascading-core-x.y.z.jar`                   - all Cascading Core class files\n* `cascading-expression-x.y.z.jar`             - all Cascading Janino expression operations class files\n* `cascading-nested-json-x.y.z.jar`            - all Cascading JSON operations\n* `cascading-nested-x.y.z.jar`                 - all Cascading base classes for nested data-type operations\n* `cascading-local-x.y.z.jar`                  - all Cascading Local in-memory mode class files\n* `cascading-local-kafka-x.y.z.jar`            - all Cascading Local support for Apache Kafka\n* `cascading-local-neo4j-x.y.z.jar`            - all Cascading Local support for Neo4j\n* `cascading-local-s3-x.y.z.jar`               - all Cascading Local support for AWS S3\n* `cascading-local-splunk-x.y.z.jar`           - all Cascading Local support for Splunk\n* `cascading-local-hadoop3-io-x.y.z.jar`       - all Cascading Local in-memory mode class files used with Hadoop\n* `cascading-hadoop3-common-x.y.z.jar`         - all Cascading Hadoop 3.x common class files\n* `cascading-hadoop3-io-x.y.z.jar`             - all Cascading Hadoop 3.x HDFS and IO related class files\n* `cascading-hadoop3-mr1-x.y.z.jar`            - all Cascading Hadoop 3.x MapReduce mode class files\n* `cascading-hadoop3-tez-x.y.z.jar`            - all Cascading Hadoop 3.x Tez mode class files\n* `cascading-hadoop3-tez-stats-x.y.z.jar`      - all Cascading Tez YARN timeline server class files\n* `cascading-hadoop3-parquet-x.y.z.jar`        - all Cascading Parquet class files\n* `cascading-hadoop3-parquet-thrift-x.y.z.jar` - all Cascading Parquet Thrift class files\n\nThese class jars, along with, tests, source and javadoc jars, are all available via the Maven repository.\n\nLocal mode is where the Cascading application will run locally in memory without cluster distribution. This\nimplementation has minimal to no robustness in low memory situations, by design.\n\nHadoop 3.x MR1 mode is for running on Hadoop 3.x releases.\n\nHadoop 3.x Tez mode is where the Cascading application should run on an Apache Tez *DAG* cluster.\n\nAs of Cascading 4.x, all above jar files are built against Java 1.8. Prior versions of Cascading are built against Java\n1.7 and 1.6.\n\n## Local Mode\n\nLocal mode has been much improved for production use in applications that do not need to run distributed across a\ncluster. Specifically in applications that trivially parallelize and run within AWS Lambda or Batch applications.\n\nSee https://github.com/cwensel/cascading-local for a collection of local mode integrations.\n\nNote this project will merge into Cascading in then next minor release.\n\n## Extensions, the SDK, and DSLs\n\nThere are a number of projects based on and extensions to Cascading available.\n\nVisit https://cascading.org/ for links. As these projects are updated to depend on 4.x, we will update the main site.\n\nNote many projects built and released against Cascading 3.x will work without modification with Cascading 4.x.\n\n## Versioning\n\nCascading stable releases are always of the form `x.y.z`, where `z` is the current maintenance release.\n\n`x.y.z` releases are maintenance releases. No public incompatible API changes will be made, but in an effort to fix\nbugs, remediation may entail throwing new Exceptions.\n\n`x.y` releases are minor releases. New features are added. No public incompatible API changes will be made on the core\nprocessing APIs (Pipes, Functions, etc), but in an effort to resolve inconsistencies, minor semantic changes may be\nnecessary.\n\nIt is important to note that *we do reserve to make breaking changes to the new query planner API through the 4.x\nreleases*. This allows us to respond to bugs and performance issues without issuing new major releases.\n\nAll releases will be maintained here:\n[https://github.com/cwensel/cascading](https://github.com/cwensel/cascading)\n\nWIP (work in progress) releases are fully tested builds of code not yet deemed fully stable. On every build by our\ncontinuous integration servers, the WIP build number is increased. Successful builds are then tagged and published.\n\nThe WIP releases are always of the form `x.y.z-wip-n`, where `x.y.z` will be the next stable release version the WIP\nreleases are leading up to. `n` is the current successfully tested build.\n\nThe source, working branches, and tags for all WIP releases can be found here:\n[https://github.com/cwensel/cascading](https://github.com/cwensel/cascading)\n\nWhen a WIP is deemed stable and ready for production use, it will be published as a `x.y.z` release, and made available\nfrom Maven Central.\n\n## Writing and Running Tests\n\nComprehensive tests should be written against the `cascading.PlatformTestCase`.\n\nWhen running tests built against the PlatformTestCase, the local cluster can be disabled (if enabled by the test)\nby setting:\n\n    -Dtest.cluster.enabled=false\n\nFrom Gradle, to run a single test case:\n\n    \u003e ./gradlew :cascading-hadoop3-mr1:platformTest --tests=*.FieldedPipesPlatformTest -i\n\nor a single test method:\n\n    \u003e ./gradlew :cascading-hadoop3-mr1:platformTest --tests=*.FieldedPipesPlatformTest.testNoGroup -i\n\n## Debugging the Planner\n\nWhen running tests, set the following\n\n    -Dtest.traceplan.enabled=true\n\nIf you are on Mac OS X and have installed GraphViz, dot files can be converted to pdf on the fly. To enable, set:\n\n    -Dutil.dot.to.pdf.enabled=true\n\nOptionally, for stand alone applications, statistics and tracing can be enabled selectively with the following\nproperties:\n\n* `cascading.planner.stats.path` - outputs detailed statistics on time spent by the planner\n* `cascading.planner.plan.path` - basic planner information\n* `cascading.planner.plan.transforms.path` - detailed information for each rule\n\n## Contributing and Reporting Issues\n\nSee __CONTRIBUTING.md__ at https://github.com/Cascading/cascading.\n\n## Using with Maven/Ivy\n\nIt is strongly recommended developers pull Cascading from Maven Central.\n\nAlternatively, see GitHub Packages for latest WIP releases:\n\n* https://maven.pkg.github.com/cwensel/cascading\n\nWhen creating tests, make sure to add any of the relevant above dependencies to your `test` scope or equivalent\nconfiguration along with the `cascading-platform` dependency.\n\nNote the `cascading-platform` compile dependency has no classes, you must pull the tests dependency with the\n`tests` classifier.\n\nSource and Javadoc artifacts (using the appropriate classifier) are also available through Maven.\n\nNote that `cascading-hadoop3-mr1`, and `cascading-hadoop3-tez` have a `provided` dependency on the Hadoop jars so that\nit won't get sucked into any application packaging as a dependency, typically.\n\n## Building and IDE Integration\n\nFor most cases, building Cascading is unnecessary as it has been pre-built, tested, and published to our Maven\nrepository (above).\n\nTo build Cascading, run the following in the shell:\n\n```bash\n\u003e git clone https://github.com/cwensel/cascading.git\n\u003e cd cascading\n\u003e ./gradlew build\n```\n\n## Using with Apache Hadoop\n\nFirst confirm you are using a supported version of Apache Hadoop by checking the\n[Compatibility](https://cascading.wensel.net/support/compatibility/) page.\n\nTo use Cascading with Hadoop, we suggest stuffing `cascading-core` and `cascading-hadoop3-mr1`, jar files and all\nthird-party libs into the `lib` folder of your job jar and executing your job via\n`$HADOOP_HOME/bin/hadoop jar your.jar \u003cyour args\u003e`.\n\nFor example, your job jar would look like this (via: `jar -t your.jar`)\n\n```bash\n/\u003call your class and resource files\u003e\n/lib/cascading-core-x.y.z.jar\n/lib/cascading-hadoop3-common-x.y.z.jar\n/lib/cascading-hadoop3-mr1-x.y.z.jar\n/lib/cascading-hadoop3-io-x.y.z.jar\n/lib/cascading-expression-x.y.z.jar\n/lib/cascading-nested-json-x.y.z.jar\n/lib/\u003ccascading third-party jar files\u003e\n```\n\nHadoop will unpack the jar locally and remotely (in the cluster) and add any libraries in `lib` to the classpath. This\nis a feature specific to Hadoop.\n\n## History and Status\n\nCascading was started in 2007 by Chris K Wensel.\n\nAfter a series of acquisitions, it was left unsupported and unmaintained by the copyright, domain name, and GitHub\norganization owners.\n\nChris has since continued his noodling with Cascading and has been pushing changes to the original repo.\n\nCascading remains under the Apache License v2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcwensel%2Fcascading","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcwensel%2Fcascading","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcwensel%2Fcascading/lists"}