{"id":18419689,"url":"https://github.com/teradata/stampede","last_synced_at":"2025-04-07T13:31:39.471Z","repository":{"id":5648434,"uuid":"6857226","full_name":"Teradata/stampede","owner":"Teradata","description":"Stampede is the workflow tool for *nix that Cthulhu intended. It uses make for dependencies, bash for scripting, and cron for scheduling.","archived":false,"fork":false,"pushed_at":"2017-03-02T01:29:37.000Z","size":266,"stargazers_count":33,"open_issues_count":4,"forks_count":9,"subscribers_count":255,"default_branch":"master","last_synced_at":"2025-03-22T19:12:06.039Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Teradata.png","metadata":{"files":{"readme":"README.html","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-11-25T23:29:05.000Z","updated_at":"2022-03-20T21:46:53.000Z","dependencies_parsed_at":"2022-09-22T04:51:21.968Z","dependency_job_id":null,"html_url":"https://github.com/Teradata/stampede","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Teradata%2Fstampede","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Teradata%2Fstampede/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Teradata%2Fstampede/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Teradata%2Fstampede/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Teradata","download_url":"https://codeload.github.com/Teradata/stampede/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247661756,"owners_count":20975111,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T04:17:55.566Z","updated_at":"2025-04-07T13:31:38.840Z","avatar_url":"https://github.com/Teradata.png","language":"Shell","readme":"\u003c!DOCTYPE html\u003e  \n\u003chtml lang=\"en\"\u003e\n\u003chead\u003e\n  \u003cmeta charset=\"utf-8\"\u003e\n  \u003cmeta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\"\u003e\n  \u003ctitle\u003eREADME\u003c/title\u003e\n  \u003cmeta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\"\u003e\n  \u003cstyle\u003e\n/* \n   This document has been created with Marked.app \u003chttp://markedapp.com\u003e, Copyright 2011 Brett Terpstra\n   Please leave this notice in place, along with any additional credits below.\n   ---------------------------------------------------------------\n   Title: GitHub\n   Author: Brett Terpstra\n   Description: Github README style. Includes theme for Pygmentized code blocks.\n*/\nhtml,body{color:black}*{margin:0;padding:0}body{font:13.34px helvetica,arial,freesans,clean,sans-serif;-webkit-font-smoothing:antialiased;line-height:1.4;padding:3px;background:#fff;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px}p{margin:1em 0}a{color:#4183c4;text-decoration:none}#wrapper{background-color:#fff;border:3px solid #eee!important;padding:0 30px;margin:15px}#wrapper{font-size:14px;line-height:1.6}#wrapper\u003e*:first-child{margin-top:0!important}#wrapper\u003e*:last-child{margin-bottom:0!important}h1,h2,h3,h4,h5,h6{margin:0;padding:0}h1{margin:15px 0;padding-bottom:2px;font-size:24px;border-bottom:1px solid #eee}h2{margin:20px 0 10px 0;font-size:18px}h3{margin:20px 0 10px 0;padding-bottom:2px;font-size:14px;border-bottom:1px solid #ddd}h4{font-size:14px;line-height:26px;padding:18px 0 4px;font-weight:bold;text-transform:uppercase}h5{font-size:13px;line-height:26px;padding:14px 0 0;font-weight:bold;text-transform:uppercase}h6{color:#666;font-size:14px;line-height:26px;padding:18px 0 0;font-weight:normal;font-variant:italic}hr{background:transparent url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAYAAAAECAYAAACtBE5DAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAyJpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuMC1jMDYwIDYxLjEzNDc3NywgMjAxMC8wMi8xMi0xNzozMjowMCAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bWxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RSZWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZVJlZiMiIHhtcDpDcmVhdG9yVG9vbD0iQWRvYmUgUGhvdG9zaG9wIENTNSBNYWNpbnRvc2giIHhtcE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6OENDRjNBN0E2NTZBMTFFMEI3QjRBODM4NzJDMjlGNDgiIHhtcE1NOkRvY3VtZW50SUQ9InhtcC5kaWQ6OENDRjNBN0I2NTZBMTFFMEI3QjRBODM4NzJDMjlGNDgiPiA8eG1wTU06RGVyaXZlZEZyb20gc3RSZWY6aW5zdGFuY2VJRD0ieG1wLmlpZDo4Q0NGM0E3ODY1NkExMUUwQjdCNEE4Mzg3MkMyOUY0OCIgc3RSZWY6ZG9jdW1lbnRJRD0ieG1wLmRpZDo4Q0NGM0E3OTY1NkExMUUwQjdCNEE4Mzg3MkMyOUY0OCIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRGPiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/PqqezsUAAAAfSURBVHjaYmRABcYwBiM2QSA4y4hNEKYDQxAEAAIMAHNGAzhkPOlYAAAAAElFTkSuQmCC) repeat-x 0 0;border:0 none;color:#ccc;height:4px;margin:20px 0;padding:0}#wrapper\u003eh2:first-child,#wrapper\u003eh1:first-child,#wrapper\u003eh1:first-child+h2{border:0;margin:0;padding:0}#wrapper\u003eh3:first-child,#wrapper\u003eh4:first-child,#wrapper\u003eh5:first-child,#wrapper\u003eh6:first-child{margin:0;padding:0}h4+p,h5+p,h6+p{margin-top:0}li p.first{display:inline-block}ul,ol{margin:15px 0 15px 25px}ul li,ol li{margin-top:7px;margin-bottom:7px}ul li\u003e*:last-child,ol li\u003e*:last-child{margin-bottom:0}ul li\u003e*:first-child,ol li\u003e*:first-child{margin-top:0}#wrapper\u003eul,#wrapper\u003eol{margin-top:21px;margin-left:36px}dl{margin:0;padding:20px 0 0}dl dt{font-size:14px;font-weight:bold;line-height:normal;margin:0;padding:20px 0 0}dl dt:first-child{padding:0}dl dd{font-size:13px;margin:0;padding:3px 0 0}blockquote{margin:14px 0;border-left:4px solid #ddd;padding-left:11px;color:#555}table{border-collapse:collapse;margin:20px 0 0;padding:0}table tr{border-top:1px solid #ccc;background-color:#fff;margin:0;padding:0}table tr:nth-child(2n){background-color:#f8f8f8}table tr th,table tr td{border:1px solid #ccc;text-align:left;margin:0;padding:6px 13px}img{max-width:100%;height:auto}code,tt{margin:0 2px;padding:2px 5px;white-space:nowrap;border:1px solid #ccc;background-color:#f8f8f8;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;font-size:12px}pre\u003ecode{margin:0;padding:0;white-space:pre;border:0;background:transparent;font-size:13px}.highlight pre,pre{background-color:#f8f8f8;border:1px solid #ccc;font-size:13px;line-height:19px;overflow:auto;padding:6px 10px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px}#wrapper\u003epre,#wrapper\u003ediv.highlight{margin:10px 0 0}pre code,pre tt{background-color:transparent;border:0}#wrapper{background-color:#fff;border:1px solid #cacaca;padding:30px}.poetry pre{font-family:Georgia,Garamond,serif!important;font-style:italic;font-size:110%!important;line-height:1.6em;display:block;margin-left:1em}.poetry pre code{font-family:Georgia,Garamond,serif!important}sup,sub,a.footnote{font-size:1.4ex;height:0;line-height:1;vertical-align:super;position:relative}sub{vertical-align:sub;top:-1px}@media print{body{background:#fff}img,pre,blockquote,table,figure{page-break-inside:avoid}#wrapper{background:#fff;border:0}code{background-color:#fff;color:#444!important;padding:0 .2em;border:1px solid #dedede}pre code{background-color:#fff!important;overflow:visible}pre{background:#fff}}@media screen{body.inverted,.inverted #wrapper,.inverted hr .inverted p,.inverted td,.inverted li,.inverted h1,.inverted h2,.inverted h3,.inverted h4,.inverted h5,.inverted h6,.inverted th,.inverted .math,.inverted caption,.inverted dd,.inverted dt,.inverted blockquote{color:#eee!important;border-color:#555}.inverted td,.inverted th{background:#333}.inverted pre,.inverted code,.inverted tt{background:#444!important}.inverted h2{border-color:#555}.inverted hr{border-color:#777;border-width:1px!important}::selection{background:rgba(157,193,200,.5)}h1::selection{background-color:rgba(45,156,208,.3)}h2::selection{background-color:rgba(90,182,224,.3)}h3::selection,h4::selection,h5::selection,h6::selection,li::selection,ol::selection{background-color:rgba(133,201,232,.3)}code::selection{background-color:rgba(0,0,0,.7);color:#eee}code span::selection{background-color:rgba(0,0,0,.7)!important;color:#eee!important}a::selection{background-color:rgba(255,230,102,.2)}.inverted a::selection{background-color:rgba(255,230,102,.6)}td::selection,th::selection,caption::selection{background-color:rgba(180,237,95,.5)}.inverted{background:#0b2531}.inverted #wrapper,.inverted{background:rgba(37,42,42,1)}.inverted a{color:rgba(172,209,213,1)}}.highlight .c{color:#998;font-style:italic}.highlight .err{color:#a61717;background-color:#e3d2d2}.highlight .k{font-weight:bold}.highlight .o{font-weight:bold}.highlight .cm{color:#998;font-style:italic}.highlight .cp{color:#999;font-weight:bold}.highlight .c1{color:#998;font-style:italic}.highlight .cs{color:#999;font-weight:bold;font-style:italic}.highlight .gd{color:#000;background-color:#fdd}.highlight .gd .x{color:#000;background-color:#faa}.highlight .ge{font-style:italic}.highlight .gr{color:#a00}.highlight .gh{color:#999}.highlight .gi{color:#000;background-color:#dfd}.highlight .gi .x{color:#000;background-color:#afa}.highlight .go{color:#888}.highlight .gp{color:#555}.highlight .gs{font-weight:bold}.highlight .gu{color:#800080;font-weight:bold}.highlight .gt{color:#a00}.highlight .kc{font-weight:bold}.highlight .kd{font-weight:bold}.highlight .kn{font-weight:bold}.highlight .kp{font-weight:bold}.highlight .kr{font-weight:bold}.highlight .kt{color:#458;font-weight:bold}.highlight .m{color:#099}.highlight .s{color:#d14}.highlight .na{color:#008080}.highlight .nb{color:#0086b3}.highlight .nc{color:#458;font-weight:bold}.highlight .no{color:#008080}.highlight .ni{color:#800080}.highlight .ne{color:#900;font-weight:bold}.highlight .nf{color:#900;font-weight:bold}.highlight .nn{color:#555}.highlight .nt{color:#000080}.highlight .nv{color:#008080}.highlight .ow{font-weight:bold}.highlight .w{color:#bbb}.highlight .mf{color:#099}.highlight .mh{color:#099}.highlight .mi{color:#099}.highlight .mo{color:#099}.highlight .sb{color:#d14}.highlight .sc{color:#d14}.highlight .sd{color:#d14}.highlight .s2{color:#d14}.highlight .se{color:#d14}.highlight .sh{color:#d14}.highlight .si{color:#d14}.highlight .sx{color:#d14}.highlight .sr{color:#009926}.highlight .s1{color:#d14}.highlight .ss{color:#990073}.highlight .bp{color:#999}.highlight .vc{color:#008080}.highlight .vg{color:#008080}.highlight .vi{color:#008080}.highlight .il{color:#099}.highlight .gc{color:#999;background-color:#eaf2f5}.type-csharp .highlight .k{color:#00F}.type-csharp .highlight .kt{color:#00F}.type-csharp .highlight .nf{color:#000;font-weight:normal}.type-csharp .highlight .nc{color:#2b91af}.type-csharp .highlight .nn{color:#000}.type-csharp .highlight .s{color:#a31515}.type-csharp .highlight .sc{color:#a31515}\n\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody class=\"normal\"\u003e\n  \u003cdiv id=\"wrapper\"\u003e\n      \u003ch1 id=\"stampedereadme\"\u003eStampede README\u003c/h1\u003e\n\n\u003cp\u003eDean Wampler\u003cbr/\u003e\n\u003ca href=\"\u0026#x6d;\u0026#x61;\u0026#x69;\u0026#x6c;\u0026#x74;\u0026#x6f;\u0026#x3a;\u0026#x64;\u0026#x65;\u0026#x61;\u0026#110;\u0026#46;\u0026#x77;\u0026#x61;\u0026#x6d;\u0026#112;\u0026#108;\u0026#x65;\u0026#x72;\u0026#64;\u0026#x74;\u0026#x68;\u0026#x69;\u0026#x6e;\u0026#107;\u0026#x62;\u0026#105;\u0026#103;\u0026#x61;\u0026#110;\u0026#97;\u0026#108;\u0026#x79;\u0026#116;\u0026#x69;\u0026#x63;\u0026#x73;\u0026#46;\u0026#99;\u0026#111;\u0026#109;\"\u003e\u0026#100;\u0026#101;\u0026#97;\u0026#110;\u0026#46;\u0026#119;\u0026#x61;\u0026#109;\u0026#112;\u0026#108;\u0026#101;\u0026#114;\u0026#x40;\u0026#x74;\u0026#104;\u0026#x69;\u0026#x6e;\u0026#x6b;\u0026#98;\u0026#105;\u0026#103;\u0026#x61;\u0026#110;\u0026#x61;\u0026#108;\u0026#121;\u0026#116;\u0026#x69;\u0026#99;\u0026#115;\u0026#x2e;\u0026#x63;\u0026#x6f;\u0026#x6d;\u003c/a\u003e\u003cbr/\u003e\n\u003ca href=\"https://twitter.com/StampedeWkFlow\"\u003e@StampedeWkFlow\u003c/a\u003e\u003cbr/\u003e\nJanuary 8, 2013\u003c/p\u003e\n\n\u003cp\u003e\u003cem\u003eCopyright (c) 2011\u0026#8211;2013, Think Big Analytics, Inc. All Rights Reserved.\u003c/em\u003e\u003c/p\u003e\n\n\u003cp\u003eWelcome to \u003cem\u003eStampede\u003c/em\u003e, the workflow tool that works as \u003ca href=\"http://en.wikipedia.org/wiki/Cthulhu\"\u003eCthulhu\u003c/a\u003e intended for *nix systems, using \u003ccode\u003emake\u003c/code\u003e for dependency management and task seqeuencing, \u003ccode\u003ebash\u003c/code\u003e for scripting, and \u003ccode\u003ecron\u003c/code\u003e for scheduling.\u003c/p\u003e\n\n\u003cp\u003e\u003cem\u003eStampede\u003c/em\u003e originated as an alternative workflow tool for \u003ca href=\"http://hadoop.apache.org\"\u003eHadoop\u003c/a\u003e, but it is not limited to Hadoop scenarios.\u003c/p\u003e\n\n\u003cp\u003eIf you like \u003cem\u003eStampede\u003c/em\u003e, please consider joining the \u003ca href=\"https://groups.google.com/forum/#!forum/stampede-users\"\u003estampede-users\u003c/a\u003e Google group and following us on Twitter \u003ca href=\"https://twitter.com/StampedeWkFlow\"\u003e@StampedeWkFlow\u003c/a\u003e. Also, contributions in the form of patches are always welcome and appreciated.\u003c/p\u003e\n\n\u003ch2 id=\"installation\"\u003eInstallation\u003c/h2\u003e\n\n\u003cp\u003eFirst, clone this repo or expand the distribution archive somewhere useful, e.g., \u003ccode\u003e$HOME/stampede\u003c/code\u003e. \u003c/p\u003e\n\n\u003cp\u003eSince \u003cem\u003eStampede\u003c/em\u003e uses \u003ccode\u003emake\u003c/code\u003e and \u003ccode\u003ebash\u003c/code\u003e as its weapon\u0026#8217;s of choice, run this \u003ccode\u003emake\u003c/code\u003e command to test \u003cem\u003eStampede\u003c/em\u003e on your system and then install it:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003emake test install\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eThe \u003ccode\u003etests\u003c/code\u003e target is not required, but we recommend it as a sanity check for your environment. The \u003ccode\u003einstall\u003c/code\u003e target will ask you for details like the target installation directory (the default is \u003ccode\u003e/usr/local/stampede\u003c/code\u003e).\u003c/p\u003e\n\n\u003cp\u003eIf you \u003cstrong\u003edon\u0026#8217;t\u003c/strong\u003e have \u003ccode\u003esyslog\u003c/code\u003e on your system, run this command instead, which will skip the \u003ccode\u003esyslog\u003c/code\u003e-related tests:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003emake test-core install\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eFinally, the \u003ccode\u003etest\u003c/code\u003e target does \u003cem\u003enot\u003c/em\u003e test the \u0026#8220;extras\u0026#8221; included with \u003cem\u003eStampede\u003c/em\u003e, currently limited to \u003ca href=\"http://hadoop.apache.org\"\u003eHadoop-specific\u003c/a\u003e tools. To test these tools, first ensure that \u003ccode\u003e$HADOOP_HOME\u003c/code\u003e is defined, then run this command: \u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003emake test-extras install\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eThe \u003ccode\u003einstall\u003c/code\u003e target installs everything, whether you want to use \u003ccode\u003esyslog\u003c/code\u003e and the \u0026#8220;extras\u0026#8221; or not. They are small and harmless, if left alone in a cold, dark room\u0026#8230; ;^)\u003c/p\u003e\n\n\u003cp\u003eNext, assuming you installed in \u003ccode\u003e/usr/local/stampede/\u003c/code\u003e, which we\u0026#8217;ll call \u003ccode\u003e$STAMPEDE_HOME\u003c/code\u003e from now on, add \u003ccode\u003e$STAMPEDE_HOME/bin\u003c/code\u003e to the \u003ccode\u003ePATH\u003c/code\u003e for any user who plans to use \u003cem\u003eStampede\u003c/em\u003e. Also, the installation will include *nix \u003ccode\u003eman\u003c/code\u003e pages, so add \u003ccode\u003e$STAMPEDE_HOME/man\u003c/code\u003e to the \u003ccode\u003eMANPATH\u003c/code\u003e.\u003c/p\u003e\n\n\u003cp\u003eAs part of the installation, the installer will ask you if you want a global \u003ccode\u003estampederc\u003c/code\u003e file installed in \u003ccode\u003e/etc\u003c/code\u003e, \u003ccode\u003e/etc/sysconfig\u003c/code\u003e, or somewhere else. All statements in this file are commented out. If you want to make global changes to \u003cem\u003eStampede\u0026#8217;s\u003c/em\u003e environment variables, edit this file appropriately. Note these \u0026#8220;rc\u0026#8221; files won\u0026#8217;t contain all the possible variables you can define, see \u003ccode\u003e$STAMPEDE_HOME/bin/env.sh\u003c/code\u003e for the complete list of variables, their default values, and comments that describe them.\u003c/p\u003e\n\n\u003cp\u003eSimilarly, if you told the installer to copy \u003ccode\u003estampederc\u003c/code\u003e file to \u003ccode\u003e$HOME/.stampederc\u003c/code\u003e, edit that file for your personal tasks.\u003c/p\u003e\n\n\u003cp\u003eWhenever you create a new \u003cem\u003eStampede\u003c/em\u003e project, it will also get its own \u003ccode\u003e$PROJECT_HOME/.stampederc\u003c/code\u003e file, as we discuss next.\u003c/p\u003e\n\n\u003ch3 id=\"buildingjavacomponents\"\u003eBuilding Java Components\u003c/h3\u003e\n\n\u003cp\u003eAs of this release, there is a small Hadoop application written in Java, in \u003ccode\u003esrc/hadop/mapreduce-configuration\u003c/code\u003e. It is used by the \u003ccode\u003ebin/hadoop/mapreduce-prop\u003c/code\u003e command. For your convenience, a pre-built jar file is already provided. However, it is built with Java 1.6 (for maximum portability) and Hadoop v1.0.3. So, you may need to rebuild it if you use a different version of Hadoop or you want to use a newer version of Java. See \u003ccode\u003esrc/hadop/mapreduce-configuration/README.md\u003c/code\u003e for details.\u003c/p\u003e\n\n\u003ch2 id=\"usage\"\u003eUsage\u003c/h2\u003e\n\n\u003cp\u003eAn individual workflow definition is called a \u003cem\u003estampede\u003c/em\u003e. \u003c/p\u003e\n\n\u003cp\u003eTo create a stampede, run the following command:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003estampede create\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eIt will prompt you for properties such as the name of the stampede and the project\u0026#8217;s working directory.\u003c/p\u003e\n\n\u003cp\u003eEdit the \u003ccode\u003e.stampederc\u003c/code\u003e and \u003ccode\u003emakefile\u003c/code\u003e created in the project directory to define your workflow. See the \u003ccode\u003e$STAMPEDE_HOME/examples\u003c/code\u003e for ideas. Note that \u003ccode\u003e$STAMPEDE_HOME/bin\u003c/code\u003e contains helper scripts to ease the development of workflows. See also \u003ca href=\"MakeAndBashNotes.html\"\u003eMake and Bash Notes\u003c/a\u003e in this directory for some tips.\u003c/p\u003e\n\n\u003cp\u003eOnce a stampede has been created, you can invoke it using this command:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003estampede -f /path/to/makefile [options] [make_targets]\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eFor help on the \u003ccode\u003estampede\u003c/code\u003e options:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003estampede --help\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003ch2 id=\"requiredtools\"\u003eRequired Tools\u003c/h2\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003ebash\u003c/code\u003e v3+ - Because OS X ships with an older bash version, all the scripts supplied are v3 compatible. You can use newer constructs if your version of bash supports them.\u003c/li\u003e\n\u003cli\u003eGNU \u003ccode\u003emake\u003c/code\u003e v3.8+ - The \u003ccode\u003eMakefile\u003c/code\u003e in this directory that\u0026#8217;s used to test and install \u003cem\u003eStampede\u003c/em\u003e requires GNU \u003ccode\u003emake\u003c/code\u003e v3.8+, as do the \u003ccode\u003eexamples\u003c/code\u003e. However, you can adapt your project \u003ccode\u003eMakefiles\u003c/code\u003e to use any version of \u003ccode\u003emake\u003c/code\u003e you prefer.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ecron\u003c/code\u003e - If you plan to use \u003ccode\u003ecron\u003c/code\u003e for scheduling workflows. In fact, \u003cem\u003eStampede\u003c/em\u003e doesn\u0026#8217;t really do anything with \u003ccode\u003ecron\u003c/code\u003e itself; we just recommend that you use it first, before adopting something more heavyweight and proprietary. \u003cem\u003eStampede\u003c/em\u003e projects will work fine with any scheduling tool that can invoke shell commands.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003esyslog\u003c/code\u003e - If you plan to use the *nix logging facility \u003ccode\u003esyslog\u003c/code\u003e. See also the Installation section above.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://hadoop.apache.org\"\u003eHadoop\u003c/a\u003e - \u003cem\u003eStampede\u003c/em\u003e was originally designed as a flyweight option for \u003ca href=\"http://hadoop.apache.org\"\u003eHadoop\u003c/a\u003e. However, it is not restricted to Hadoop scenarios. All of the Hadoop support consists of helper scripts in \u003ccode\u003e$STAMPEDE_HOME/bin/hadoop\u003c/code\u003e (and corresponding tests and \u003ccode\u003eman\u003c/code\u003e pages).\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cp\u003e\u003cem\u003eStampede\u003c/em\u003e is mostly agnostic to tool versions. For any particular tool, including its own scripts, Stampede relies on finding the tool in the user\u0026#8217;s \u003ccode\u003ePATH\u003c/code\u003e.\u003c/p\u003e\n\n\u003ch2 id=\"supportedplatforms\"\u003eSupported Platforms\u003c/h2\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eLinux\u003c/strong\u003e - All recent Linux distributions with \u003ccode\u003ebash\u003c/code\u003e v3+. However, we have not tested all possible Linux variants!\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMac OS X\u003c/strong\u003e - All recent OS X versions.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003ch3 id=\"plannedsupport\"\u003ePlanned Support\u003c/h3\u003e\n\n\u003cp\u003eCurrently, \u003ccode\u003ecygwin\u003c/code\u003e and similar \u0026#8220;Unix on Windows\u0026#8221; toolkits are not supported, but only because we haven\u0026#8217;t tried them. We have tried to avoid any assumptions that would preclude this support. We welcome patches!\u003c/p\u003e\n\n\u003cp\u003eNote that as of this writing, support for running Hadoop in Windows environments was \u003ca href=\"https://www.hadooponazure.com/\"\u003ejust recently announced\u003c/a\u003e.\u003c/p\u003e\n\n\u003ch2 id=\"manifest\"\u003eManifest\u003c/h2\u003e\n\n\u003cp\u003eThe top-level directory contains the following files, in addition to directories that will be described next:\u003c/p\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eREADME.md\u003c/code\u003e - This file, as well as an HTML version of it.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eLICENSE\u003c/code\u003e - The copyright and license (Apache 2.0).\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eFAQs.md\u003c/code\u003e - Frequently-asked Questions.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eMakefile\u003c/code\u003e - The \u003ccode\u003emakefile\u003c/code\u003e used to test and install \u003cem\u003eStampede\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eVERSION\u003c/code\u003e - The version number, used by the \u003ccode\u003eMakefile\u003c/code\u003e for building releases.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ebin\u003c/code\u003e - See the following section.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003esrc\u003c/code\u003e - The directory for tools implemented with Java. The corresponding jars are prebuilt and included in the distribution, but if you want to build them yourself, see the \u003ccode\u003eREADME.md\u003c/code\u003e files in the corresponding directories under \u003ccode\u003esrc\u003c/code\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eman\u003c/code\u003e - *nix \u003ccode\u003eman\u003c/code\u003e pages for all the tools.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003etest\u003c/code\u003e - Tests for the tools.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003ch3 id=\"bindirectory\"\u003eBin Directory\u003c/h3\u003e\n\n\u003cp\u003e\u003cem\u003eStampede\u003c/em\u003e supplies helper \u003ccode\u003ebash\u003c/code\u003e scripts in the \u003ccode\u003ebin\u003c/code\u003e directory and \u0026#8220;extras\u0026#8221; for specific applications (e.g., \u003ca href=\"http://hadoop.apache.org\"\u003eHadoop\u003c/a\u003e) in subdirectories. All the scripts that end with \u003ccode\u003e.sh\u003c/code\u003e are used internally by \u003cem\u003eStampede\u003c/em\u003e. The files without this extension are user-callable utilities for building workflows.\u003c/p\u003e\n\n\u003cp\u003e\u003cem\u003eNOTE:\u003c/em\u003e All of these tools assume that \u003ccode\u003e$STAMPEDE_HOME\u003c/code\u003e is defined. This is true when they are called in a \u003cem\u003estampede\u003c/em\u003e workflow, e.g., a \u003ccode\u003eMakefile\u003c/code\u003e.\u003c/p\u003e\n\n\u003ch4 id=\"binutilities\"\u003e\u003ccode\u003ebin\u003c/code\u003e Utilities\u003c/h4\u003e\n\n\u003cp\u003eBriefly, here are the utilities in the \u003ccode\u003ebin\u003c/code\u003e directory. All support a \u003ccode\u003e--help\u003c/code\u003e option for more information:\u003c/p\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003estampede\u003c/code\u003e - The \u0026#8220;stampede\u0026#8221; (workflow) driver script. It can be invoked manually or by \u003ccode\u003ecron\u003c/code\u003e. It has several options to configure behavior. Run \u003ccode\u003estampede --help\u003c/code\u003e for details.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003einstall\u003c/code\u003e - Called by \u003ccode\u003emake install\u003c/code\u003e to install \u003cem\u003eStampede\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eabs-path\u003c/code\u003e - Return the absolute value for the specified paths.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ecreate-project\u003c/code\u003e - Called by the \u003ccode\u003estampede\u003c/code\u003e script to create new projects.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003edates\u003c/code\u003e - Format dates and perform date arithmetic in a platform-portable way.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003efind-tool\u003c/code\u003e - Locate the command for the specified \u003ccode\u003etool\u003c/code\u003e (e.g., \u003ccode\u003ecurl\u003c/code\u003e) in the user\u0026#8217;s path, several possible system directories, or in user-specified directories.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003efrom-log-level\u003c/code\u003e and \u003ccode\u003eto-log-level\u003c/code\u003e - Convert from a log-level string, e.g., \u003ccode\u003eDEBUG\u003c/code\u003e to the corresponding \u003ccode\u003esyslog\u003c/code\u003e-compatible number and back again.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eformat-log-message\u003c/code\u003e - Format messages that are logged. If you want to customize the format beyond what\u0026#8217;s possible by editing the environment variables \u003ccode\u003eSTAMPEDE_LOG_TIME_FORMAT\u003c/code\u003e and \u003ccode\u003eSTAMPEDE_LOG_MESSAGE_FORMAT_STRING\u003c/code\u003e (see \u003ccode\u003eenv.sh\u003c/code\u003e), you can create your own version of this script and drop it in \u003ccode\u003e$STAMPEDE_HOME/custom\u003c/code\u003e, which is on the \u003ccode\u003ePATH\u003c/code\u003e BEFORE \u003ccode\u003e$STAMPEDE_HOME/bin\u003c/code\u003e. See the \u003cem\u003eCustom\u003c/em\u003e section below for more details.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003einstall\u003c/code\u003e - Install \u003cem\u003eStampede\u003c/em\u003e on your system.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003elog-file\u003c/code\u003e - Return the name of the log file used by \u003cem\u003estampede\u003c/em\u003e or \u003ccode\u003eSYSLOG\u003c/code\u003e if \u003ccode\u003esyslog\u003c/code\u003e is being used.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003esend-email\u003c/code\u003e - Use the *nux \u003ccode\u003email\u003c/code\u003e command (if configured) to send alerts.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003esplit-string\u003c/code\u003e - Split a string on a delimiter and return an array or echo the elements to \u003ccode\u003estdout\u003c/code\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003estampede-log\u003c/code\u003e - Write your own messages to the log file (or \u003ccode\u003esyslog\u003c/code\u003e) configured for \u003cem\u003eStampede\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003esuccess-or-failure\u003c/code\u003e - Return one of two strings depending on a \u0026#8220;success\u0026#8221; flag.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003etrue-or-false\u003c/code\u003e - Return one of two strings depending on a whether a variable is empty or not.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eto-seconds\u003c/code\u003e - Return the number of seconds specified for an input number of seconds, minutes, or hours.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eto-time-interval\u003c/code\u003e - Like \u003ccode\u003eto-seconds\u003c/code\u003e, but returns a nicely formatted string.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003etry-for\u003c/code\u003e - Repeated attempt an operation for a specified duration of time, until success or timeout.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003etry-until\u003c/code\u003e - Like \u003ccode\u003etry-for\u003c/code\u003e, but tries until a user-specified timestamp.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ewaiting\u003c/code\u003e - A \u003ccode\u003esleep(1)\u003c/code\u003e wrapper with logging for use in loops, like the one in the \u003ccode\u003etry-*\u003c/code\u003e scripts.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eymd\u003c/code\u003e - Return the year, month, and day for the workflow\u0026#8217;s start time, which defaults to today\u0026#8217;s date.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eyesterday-ymd\u003c/code\u003e - Return the year, month, and day for the day before the workflow\u0026#8217;s start time, i.e., yesterday\u0026#8217;s date, by default. For example, if you need to process yesterday\u0026#8217;s data, this is a convenient way to compute the correct date.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cp\u003eThe following \u0026#8220;helper\u0026#8221; files are used by these scripts:\u003c/p\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eenv.sh\u003c/code\u003e - Defines global shell variables. Start here to find variables you can set in \u003ccode\u003erc\u003c/code\u003e files to configure behavior.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ecommon.sh\u003c/code\u003e - Many \u0026#8220;common\u0026#8221; \u003ccode\u003ebash\u003c/code\u003e functions used in several scripts.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003elog.sh\u003c/code\u003e - Support functions for logging.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003ch4 id=\"binhadooputilities\"\u003e\u003ccode\u003ebin/hadoop\u003c/code\u003e Utilities\u003c/h4\u003e\n\n\u003cp\u003e\u003ca href=\"http://hadoop.apache.org\"\u003eHadoop-specific\u003c/a\u003e helper tools are in the \u003ccode\u003ebin/hadoop\u003c/code\u003e directory. As for the \u003ccode\u003ebin\u003c/code\u003e scripts, use \u003ccode\u003e--help\u003c/code\u003e for more information on each tool.\u003c/p\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003emapreduce-prop\u003c/code\u003e - Return one or more property definitions for Hadoop MapReduce jobs.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ehive-prop\u003c/code\u003e - Return one or more property definitions for \u003ca href=\"http://hive.apache.org\"\u003eHive\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003epig-prop\u003c/code\u003e - Return one or more property definitions for \u003ca href=\"http://pig.apache.org\"\u003ePig\u003c/a\u003e.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cp\u003e\u003cstrong\u003eNote:\u003c/strong\u003e While the \u003ccode\u003e*-prop\u003c/code\u003e utilities behave similarly and take similar arguments, there are some differences\nin the results they produce that reflect differences in how they were implemented. See their \u003ccode\u003e*-prop --help\u003c/code\u003e messages or \u003ccode\u003eman\u003c/code\u003e pages for details.\u003c/p\u003e\n\n\u003ch3 id=\"customandcontribdirectories\"\u003eCustom and Contrib Directories\u003c/h3\u003e\n\n\u003cp\u003eIf you want to override the behavior of any particular script, drop a new version in the \u003ccode\u003ecustom\u003c/code\u003e directory (or a subdirectory), which are added to the \u003ccode\u003ePATH\u003c/code\u003e first.\u003c/p\u003e\n\n\u003cp\u003eWe intend for \u003ccode\u003econtrib\u003c/code\u003e to be a place where unsupported, community-contributed tools will go. This directory and any subdirectories will also be added to the path, after \u003ccode\u003ecustom\u003c/code\u003e and \u003ccode\u003ebin\u003c/code\u003e.\u003c/p\u003e\n\n\u003ch3 id=\"exampledirectory\"\u003eExample Directory\u003c/h3\u003e\n\n\u003cp\u003eThe \u003ccode\u003eexample\u003c/code\u003e directory contains example \u003cem\u003estampedes\u003c/em\u003e that you can adapt for your purposes as well as a sample configuration file.\u003c/p\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003ecrontab\u003c/code\u003e - A sample \u003ccode\u003ecrontab\u003c/code\u003e file.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003estampederc\u003c/code\u003e - A sample file that overrides environment variable definitions to customize the environment or a particular project. See \u003ccode\u003ebin/env.sh\u003c/code\u003e for recommendations on where to install one or more of these \u003ccode\u003erc\u003c/code\u003e files and for the complete list of variables available.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ehadoop\u003c/code\u003e - An example sequencing several Hadoop jobs into a typical ETL and analytics workflow.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003ch3 id=\"testdirectory\"\u003eTest Directory\u003c/h3\u003e\n\n\u003cp\u003eTests of \u003cem\u003eStampede\u003c/em\u003e itself are in the \u003ccode\u003etest\u003c/code\u003e directory. The tests provide good examples of the individual tools in action. To execute the tests, run \u003ccode\u003emake test\u003c/code\u003e. This \u003ccode\u003emake\u003c/code\u003e target won\u0026#8217;t run the \u0026#8220;extras\u0026#8221; tests, e.g., for Hadoop. To run \u003cem\u003eall\u003c/em\u003e tests, run \u003ccode\u003emake test-with-extras\u003c/code\u003e.\u003c/p\u003e\n\n\u003ch2 id=\"notes\"\u003eNotes\u003c/h2\u003e\n\n\u003cul\u003e\n\u003cli\u003eThe \u003ccode\u003ebin/send-email\u003c/code\u003e script requires the *nix mail service to be running on the server hosting the stampede.\u003c/li\u003e\n\u003cli\u003eSupporting both Linux and Mac \u003ccode\u003edate\u003c/code\u003e commands added some complexity to the code. We might consider deprecating support for the Mac \u003ccode\u003edate\u003c/code\u003e command (based on a BSD legacy) and instead require Mac users to install the Linux version of \u003ccode\u003edate\u003c/code\u003e using \u003ca href=\"http://mxcl.github.com/homebrew/\"\u003eHomebrew\u003c/a\u003e or \u003ca href=\"http://www.macports.org/\"\u003eMacPorts\u003c/a\u003e. Feedback welcome.\u003c/li\u003e\n\u003c/ul\u003e\n    \u003c/div\u003e\n\u003c/body\u003e\n\u003c/html\u003e","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteradata%2Fstampede","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fteradata%2Fstampede","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteradata%2Fstampede/lists"}