{"id":15168442,"url":"https://github.com/darule0/yarndiff","last_synced_at":"2026-02-19T15:35:52.757Z","repository":{"id":216172499,"uuid":"424016418","full_name":"darule0/yarndiff","owner":"darule0","description":"A rudimentary command line utility for contrasting Apache Yarn container logs. ","archived":false,"fork":false,"pushed_at":"2024-01-08T19:06:55.000Z","size":61,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-29T19:59:19.052Z","etag":null,"topics":["diff","difference","diffing","hadoop","hadoop-mapreduce","hive","log4j","mapreduce","pig","spark","yarn","yarn2"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/darule0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-11-02T22:21:36.000Z","updated_at":"2024-03-13T11:41:06.000Z","dependencies_parsed_at":"2024-01-08T20:51:40.898Z","dependency_job_id":null,"html_url":"https://github.com/darule0/yarndiff","commit_stats":null,"previous_names":["darule0/yarndiff"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/darule0/yarndiff","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darule0%2Fyarndiff","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darule0%2Fyarndiff/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darule0%2Fyarndiff/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darule0%2Fyarndiff/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/darule0","download_url":"https://codeload.github.com/darule0/yarndiff/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darule0%2Fyarndiff/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29620144,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T13:04:20.082Z","status":"ssl_error","status_checked_at":"2026-02-19T13:03:33.775Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diff","difference","diffing","hadoop","hadoop-mapreduce","hive","log4j","mapreduce","pig","spark","yarn","yarn2"],"created_at":"2024-09-27T06:03:04.488Z","updated_at":"2026-02-19T15:35:52.742Z","avatar_url":"https://github.com/darule0.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# yarndiff \u003cyarn container log 1\u003e \u003cyarn container log 2\u003e\nA rudimentary command line utility for contrasting Apache Yarn container logs.\n\n## Motivation\nI have been troubleshooting Apache Yarn application issues full-time since around 2015. When a yarn application slows down or stops working, I try to find out more information such as: Were there any new errors or other kinds of log messages in the container logs which were not there before?\n\nYarn logs from two runs of the same application cannot be contrasted using a general purpose diff tool as there would be thousands of changes detected which are not useful for troubleshooting.\n\nI have decided to automate this part my job function in the form of a bash script which examines yarn logs and identifies differences which I find useful when troubleshooting yarn application performance and functionality problems. \n  \n## Description\nyarndiff is a Linux command line utility which contrasts yarn logs from two runs of a yarn application and displays a sample log entry for each kind of log entry that is found to be unique to either of the log files.\n\nFor example, if a yarn application has been running without problems for years and then suddenly slows down or stops working, then I will pass in the container logs from both a known working run as well as the container logs from the run which had problems. With a little luck, the yarndiff output helps guide me towards the root cause and solution.\n\n## Online Installation w/ CI\n```console\n\nmkdir ~/bin\nchmod u+rx ~/bin\nwget -O ~/bin/yarndiff https://github.com/darule0/yarndiff/blob/main/yarndiff?raw=true\nchmod u+rx ~/bin/yarndiff\nsource ~/.profile\n\n```\n\n\n\n## Offline Installation w/o CI\n```console\n\nsudo mkdir /opt/yarndiff\nsudo chmod o+rx /opt/yarndiff\nsudo git clone https://github.com/darule0/yarndiff.git /opt/yarndiff\nsudo chmod o+rx /opt/yarndiff/yarndiff.sh\nsudo ln -s /opt/yarndiff/yarndiff.sh /usr/bin/yarndiff\n\n```\n\n## How to obtain container logs for a yarn application run?\nThe logs switch on the yarn command can be used to obtain container logs for a recently run yarn application.\n```console\nyarn logs -applicationId APP_ID \u003e APP_ID_yarn.log\n```\n\n## Tutorial\n```console\n\n# install yarndiff w/ CI\nmkdir ~/bin\nchmod u+rx ~/bin\nwget -O ~/bin/yarndiff https://github.com/darule0/yarndiff/blob/main/yarndiff?raw=true\nchmod u+rx ~/bin/yarndiff\n\n# display yarndiff usage\nyarndiff\n\n# contrast container logs from a two runs of the same yarn application\nyarndiff container_log1 container_log2\n\n```\n\n![alt text](https://raw.githubusercontent.com/darule0/yarndiff/main/yarndiff.png)\n\n## Directories Used\n| directory | purpose |\n| :--- | :--- |\n| $HOME/.yarndiff.dd4b66ed-a43d-48ec-8e32-1b901bc8ea8e | The latest yarndiff is automatically downlaoded here when Online Installation w/ CI. |\n| $HOME/.yarndiff | Intermediate data for yarndiff processing. |\n\n## Container Log Parsing Logic\n| special entry | purpose |\n| :--- | :--- |\n| container.log.file | name of input log4j file |\n| container.count | number of lines which begin with the word \"Container: \"|\n\n| pseudo code: regular expression list generation derived from each container log file |\n| :--- |\n```console\nfor each line in a container log\n  keep only lines which contain CRIT or ERROR or WARN or INFO or DEBUG or TRACE\n  remove lines which contain the phrase \"has been replaced by\"\n  replace all of the following characters with a the wildcard character '.'\n      0123456789^$*+-?()[]{}|—/\\\\\n  keep only the first 80 characters\nsort the regular expressions\nremove any duplicate regular expressions\n```\n| pseudo code: contrast and use both regular expressions lists |\n| :--- |\n```console\ndiff regular expression lists\nfor both regular expression lists\n  obtain sample from applicable container log\n  obtain number of matches from applicable container log\n```\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdarule0%2Fyarndiff","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdarule0%2Fyarndiff","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdarule0%2Fyarndiff/lists"}