{"id":15971158,"url":"https://github.com/glurp/rfilter","last_synced_at":"2026-05-13T12:44:48.940Z","repository":{"id":141875061,"uuid":"308476212","full_name":"glurp/rfilter","owner":"glurp","description":"programmable filter, as posix awq, with ruby syntaxe and embeddable function ","archived":false,"fork":false,"pushed_at":"2022-04-25T13:34:59.000Z","size":42,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-10T01:37:56.641Z","etag":null,"topics":["awq","bash","filter","plotting","ruby"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/glurp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-29T23:42:35.000Z","updated_at":"2022-02-02T14:58:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"257f563e-6a39-4743-83b9-64835ca83b57","html_url":"https://github.com/glurp/rfilter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glurp%2Frfilter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glurp%2Frfilter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glurp%2Frfilter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glurp%2Frfilter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/glurp","download_url":"https://codeload.github.com/glurp/rfilter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208018,"owners_count":20901568,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awq","bash","filter","plotting","ruby"],"created_at":"2024-10-07T20:08:08.788Z","updated_at":"2026-05-13T12:44:43.906Z","avatar_url":"https://github.com/glurp.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rfilter\n\nUsage :\n```\ncat log*.txt | rfilter 'pprint _1,_3,_.size,\" ==\u003e \",_0'\n#  extract first and third field, number of field,all line,  foreach line ( lines splitted with \\s+),\n#  _[0] == _1 ; _[1]==_2 . . .\n\nrfilter 'expression' log*.txt \n# same\n\nrfilter -F pattern -v a=b -v cc=dd 'expression' log*.txt\n#       pattern is used for splitting lines\n#       define $a with value 'b' and $cc='dd'\n\nrfilter  # without args\n  ....\u003e\u003e help\n```  \n\nTested on linux and Windows.\n\nWork (faster) with jruby.\nCode is optimised for scanning of GByte of logs :)\n\nFor plotting, gem 'gruff' is used.\n\n# mecanism for function with END action\n\nSome function need that something be done at the end of stdin.\n\nBy exemple, ```sum(_1)``` will sum  contents of first collumn, and wanted to print the value at the end of scan.\n+\nThey are 3 patterns for that :\n* eend {} : declare the bloc of instruction to be execute at the end\n* write some ruby data in $pending : at the end; $pending is pretty-printed\n* write in $pending a Hash with template : {type: :yourendname ,data: [...]}.\n  At the end, method 'eend_yourendname()' will be called. it can work with $pending[:data].\n\nSee statistics(i) function for simple usage of this pattern.\n\n# List of special functions\n\n```\nshow()                         : helper for debugging : show each field scanned by rfilter  on first line of stdin, and exit\neend(\u0026b)                       : specify a bloc to evaluate after end of input scanning\ntimediff(delta_max,date=nil)   : extract date from current line and compare with last one, call bloc if delta is \n                                 greater than delta_max, in seconds\nifdiff(field)                  : yield bloc if parameter value is distinct of last call value\n```\n\nExtract some strings, and print\n-------------------------------\n\n```\nclear()                        : clear screen\nextr(word)                     : extract first match in current line\natoi(str=nil)                  : return first number in scaned in current line (or in str parameter)\nafter(str,x=1,n=1)             : extract next word after a word in current line\nsum(i)                         : add all data in a column/data\nmult(i)                        : mutliply all data in a column\nstatistics(i)                  : register value, at end calculus and print mean/median/stddev\n```\n\nPloting\n-------\n\n\nplotting is global : one session generate one raster image, containing one zone plot with one or several curve/barrgraph.\n\nRaster file name generated, (in /tmp) is printed at the end.\n\n```\nplot(*v)                       : register some value in only-one curve, plot line at end\nsplot(name,value)              : append value to named curve, plot curves in one plot, at end\nrplot(name,lvalue,h=nil)       : append values in curve name, plot graph at end (one call \u003e\u003e one curve)\nbplot(label,*v)                : register x/y values (label and values) in curves, plot bargraphs at end (several call=\u003eseveral barrs)\n```\n\n\n\nsynthesis in Hashmap(s)\n-----------------------\n\n```\ntoh(a,b)                       : put in Hash h[a]=b ; print Hash at exit\ntohcount(a,b=\"1\")              : put in Hash h[a]+=b ; print Hash at exit\ntohlist(a,b=\"?\")               : push b in array in h[a] ; print Hash of Array at exit\ntohh(a,b,c)                    : put in Hash h[a][b]= c ; print Hash of hash at exit\ntohhcount(a,b,n=1)             : put in Hash h[a][b]+= n ; print Hash of hash at exit\n``````\n\n\nSelection and format\n--------------------\n\n```\npprint(*args)                  : print args in context : blank-separated, _1,_2... is understand, \ndelta(value,no=0)              : compare value with old-one, varname 'no'. so several delta() can be used in one session\n```\n\nstop/skip packet on stdin lines\n-------------------------------\n\n```\nskip_until(filter)             : do nothing until a line match ; eval bloc after this time \nstop_after(n)                  : stop n seconds after startup\nstop_nol(n)                    : stop if noline exceed parameter (print beginning and stop)\nstop_if(v)                     : stop if parameter is true\n```\n\n\n\n\n# simples Exemples\n\n```\ncat log*.txt | rfilter 'expression' \n```\n\nor\n```\nrfilter 'expression' *.txt`\n```\n\nCount number of file which have size bigger than 1K:\n```\nls -l | rfilter 'sell {_5.to_i\u003e1024}' | rfilter 'sum 1'\n```\n\nFormat lines\n\n```\nls -l | rfilter \"format('%-15s | %s',_9,_5)\" \n```\n\nextract data from json fragment, en sumerize at end\nA directorie have file contening this kind of fragment :\n```\n{\"Action\":\"/BootNotification\", \"chargeBoxIdentity\":\"FR*S33*E*2232*A-1\", \"chargePointModel\":\"eee\", \"chargePointSerialNumber\":\"89798 \"chargeBoxSerialNumber\":\"87909\", \"firmwareVersion\":\"23.44\", \"iccid\":\"888888888888888\", \"imsi\":\"999999999999\", \"date\":\"2018-01-13 20:16:55\"},\n. . . .\n```\n\nThis code create a CSV file contenning last value of iccid and imsi for each equipment Id (chargeboxId) :`\n```\ngrep iccid * | rfilter 'cbi=extr(/\"chargeBoxIdentity\":\"(.*?)\"/);\\\n                        iccid=extr(/\"iccid\":\"(.*?)\"/);\\\n                        imsi=extr(/\"imsi\":\"(.*?)\"/); \\\n                        toh(cbi,\"%s;%s\" % [iccid,imsi]) if iccid; \\\n                        eend { $pending.each {|k,v| puts \"#{k};#{v}\" }}' \u003e iccid_imsi.csv\n\n```\n\nExplanation:\n* extr extract field frem regexp\n* toh(k,v) memorise $pending[k]=v , so at the ending, $pending contain last value of each key 'k'\n* eend declare code to be executed : it scan $pending and format 'chargeboxId;iccid;imsi' foreach equipment in log.\n(the eend declaration is repeated for each line : stupid but efficace )\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglurp%2Frfilter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fglurp%2Frfilter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglurp%2Frfilter/lists"}