{"id":18031669,"url":"https://github.com/priv-kweihmann/tlv","last_synced_at":"2026-02-24T08:40:15.820Z","repository":{"id":57475943,"uuid":"193227583","full_name":"priv-kweihmann/tlv","owner":"priv-kweihmann","description":"Too less variation - A tool to discover code duplication in various languages","archived":false,"fork":false,"pushed_at":"2025-02-17T11:25:44.000Z","size":67,"stargazers_count":10,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-19T02:05:22.250Z","etag":null,"topics":["code-duplication","metrics","metrics-gathering","qatools","static-analysis"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/priv-kweihmann.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-22T11:43:50.000Z","updated_at":"2025-02-17T11:25:45.000Z","dependencies_parsed_at":"2023-02-10T14:16:58.930Z","dependency_job_id":"3a48da7c-1fe0-4e1f-ad57-0fb3ad28b4bb","html_url":"https://github.com/priv-kweihmann/tlv","commit_stats":{"total_commits":72,"total_committers":4,"mean_commits":18.0,"dds":0.5,"last_synced_commit":"baae5b136feaaa5591910f2a05bc712ff69b45b8"},"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/priv-kweihmann%2Ftlv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/priv-kweihmann%2Ftlv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/priv-kweihmann%2Ftlv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/priv-kweihmann%2Ftlv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/priv-kweihmann","download_url":"https://codeload.github.com/priv-kweihmann/tlv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245791328,"owners_count":20672664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-duplication","metrics","metrics-gathering","qatools","static-analysis"],"created_at":"2024-10-30T10:10:18.385Z","updated_at":"2026-02-24T08:40:15.758Z","avatar_url":"https://github.com/priv-kweihmann.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tlv\n\n![Build status](https://github.com/priv-kweihmann/tlv/workflows/Build/badge.svg)\n[![PyPI version](https://badge.fury.io/py/tlv.svg)](https://badge.fury.io/py/tlv)\n[![Python version](https://img.shields.io/pypi/pyversions/tlv)](https://img.shields.io/pypi/pyversions/tlv)\n[![Downloads](https://img.shields.io/pypi/dm/tlv)](https://img.shields.io/pypi/dm/tlv)\n\nToo less variation - A tool to discover code duplication in various languages\n\n## Purpose\n\nCode duplication are hard to track across many files.\nThere are tools existing like [CPD](https://pmd.github.io/latest/pmd_userdocs_cpd.html) (from PMD) which are doing a great job, but they require JAVA (or even some more obscure language) to run.\nSo I decided to write a small tool which uses [Pygments](http://pygments.org/) and python [difflib](https://docs.python.org/3/library/difflib.html) to catch code duplications for as many languages as possible\n\n## Requirements\n\n* python3\n* [Pygments](http://pygments.org/)\n\n## Installation\n\n### PyPi\n\nsimply run\n\n```sh\npip3 install tlv\n```\n\n### From source\n\n* Install the needed requirements by running ```pip3 install Pygments```\n* git clone this repository\n* cd to \\\u003cclone folder\\\u003e\n* run ```sudo ./build.sh```\n\n## Usage\n\n```shell\nusage: tlv [-h] [--minlines MINLINES] [--mintoken MINTOKEN] [-j JOBS]\n           [--lexer {...}]\n           [--wildcard_token {...}]\n           [--verbose] [--nodetails]\n           files [files ...]\n\nFind code duplications across various languages\npositional arguments:\n  files                 Files to parse\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --minlines MINLINES   Minimum length of block in lines before reporting\n  --mintoken MINTOKEN   Minimum length of block in token before reporting\n  -j JOBS, --jobs JOBS  Number of jobs to run in parallel\n  --lexer {3d,aap,actionscript,actionscript3,ada,ada2005,ada95,adl,agda,ahk,al,alloy,amienttalk,amienttalk/2,ampl,an,antlr,antlr-actionscript,antlr-as,antlr-c#,antlr-cpp,antlr-csharp,antlr-java,antlr-perl,antlr-python,antlr-r,antlr-ruy,apachecon,apl,applescript,arduino,arexx,as,as3,asemake,asic,asm,aspectj,aspx-cs,asy,asymptote,at,atch,augeas,autohotkey,autoit,awk,c,c#,c++,c++-ojdum,c-ojdump,ca65,cadl,camkes,capdl,capnp,casic,ceylon,chai,chaiscript,chapel,charmci,cheetah,chpl,cirru,cl,clay,clean,clj,cljs,clojure,clojurescript,cmake,cmas,co,code,common-lisp,componentpascal,console,control,cool,coq,cp,cpp,cpp-ojdump,cpsa,cr,crmsh,croc,cry,cryptol,crystal,csh,csharp,csound,csound-csd,csound-document,csound-orc,csound-sco,csound-score,css,css+django,css+er,css+genshi,css+genshitext,css+jinja,css+lasso,css+mako,css+mozpreproc,css+php,css+ruy,css+smarty,cu,cucumer,cuda,cxx-ojdump,cypher,cython,d,d-ojdump,dart,dasm16,decontrol,delphi,desources,dg,di,django,do,docker,dosatch,doscon,dpatch,dtd,duy,dylan,dylan-console,dylan-lid,dylan-repl,e,earl-grey,earlgrey,easytrieve,ec,ecl,eg,ei,elisp,elixir,elm,emacs,emacs-lisp,en,er,erl,erlang,evoque,ex,exs,extempore,ezhil,gap,gas,gawk,genshi,genshitext,gherkin,glsl,gnuplot,go,golo,gooddata-cl,gosu,gro,groovy,gst,haml,handlears,haskell,haxe,haxeml,hexdump,hlsl,hs,hsa,hsail,hspec,html,html+cheetah,html+django,html+er,html+evoque,html+genshi,html+handlears,html+jinja,html+kid,html+lasso,html+mako,html+myghty,html+ng2,html+php,html+ruy,html+smarty,html+spit,html+twig,htmldjango,http,hx,hxml,hxsl,hylang,i,i6t,idl,idl4,idr,idris,iex,igor,igorpro,ik,in,ini,io,ioke,ir,irc,isaelle,itex,j,jade,jags,jasmin,jasminxt,java,javascript,javascript+cheetah,javascript+django,javascript+er,javascript+genshi,javascript+genshitext,javascript+jinja,javascript+lasso,javascript+mako,javascript+myghty,javascript+php,javascript+ruy,javascript+smarty,jcl,jinja,jl,jlcon,jproperties,js,js+cheetah,js+django,js+er,js+genshi,js+genshitext,js+jinja,js+lasso,js+mako,js+myghty,js+php,js+ruy,js+smarty,js+spit,jsg,json,json-ld,json-oject,jsonld,jsp,julia,juttle,kal,kcon,kid,koka,kotlin,lagda,lasso,lassoscript,lcry,lcryptol,lean,less,lhaskell,lhs,lid,lidr,lidris,lighttpd,lighty,limo,liquid,lisp,literate-agda,literate-cryptol,literate-haskell,literate-idris,litzasic,litzmax,live-script,livescript,llvm,logos,logtalk,lua,m2,make,make,mako,maql,mask,mason,mathematica,matla,matlasession,mawk,max,md,minid,mma,modelica,modula2,moin,monkey,monte,moo,moocode,moon,moonscript,mq4,mq5,mql,mql4,mql5,msc,mscgen,mupad,mxml,myghty,mysql,n,nasm,nawk,ncl,nesc,newlisp,newspeak,ng2,nginx,nim,nimrod,nit,nix,nixos,nn,nsh,nsi,nsis,numpy,nusmv,oa,ocaml,octave,odin,oj-c,oj-c++,oj-j,ojc,ojc++,ojdump,ojdump-nasm,ojective-c,ojective-c++,ojective-j,ojectivec,ojectivec++,ojectivej,ojectpascal,ojj,oo,ooc,oogie,opa,openedge,openugs,pacmancon,pan,parasail,pas,pascal,pawn,pcmk,perl,perl6,php,php3,php4,php5,pig,pike,pkgcon,pl,pl6,plpgsql,plus,po,posh,postgres,postgres-console,postgresql,postgresql-console,postscr,postscript,pot,pov,powershell,praat,progress,prolog,properties,protou,ps1,ps1con,psm1,psql,pug,puppet,py,py3,py3t,pycon,pypy,pypylog,pyrex,pyt,python,python3,pyx,qasic,qml,qs,qvt,qvto,r,racket,ragel,ragel-c,ragel-cpp,ragel-d,ragel-em,ragel-java,ragel-ojc,ragel-r,ragel-ruy,rain,raw,rcon,rd,red,red/system,redcode,registry,reol,resource,resourceundle,rexx,rhtml,rkt,rnc,rng-compact,ro,roocon,root,rql,rs,rsl,rts,rust,ruy,s,sage,salt,sarl,sas,sass,satch,sc,scala,scaml,scheme,scila,scm,scss,shell-session,silver,slash,slim,sls,slurm,smali,smalltalk,smarty,sml,snool,sources.list,sourceslist,sp,sparql,spec,spit,splus,sql,sqlite3,squeak,squidcon,st,st-pytex,stan,stata,supercollider,sv,swi,swig,systemverilog,t-sql,tads3,tap,tasm,tcl,tcsh,tcshcon,tea,teraterm,teratermmacro,termcap,termin,terra,thri,todotxt,toml,tra,trac-wiki,treetop,ts,tsql,ttl,turtle,twig,typescript,typoscript,typoscriptcssdata,typoscripthtmldata,ucode,ugs,unicon,uriscript,v.net,vala,vapi,vcl,vclsnippet,vclsnippets,vctreestatus,velocity,vgl,vhdl,vim,vnet,vscript,wdi,whiley,winatch,winugs,x10,xml,xml+cheetah,xml+django,xml+er,xml+evoque,xml+genshi,xml+jinja,xml+kid,xml+lasso,xml+mako,xml+myghty,xml+php,xml+ruy,xml+smarty,xml+spit,xorg,xq,xql,xqm,xquery,xqy,xten,xtend,yaml,yaml+jinja}\n                        Manually set a lexer to use on all files\n  --wildcard_token {Token.Keyword,Token.Keyword.Constant,Token.Keyword.Declaration,Token.Keyword.Namespace,Token.Keyword.Pseudo,Token.Keyword.Reserved,Token.Keyword.Type,Token.Name,Token.Name.Attribute,Token.Name.Builtin,Token.Name.Builtin.Pseudo,Token.Name.Class,Token.Name.Constant,Token.Name.Decorator,Token.Name.Entity,Token.Name.Exception,Token.Name.Function,Token.Name.Function.Magic,Token.Name.Label,Token.Name.Namespace,Token.Name.Other,Token.Name.Tag,Token.Name.Variable,Token.Name.Variable.Class,Token.Name.Variable.Global,Token.Name.Variable.Instance,Token.Name.Variable.Magic,Token.Literal,Token.Literal.Date,Token.Literal.String,Token.Literal.String.Affix,Token.Literal.String.Backtick,Token.Literal.String.Char,Token.Literal.String.Delimiter,Token.Literal.String.Doc,Token.Literal.String.Double,Token.Literal.String.Escape,Token.Literal.String.Heredoc,Token.Literal.String.Interpol,Token.Literal.String.Other,Token.Literal.String.Regex,Token.Literal.String.Single,Token.Literal.String.Symbol,Token.Literal.Number,Token.Literal.Number.Bin,Token.Literal.Number.Float,Token.Literal.Number.Hex,Token.Literal.Number.Integer,Token.Literal.Number.Integer.Long,Token.Literal.Number.Oct,Token.Operator,Token.Operator.Word,Token.Punctuation,Token.Comment,Token.Comment.Hashbang,Token.Comment.Multiline,Token.Comment.Preproc,Token.Comment.Single,Token.Comment.Special,Token.Generic,Token.Generic.Deleted,Token.Generic.Emph,Token.Generic.Error,Token.Generic.Heading,Token.Generic.Inserted,Token.Generic.Output,Token.Generic.Prompt,Token.Generic.Strong,Token.Generic.Subheading,Token.Generic.Traceback,Token.Text.Whitespace}\n                        Token types that are threated as wildcards (actual\n                        value doesn't matter)\n  --verbose             Verbose output\n  --nodetails           Dump the details of a finding\n```\n\nBy default tool guesses the content type by the filename, if that doesn't work for you please see below\n\n### Specify a lexer\n\nYou can use a specific lexer by running the tool with the **--lexer=** option.\nWhen doing this all input files are processed by the specified lexer, you have to ensure that all passed files are of the specified file, else the results might be bogus.\n\n## Output\n\nOutput will be written to stdout.\nThere are 2 possible finding types\n\n* Duplicate - The code is exactly the same\n* TooLessVariation - The code is the same, when removing all token types specified by --wildcard_token\n\n### Output example\n\n```shell\n/someplace/busybox/modutils/modutils.c:9:0:[TooLessVariation]:Block till 16:0 is nearly the same as in ../modprobe-small.c from 29:0 till 36:0\n\u003e\u003e\u003e #include \u003csys/syscall.h\u003e\n\u003e\u003e\u003e \n\u003e\u003e\u003e #define init_module(mod, len, opts) syscall(__NR_init_module, mod, len, opts)\n\u003e\u003e\u003e #if defined(__NR_finit_module)\n\u003e\u003e\u003e # define finit_module(fd, uargs, flags) syscall(__NR_finit_module, fd, uargs, flags)\n\u003e\u003e\u003e #endif\n\u003e\u003e\u003e #define delete_module(mod, flags) syscall(__NR_delete_module, mod, flags)\n\u003e\u003e\u003e \n\u003c\u003c\u003c\n\u003e\u003e\u003e #include \u003csys/syscall.h\u003e\n\u003e\u003e\u003e \n\u003e\u003e\u003e #define init_module(mod, len, opts) syscall(__NR_init_module, mod, len, opts)\n\u003e\u003e\u003e #define delete_module(mod, flags) syscall(__NR_delete_module, mod, flags)\n\u003e\u003e\u003e #ifdef __NR_finit_module\n\u003e\u003e\u003e # define finit_module(fd, uargs, flags) syscall(__NR_finit_module, fd, uargs, flags)\n\u003e\u003e\u003e #endif\n\u003e\u003e\u003e \n\u003c\u003c\u003c\n/someplace/busybox/modutils/modutils.c:165:32:[Duplicate]:Block till 169:3 is the same as in ../modprobe-small.c from 324:18 till 327:0\n\u003e\u003e\u003e \n\u003e\u003e\u003e \tfstat(fd, \u0026st);\n\u003e\u003e\u003e \timage = NULL;\n\u003e\u003e\u003e \t/* st.st_size is off_t, we can't just pass it to mmap */\n\u003e\u003e\u003e \tif\n\u003c\u003c\u003c\n/someplace/busybox/modutils/modprobe-small.c:236:15:[Duplicate]:Block till 240:10 is the same as in ../modutils.c from 265:26 till 269:10\n\u003e\u003e\u003e \n\u003e\u003e\u003e \tcase ENOEXEC:\n\u003e\u003e\u003e \t\treturn \"invalid module format\";\n\u003e\u003e\u003e \tcase ENOENT:\n\u003e\u003e\u003e \t\treturn \"\n\u003c\u003c\u003c\n/someplace/busybox/modutils/modutils.c:115:0:[Duplicate]:Block till 130:43 is the same as in ../modprobe-small.c from 177:0 till 194:36\n\u003e\u003e\u003e \n\u003e\u003e\u003e #if ENABLE_FEATURE_CMDLINE_MODULE_OPTIONS\n\u003e\u003e\u003e char* FAST_FUNC parse_cmdline_module_options(char **argv, int quote_spaces)\n\u003e\u003e\u003e {\n\u003e\u003e\u003e \tchar *options;\n\u003e\u003e\u003e \tint optlen;\n\u003e\u003e\u003e \n\u003e\u003e\u003e \toptions = xzalloc(1);\n\u003e\u003e\u003e \toptlen = 0;\n\u003e\u003e\u003e \twhile (*++argv) {\n\u003e\u003e\u003e \t\tconst char *fmt;\n\u003e\u003e\u003e \t\tconst char *var;\n\u003e\u003e\u003e \t\tconst char *val;\n\u003e\u003e\u003e \n\u003e\u003e\u003e \t\tvar = *argv;\n\u003e\u003e\u003e \t\toptions = xrealloc(options, optlen + 2 + \n\u003c\u003c\u003c\n```\n\nif you don't want to see the code itself, pass the **--nodetails** option\n\n## Further reading\n\n* [Pygments](http://pygments.org/)\n\n## Bugs \u0026 Contribution\n\nFeel free to create issues or pull requests\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpriv-kweihmann%2Ftlv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpriv-kweihmann%2Ftlv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpriv-kweihmann%2Ftlv/lists"}