{"id":13526700,"url":"https://github.com/src-d/enry","last_synced_at":"2025-05-16T10:06:12.696Z","repository":{"id":46106918,"uuid":"63264762","full_name":"src-d/enry","owner":"src-d","description":"A faster file programming language detector","archived":false,"fork":false,"pushed_at":"2021-11-14T12:23:25.000Z","size":7503,"stargazers_count":459,"open_issues_count":22,"forks_count":50,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-05-05T05:05:11.165Z","etag":null,"topics":["cli","golang","java","language-detection","linguist"],"latest_commit_sha":null,"homepage":"https://blog.sourced.tech/post/enry/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/src-d.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null}},"created_at":"2016-07-13T17:06:24.000Z","updated_at":"2025-02-07T03:56:33.000Z","dependencies_parsed_at":"2022-08-30T19:22:29.055Z","dependency_job_id":null,"html_url":"https://github.com/src-d/enry","commit_stats":null,"previous_names":["src-d/simple-linguist"],"tags_count":29,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fenry","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fenry/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fenry/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fenry/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/src-d","download_url":"https://codeload.github.com/src-d/enry/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254509477,"owners_count":22082891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","golang","java","language-detection","linguist"],"created_at":"2024-08-01T06:01:33.392Z","updated_at":"2025-05-16T10:06:12.640Z","avatar_url":"https://github.com/src-d.png","language":"Go","readme":"## WE CONTINUE THE DEVELOPMENT AT [go-enry/go-enry](https://github.com/go-enry/go-enry). This repository is abandoned, and no further updates will be done on the code base, nor issue/prs will be answered or attended.\n\n# enry [![GoDoc](https://godoc.org/github.com/src-d/enry?status.svg)](https://godoc.org/github.com/src-d/enry) [![Build Status](https://travis-ci.com/src-d/enry.svg?branch=master)](https://travis-ci.com/src-d/enry) [![codecov](https://codecov.io/gh/src-d/enry/branch/master/graph/badge.svg)](https://codecov.io/gh/src-d/enry)\n\nProgramming language detector and toolbox to ignore binary or vendored files. *enry*, started as a port to _Go_ of the original [linguist](https://github.com/github/linguist) _Ruby_ library, that has an improved *2x performance*.\n\n* [CLI](#cli)\n* [Library](#library)\n    * [Go](#go)\n    * [Java bindings](#java-bindings)\n    * [Python bindings](#python-bindings)\n* [Divergences from linguist](#divergences-from-linguist)\n* [Benchmarks](#benchmarks)\n* [Why Enry?](#why-enry)\n* [Development](#development)\n    * [Sync with github/linguist upstream](#sync-with-githublinguist-upstream)\n* [Misc](#misc)\n* [License](#license)\n\n# CLI\n\nThe recommended way to install the `enry` command-line tool is to either\n[download a release](https://github.com/src-d/enry/releases) or run:\n\n```\n(cd \"$(mktemp -d)\" \u0026\u0026 go mod init enry \u0026\u0026 go get github.com/src-d/enry/v2/cmd/enry)\n```\n\n*enry* CLI accepts similar flags (`--breakdown/--json`) and produce an output, similar to *linguist*:\n\n```bash\n$ enry\n97.71%\tGo\n1.60%\tC\n0.31%\tShell\n0.22%\tJava\n0.07%\tRuby\n0.05%\tMakefile\n0.04%\tScala\n0.01%\tGnuplot\n```\n\nNote that enry's CLI **_does not need an actual git repository to work_**, which is an intentional difference from linguist.\n\n# Library\n\n*enry* is also available as a native Go library with FFI bindings for multiple programming languages.\n\n## Go\n\nIn a [Go module](https://github.com/golang/go/wiki/Modules),\nimport `enry` to the module by running:\n\n```go\ngo get github.com/src-d/enry/v2\n```\n\nThe rest of the examples will assume you have either done this or fetched the\nlibrary into your `GOPATH`.\n\n```go\n// The examples here and below assume you have imported the library.\nimport \"github.com/src-d/enry/v2\"\n\nlang, safe := enry.GetLanguageByExtension(\"foo.go\")\nfmt.Println(lang, safe)\n// result: Go true\n\nlang, safe := enry.GetLanguageByContent(\"foo.m\", []byte(\"\u003cmatlab-code\u003e\"))\nfmt.Println(lang, safe)\n// result: Matlab true\n\nlang, safe := enry.GetLanguageByContent(\"bar.m\", []byte(\"\u003cobjective-c-code\u003e\"))\nfmt.Println(lang, safe)\n// result: Objective-C true\n\n// all strategies together\nlang := enry.GetLanguage(\"foo.cpp\", []byte(\"\u003ccpp-code\u003e\"))\n// result: C++ true\n```\n\nNote that the returned boolean value `safe` is `true` if there is only one possible language detected.\n\nTo get a list of all possible languages for a given file, there is a plural version of the same API.\n\n```go\nlangs := enry.GetLanguages(\"foo.h\",  []byte(\"\u003ccpp-code\u003e\"))\n// result: []string{\"C\", \"C++\", \"Objective-C}\n\nlangs := enry.GetLanguagesByExtension(\"foo.asc\", []byte(\"\u003ccontent\u003e\"), nil)\n// result: []string{\"AGS Script\", \"AsciiDoc\", \"Public Key\"}\n\nlangs := enry.GetLanguagesByFilename(\"Gemfile\", []byte(\"\u003ccontent\u003e\"), []string{})\n// result: []string{\"Ruby\"}\n```\n\n## Java bindings\n\nGenerated Java bindings using a C shared library and JNI are available under [`java`](https://github.com/src-d/enry/blob/master/java).\n\nA library is published on Maven as [tech.sourced:enry-java](https://mvnrepository.com/artifact/tech.sourced/enry-java) for macOS and linux platforms. Windows support is planned under [src-d/enry#150](https://github.com/src-d/enry/issues/150).\n\n# Python bindings\n\nGenerated Python bindings using a C shared library and cffi are WIP under [src-d/enry#154](https://github.com/src-d/enry/issues/154).\n\nA library is going to be published on pypi as [enry](https://pypi.org/project/enry/) for\nmacOS and linux platforms. Windows support is planned under [src-d/enry#150](https://github.com/src-d/enry/issues/150).\n\nDivergences from linguist\n------------\n\nThe `enry` library is based on the data from `github/linguist` version **v7.5.1**.\n\nAs opposed to linguist, `enry` [CLI tool](#cli) does *not* require a full Git repository in the filesystem in order to report languages.\n\nParsing [linguist/samples](https://github.com/github/linguist/tree/master/samples) the following `enry` results are different from linguist:\n\n* [Heuristics for \".es\" extension](https://github.com/github/linguist/blob/e761f9b013e5b61161481fcb898b59721ee40e3d/lib/linguist/heuristics.yml#L103) in JavaScript could not be parsed, due to unsupported backreference in RE2 regexp engine.\n\n* [Heuristics for \".rno\" extension](https://github.com/github/linguist/blob/3a1bd3c3d3e741a8aaec4704f782e06f5cd2a00d/lib/linguist/heuristics.yml#L365) in RUNOFF could not be parsed, due to unsupported lookahead in RE2 regexp engine.\n\n* As of [Linguist v5.3.2](https://github.com/github/linguist/releases/tag/v5.3.2) it is using [flex-based scanner in C for tokenization](https://github.com/github/linguist/pull/3846). Enry still uses [extract_token](https://github.com/github/linguist/pull/3846/files#diff-d5179df0b71620e3fac4535cd1368d15L60) regex-based algorithm. See [#193](https://github.com/src-d/enry/issues/193).\n\n* Bayesian classifier can't distinguish \"SQL\" from \"PLpgSQL. See [#194](https://github.com/src-d/enry/issues/194).\n\n* Detection of [generated files](https://github.com/github/linguist/blob/bf95666fc15e49d556f2def4d0a85338423c25f3/lib/linguist/generated.rb#L53) is not supported yet.\n (Thus they are not excluded from CLI output). See [#213](https://github.com/src-d/enry/issues/213).\n\n* XML detection strategy is not implemented. See [#192](https://github.com/src-d/enry/issues/192).\n\n* Overriding languages and types though `.gitattributes` is not yet supported. See [#18](https://github.com/src-d/enry/issues/18).\n\n* `enry` CLI output does NOT exclude `.gitignore`ed files and git submodules, as linguist does\n\nIn all the cases above that have an issue number - we plan to update enry to match Linguist behavior.\n\n\nBenchmarks\n------------\n\nEnry's language detection has been compared with Linguist's on [*linguist/samples*](https://github.com/github/linguist/tree/master/samples).\n\nWe got these results:\n\n![histogram](benchmarks/histogram/distribution.png)\n\nThe histogram shows the _number of files_ (y-axis) per _time interval bucket_ (x-axis).\nMost of the files were detected faster by enry.\n\nThere are several cases where enry is slower than linguist due to\nGo regexp engine being slower than Ruby's on, wich is based on [oniguruma](https://github.com/kkos/oniguruma) library, written in C.\n\nSee [instructions](#misc) for running enry with oniguruma.\n\n\nWhy Enry?\n------------\n\nIn the movie [My Fair Lady](https://en.wikipedia.org/wiki/My_Fair_Lady), [Professor Henry Higgins](http://www.imdb.com/character/ch0011719/) is a linguist who at the very beginning of the movie enjoys guessing the origin of people based on their accent.\n\n\"Enry Iggins\" is how [Eliza Doolittle](http://www.imdb.com/character/ch0011720/), [pronounces](https://www.youtube.com/watch?v=pwNKyTktDIE) the name of the Professor.\n\n## Development\n\nTo build enry's CLI run:\n\n    make build\n\nthis will generate a binary in the project's root directory called `enry`.\n\nTo run the tests use:\n\n    make test\n\n\n### Sync with github/linguist upstream\n\n*enry* re-uses parts of the original [github/linguist](https://github.com/github/linguist) to generate internal data structures.\nIn order to update to the latest release of linguist do:\n\n```bash\n$ git clone https://github.com/github/linguist.git .linguist\n$ cd .linguist; git checkout \u003crelease-tag\u003e; cd ..\n\n# put the new release's commit sha in the generator_test.go (to re-generate .gold test fixtures)\n# https://github.com/src-d/enry/blob/13d3d66d37a87f23a013246a1b0678c9ee3d524b/internal/code-generator/generator/generator_test.go#L18\n\n$ make code-generate\n```\n\nTo stay in sync, enry needs to be updated when a new release of the linguist includes changes to any of the following files:\n\n* [languages.yml](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml)\n* [heuristics.yml](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.yml)\n* [vendor.yml](https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml)\n* [documentation.yml](https://github.com/github/linguist/blob/master/lib/linguist/documentation.yml)\n\nThere is no automation for detecting the changes in the linguist project, so this process above has to be done manually from time to time.\n\nWhen submitting a pull request syncing up to a new release, please make sure it only contains the changes in\nthe generated files (in [data](https://github.com/src-d/enry/blob/master/data) subdirectory).\n\nSeparating all the necessary \"manual\" code changes to a different PR that includes some background description and an update to the documentation on [\"divergences from linguist\"](##divergences-from-linguist) is very much appreciated as it simplifies the maintenance (review/release notes/etc).\n\n\n\n## Misc\n\n\u003cdetails\u003e\n  \u003csummary\u003eRunning a benchmark \u0026 faster regexp engine\u003c/summary\u003e\n\n### Benchmark\n\nAll benchmark scripts are in [*benchmarks*](https://github.com/src-d/enry/blob/master/benchmarks) directory.\n\n\n#### Dependencies\nAs benchmarks depend on Ruby and Github-Linguist gem make sure you have:\n - Ruby (e.g using [`rbenv`](https://github.com/rbenv/rbenv)), [`bundler`](https://bundler.io/) installed\n - Docker\n - [native dependencies](https://github.com/github/linguist/#dependencies) installed\n - Build the gem `cd .linguist \u0026\u0026 bundle install \u0026\u0026 rake build_gem \u0026\u0026 cd -`\n - Install it `gem install --no-rdoc --no-ri --local .linguist/github-linguist-*.gem`\n\n\n#### Quick benchmark\nTo run quicker benchmarks you can either:\n\n    make benchmarks\n\nto get average times for the main detection function and strategies for the whole samples set or:\n\n    make benchmarks-samples\n\nif you want to see measures per sample file.\n\n\n#### Full benchmark\nIf you want to reproduce the same benchmarks as reported above:\n - Make sure all [dependencies](#benchmark-dependencies) are installed\n - Install [gnuplot](http://gnuplot.info) (in order to plot the histogram)\n - Run `ENRY_TEST_REPO=\"$PWD/.linguist\" benchmarks/run.sh` (takes ~15h)\n\nIt will run the benchmarks for enry and linguist, parse the output, create csv files and plot the histogram.\n\n### Faster regexp engine (optional)\n\n[Oniguruma](https://github.com/kkos/oniguruma) is CRuby's regular expression engine.\nIt is very fast and performs better than the one built into Go runtime. *enry* supports swapping\nbetween those two engines thanks to [rubex](https://github.com/moovweb/rubex) project.\nThe typical overall speedup from using Oniguruma is 1.5-2x. However, it requires CGo and the external shared library.\nOn macOS with [Homebrew](https://brew.sh/), it is:\n\n```\nbrew install oniguruma\n```\n\nOn Ubuntu, it is\n\n```\nsudo apt install libonig-dev\n```\n\nTo build enry with Oniguruma regexps use the `oniguruma` build tag\n\n```\ngo get -v -t --tags oniguruma ./...\n```\n\nand then rebuild the project.\n\n\u003c/details\u003e\n\n\nLicense\n------------\n\nApache License, Version 2.0. See [LICENSE](LICENSE)\n","funding_links":[],"categories":["Go","Software"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrc-d%2Fenry","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsrc-d%2Fenry","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrc-d%2Fenry/lists"}