https://github.com/blizzy78/textsimilarity

Analyze files for copied and pasted (and possibly slightly modified) text
https://github.com/blizzy78/textsimilarity

copy-paste

Last synced: 6 months ago
JSON representation

Analyze files for copied and pasted (and possibly slightly modified) text

• Host: GitHub
• URL: https://github.com/blizzy78/textsimilarity
• Owner: blizzy78
• License: mit
• Created: 2021-10-31T13:27:16.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2025-08-16T17:10:38.000Z (11 months ago)
• Last Synced: 2025-08-16T19:06:27.522Z (11 months ago)
• Topics: copy-paste
• Language: Go
• Homepage:
• Size: 526 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
[![GoDoc](https://pkg.go.dev/badge/github.com/blizzy78/textsimilarity)](https://pkg.go.dev/github.com/blizzy78/textsimilarity)

textsimilarity

==============

A Go package to analyze files for copied and pasted (and possibly slightly modified) text.

This can be used to not only find exactly equal occurrences, but also occurrences that have slight modifications,

such as names of variables changed in source code.

Command Line Usage

------------------

The `cmd/textsimilarity/` folder provides a standalone command line utility. You can build it like this:

```

go build -o textsimilarity ./cmd/textsimilarity/

```

To install globally, run the following:

```

go install github.com/blizzy78/textsimilarity/cmd/textsimilarity@latest

```

You can also run it directly

```

go run github.com/blizzy78/textsimilarity/cmd/textsimilarity@latest ...

```

Usage Example

-------------

After installation, you can now run it against [restic] source code, with [icdiff] as the diff tool:

~~~bash

$ git clone https://github.com/restic/restic.git

$ cd restic

$ textsimilarity \

	-progress \

	-ignoreWS -ignoreBlank -minLen 6 -minLines 10 -maxDist 3 \

	-ignoreRE '^[ \t]*(\*|//)' \

	-alwaysDiffRE '^(package|import)' \

	-diffTool 'icdiff --cols=150 --tabsize=4 --no-headers -W {{.File1}} {{.File2}}' \

	-ignoreDiffToolRC \

	-printEqual \

	$(find . -type f -name '*\.go' -not \( -name '*_test.go' -o -name build.go -o -path '*helpers/*' \))

~~~

This will result in the following output:

[![Sample Output](similarities-thumb.png)](similarities.png)

_(Click to enlarge. This is only part of the output.)_

License

-------

This package is licensed under the MIT license.

[restic]: https://github.com/restic/restic

[icdiff]: https://github.com/jeffkaufman/icdiff