https://github.com/bbc/similarity
Calculate similarity between documents using TF-IDF weights
https://github.com/bbc/similarity
Last synced: 9 months ago
JSON representation
Calculate similarity between documents using TF-IDF weights
- Host: GitHub
- URL: https://github.com/bbc/similarity
- Owner: bbc
- License: other
- Created: 2011-07-15T13:53:20.000Z (over 14 years ago)
- Default Branch: master
- Last Pushed: 2024-11-27T15:14:26.000Z (about 1 year ago)
- Last Synced: 2025-03-29T05:06:12.936Z (10 months ago)
- Language: Ruby
- Homepage:
- Size: 68.4 KB
- Stars: 115
- Watchers: 31
- Forks: 26
- Open Issues: 5
-
Metadata Files:
- Readme: README.org
- License: COPYING
- Authors: AUTHORS
Awesome Lists containing this project
README
* Similarity
** Overview
A Ruby library for calculating the similarity between pieces of text
using a [[http://en.wikipedia.org/wiki/Tf–idf][Term Frequency-Inverse Document Frequency]] method.
A [[http://en.wikipedia.org/wiki/Bag_of_words_model][bag of words]] model is used. Terms in the source documents are
downcased and punctuation is removed, but stemming is not currently
implemented.
This library was written to facilitate the creation of diagrams talked
about by Jonathan Stray in his
[[http://jonathanstray.com/a-full-text-visualization-of-the-iraq-war-logs][full-text
visualization of the Iraq War Logs]] post. An example of how to
generate a [[http://gephi.org/][Gephi]] compatible file including labelling of nodes with key
words is included in the =examples= directory.
The library depends on the [[http://www.gnu.org/software/gsl/][GNU Scientific Library]], and the [[http://rb-gsl.rubyforge.org/][gsl ruby
gem]] but does not use sparse matrix representations to speed up the
calculations, since there is no support for them in the GSL. I am
currently looking into fixing this, and would appreciate any help!
** Dependencies
Similarity depends on the [[http://www.gnu.org/software/gsl/][GNU Scientific Library]], and the [[http://rb-gsl.rubyforge.org/][gsl ruby
gem]]. On OSX with [[https://github.com/mxcl/homebrew]] the GSL can be
installed with
: brew install gsl
The =gsl= gem should then install normally. For other platforms,
please add the information to the wiki and I'll add them to this
readme.
** Usage
First we load some documents into the corpus
: require 'similarity'
:
: corpus = Corpus.new
:
: doc1 = Document.new(:content => "A document with a lot of additional words some of which are about chunky bacon")
: doc2 = Document.new(:content => "Another longer document with many words and again about chunky bacon")
: doc3 = Document.new(:content => "Some text that has nothing to do with pork products")
:
: [doc1, doc2, doc3].each { |doc| corpus << doc }
Then to compare documents we can use the =similar_documents= method
: corpus.similar_documents(doc1).each do |doc, similarity|
: puts "Similarity between doc #{doc1.id} and doc #{doc.id} is #{similarity}"
: end
:
: #=>
: Similarity between doc 70137042580340 and doc 70137042580340 is 0.9999999999999997
: Similarity between doc 70137042580340 and doc 70137042580240 is 0.06068602112714361
: Similarity between doc 70137042580340 and doc 70137042580160 is 0.04882114791611661
The cross-similarity matrix (useful for creating graphs) is also available
: similarity_matrix = corpus.similarity_matrix
For more examples, see the =examples= directory.
** Todo
- Performance improvements
- Switch to storing document vector spaces in sparse form, using linalg or csparse?
- (Optional) stemming of source terms
** Contributing
- Fork the project
- Send a pull request
- Don't touch the .gemspec, I'll do that when I release a new version
** Author
[[http://chrislowis.co.uk][Chris Lowis]] - BBC R&D