Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/raypereda/stemmify

Ruby module that converts a word to its approximate root form with the Porter stemmer. For example, observing and observation reduce to observ.
https://github.com/raypereda/stemmify

porter-stemmer-algorithm ruby stemmer

Last synced: about 21 hours ago
JSON representation

Ruby module that converts a word to its approximate root form with the Porter stemmer. For example, observing and observation reduce to observ.

Awesome Lists containing this project

README 

        
= Stemmify



== Description

This is a Rails gem for reducing words to their roots. For example, all the

following words to are stemmed to "observ", which is not a real word

in this case:

	observance

	observances

	observancy

	observant

	observants

	observation

	observe

	observed

	observer

	observers

	observing



The algorithm used here is based on the Porter stemmer.

You can read more about Martin Porter's stemmer at



http://tartarus.org/~martin/PorterStemmer/



Martin Porter explains:



	The Porter stemming algorithm (or ‘Porter stemmer’) is a process for removing

	the commoner morphological and inflexional endings from words in English. Its

	main use is as part of a term normalisation process that is usually done when

	setting up Information Retrieval systems.



== Install



  $ sudo gem install stemmify

or

  $ gem install stemmify



Note that while the source code is hosted on {github}[https://github.com/raypereda/stemmify],

the actual gem is hosted on {RubyGem.org}[https://rubygems.org/gems/stemmify].



== Usage



Let's say you are building some sort of search tool. You want

searches for "observations" and "observer" to all bring up

the same items. When you are building you index, you can

map all the words to their roots using the stem method.



Here's an example usage:



  require 'stemmify'

  print("observations".stem) # ==> "observ"



You can use stemmify as a command line tool:

  $ stem input_file.txt

  $ printf "I\nwalked\nin\nthe\nforest\n.\n" | stem



Both the file based input and the input taken from the stdin should be tokenized,

i.e. provide one token per line, tokens should be delimeted e.g. by a new line feed.



For tokenization consider using a tokenizer, for instance the

{tokenizer}[https://rubygems.org/gems/tokenizer] gem.



== Test Suite



This test is based on the sample input and output text from Martin Porter

website. It includes 23532 test words and their expected stem results.

To run the test, just type



  rake test



== Thanks



Dave Thomas from http://pragprog.com made speed improvements to this code

before it was packaged as a gem.



== License



Copyright (c) 2011 Ray Pereda



Permission is hereby granted, free of charge, to any person obtaining

a copy of this software and associated documentation files (the

"Software"), to deal in the Software without restriction, including

without limitation the rights to use, copy, modify, merge, publish,

distribute, sublicense, and/or sell copies of the Software, and to

permit persons to whom the Software is furnished to do so, subject to

the following conditions:



The above copyright notice and this permission notice shall be

included in all copies or substantial portions of the Software.



THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,

EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF

MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND

NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE

LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION

OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION

WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.