edu.mit.jwi.morph
Class SimpleStemmer

java.lang.Object
  edu.mit.jwi.morph.SimpleStemmer

All Implemented Interfaces:

IStemmer

Direct Known Subclasses:

WordnetStemmer

public class SimpleStemmer
extends Object
implements IStemmer

Provides simple a simple pattern-based stemming facility based on the "Rules of Detachment" as described in the morphy man page in the Wordnet distribution, which can be found at http://wordnet.princeton.edu/man/morphy.7WN.html It also attempts to strip "ful" endings. It does not search Wordnet to see if stems actually exist. In particular, quoting from that man page:

Rules of Detachment

The following table shows the rules of detachment used by Morphy. If a word ends with one of the suffixes, it is stripped from the word and the corresponding ending is added. ... No rules are applicable to adverbs.

POS Suffix Ending

• NOUN "s" ""
• NOUN "ses" "s"
• NOUN "xes" "x"
• NOUN "zes" "z"
• NOUN "ches" "ch"
• NOUN "shes" "sh"
• NOUN "men" "man"
• NOUN "ies" "y"
• VERB "s" ""
• VERB "ies" "y"
• VERB "es" "e"
• VERB "es" ""
• VERB "ed" "e"
• VERB "ed" ""
• VERB "ing" "e"
• VERB "ing" ""
• ADJ "er" ""
• ADJ "est" ""
• ADJ "er" "e"
• ADJ "est" "e"

Special Processing for nouns ending with 'ful'

Morphy contains code that searches for nouns ending with ful and performs a transformation on the substring preceding it. It then appends 'ful' back onto the resulting string and returns it. For example, if passed the nouns "boxesful", it will return "boxful".

Since:

JWI 1.0

Version:

2.2.2

Author:

Mark A. Finlayson

Field Summary

static String	ENDING_ch
static String	ENDING_e
static String	ENDING_man
static String	ENDING_null
static String	ENDING_s
static String	ENDING_sh
static String	ENDING_x
static String	ENDING_y
static String	ENDING_z
static String	SUFFIX_ches
static String	SUFFIX_ed
static String	SUFFIX_er
static String	SUFFIX_es
static String	SUFFIX_est
static String	SUFFIX_ful
static String	SUFFIX_ies
static String	SUFFIX_ing
static String	SUFFIX_men
static String	SUFFIX_s
static String	SUFFIX_ses
static String	SUFFIX_shes
static String	SUFFIX_xes
static String	SUFFIX_zes
static String	underscore

Constructor Summary

SimpleStemmer()

Method Summary

List<String>	findStems(String word, POS pos) Takes the surface form of a word, as it appears in the text, and the assigned Wordnet part of speech.
protected List<String>	getNounCollocationRoots(String composite) Handles stemming noun collocations.
protected List<String>	getVerbCollocationRoots(String composite) Handles stemming verb collocations.
protected String	normalize(String word) Converts all whitespace runs to single underscores.
protected List<String>	stripAdjectiveSuffix(String adj) Strips suffixes from the specified word according to the adjective rules.
protected List<String>	stripNounSuffix(String noun) Strips suffixes from the specified word according to the noun rules.
protected List<String>	stripVerbSuffix(String verb) Strips suffixes from the specified word according to the verb rules.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

underscore

public static final String underscore

See Also:

Constant Field Values

SUFFIX_ches

public static final String SUFFIX_ches

See Also:

Constant Field Values

SUFFIX_ed

public static final String SUFFIX_ed

See Also:

Constant Field Values

SUFFIX_es

public static final String SUFFIX_es

See Also:

Constant Field Values

SUFFIX_est

public static final String SUFFIX_est

See Also:

Constant Field Values

SUFFIX_er

public static final String SUFFIX_er

See Also:

Constant Field Values

SUFFIX_ful

public static final String SUFFIX_ful

See Also:

Constant Field Values

SUFFIX_ies

public static final String SUFFIX_ies

See Also:

Constant Field Values

SUFFIX_ing

public static final String SUFFIX_ing

See Also:

Constant Field Values

SUFFIX_men

public static final String SUFFIX_men

See Also:

Constant Field Values

SUFFIX_s

public static final String SUFFIX_s

See Also:

Constant Field Values

SUFFIX_ses

public static final String SUFFIX_ses

See Also:

Constant Field Values

SUFFIX_shes

public static final String SUFFIX_shes

See Also:

Constant Field Values

SUFFIX_xes

public static final String SUFFIX_xes

See Also:

Constant Field Values

SUFFIX_zes

public static final String SUFFIX_zes

See Also:

Constant Field Values

ENDING_null

public static final String ENDING_null

See Also:

Constant Field Values

ENDING_ch

public static final String ENDING_ch

See Also:

Constant Field Values

ENDING_e

public static final String ENDING_e

See Also:

Constant Field Values

ENDING_man

public static final String ENDING_man

See Also:

Constant Field Values

ENDING_s

public static final String ENDING_s

See Also:

Constant Field Values

ENDING_sh

public static final String ENDING_sh

See Also:

Constant Field Values

ENDING_x

public static final String ENDING_x

See Also:

Constant Field Values

ENDING_y

public static final String ENDING_y

See Also:

Constant Field Values

ENDING_z

public static final String ENDING_z

See Also:

Constant Field Values

Constructor Detail

SimpleStemmer

public SimpleStemmer()

Method Detail

findStems

public List<String> findStems(String word,
                              POS pos)

Description copied from interface: IStemmer

Takes the surface form of a word, as it appears in the text, and the assigned Wordnet part of speech. The surface form may or may not contain whitespace or underscores, and may be in mixed case. The part of speech may be null, which means that all parts of speech should be considered. Returns a list of stems, in preferred order. No stem should be repeated in the list. If no stems are found, this call returns an empty list. It will never return null.

Specified by:

findStems in interface IStemmer

Parameters:

word - the surface form of which to find the stem

pos - the part of speech to find stems for; if null, find stems for all parts of speech

Returns:

the set of stems found for the surface form and part of speech combination

normalize

protected String normalize(String word)

Converts all whitespace runs to single underscores. Tests first to see if there is any whitespace before converting.

Parameters:

word - the string to be normalized

Returns:

a normalized string

Throws:

NullPointerException - if the specified string is null

IllegalArgumentException - if the specified string is empty or all whitespace

Since:

JWI 2.1.1

stripNounSuffix

protected List<String> stripNounSuffix(String noun)

Strips suffixes from the specified word according to the noun rules.

Parameters:

noun - the word to be modified

Returns:

a list of modified forms that were constructed, or the empty list if none

Throws:

NullPointerException - if the specified word is null

Since:

JWI 1.0

getNounCollocationRoots

protected List<String> getNounCollocationRoots(String composite)

Handles stemming noun collocations.

Parameters:

composite - the word to be modified

Returns:

a list of modified forms that were constructed, or the empty list if none

Throws:

NullPointerException - if the specified word is null

Since:

JWI 1.1.1

stripVerbSuffix

protected List<String> stripVerbSuffix(String verb)

Strips suffixes from the specified word according to the verb rules.

Parameters:

verb - the word to be modified

Returns:

a list of modified forms that were constructed, or the empty list if none

Throws:

NullPointerException - if the specified word is null

Since:

JWI 1.0

getVerbCollocationRoots

protected List<String> getVerbCollocationRoots(String composite)

Handles stemming verb collocations.

Parameters:

composite - the word to be modified

Returns:

a list of modified forms that were constructed, or an empty list if none

Throws:

NullPointerException - if the specified word is null

Since:

JWI 1.1.1

stripAdjectiveSuffix

protected List<String> stripAdjectiveSuffix(String adj)

Strips suffixes from the specified word according to the adjective rules.

Parameters:

adj - the word to be modified

Returns:

a list of modified forms that were constructed, or an empty list if none

Throws:

NullPointerException - if the specified word is null

Since:

JWI 1.0