marmot.tokenize

Class RuleBasedTokenizer

java.lang.Object

marmot.tokenize.AbstractTokenizer

marmot.tokenize.RuleBasedTokenizer

All Implemented Interfaces:

java.io.Serializable, Tokenizer

public class RuleBasedTokenizer
extends AbstractTokenizer

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
RuleBasedTokenizer(Tokenizer tokenizer, RuleProvider provider)

Method Summary

Modifier and Type	Method and Description
java.util.List<java.lang.String>	tokenize(java.lang.String untokenized)

Methods inherited from class marmot.tokenize.AbstractTokenizer

saveToFile

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RuleBasedTokenizer

public RuleBasedTokenizer(Tokenizer tokenizer,
                          RuleProvider provider)

Method Detail

tokenize

public java.util.List<java.lang.String> tokenize(java.lang.String untokenized)