Package gate.creole.tokeniser

Class Summary

DefaultTokeniser	A composed tokeniser containing a SimpleTokeniser and a Transducer.
DFSMState	Implements a state of the deterministic finite state machine of the tokeniser.
FSMState	A state of the finite state machine that is the kernel tokeniser
SimpleTokeniser	Implementation of a Unicode rule based tokeniser.
TestTokeniser
UnicodeType	Used as an object wrapper that holds an Unicode type (the byte value of the static member of java.lang.Character).

Exception Summary

InvalidRuleException	Fired when an invalid tokeniser rule is found
TokeniserException	The top level exception for all the exceptions fired by the tokeniser