gate.creole.annic.lucene
Class LuceneTokenizer

java.lang.Object
  gate.creole.annic.apache.lucene.analysis.TokenStream
      gate.creole.annic.lucene.LuceneTokenizer

public class LuceneTokenizer
extends TokenStream

Implementation of token stream.

Author:

niraj

Field Summary

(package private) Document	document
(package private) ArrayList	featuresToExclude
(package private) int	pointer
(package private) ArrayList	tokens

Constructor Summary

LuceneTokenizer(ArrayList tokenStream)
Constructor

Method Summary

Token

next() Returns the next token in the token stream.

Methods inherited from class gate.creole.annic.apache.lucene.analysis.TokenStream

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

document

Document document

tokens

ArrayList tokens

featuresToExclude

ArrayList featuresToExclude

pointer

int pointer

Constructor Detail

LuceneTokenizer

public LuceneTokenizer(ArrayList tokenStream)

Constructor

Parameters:

tokenStream -

Method Detail

next

public Token next()
           throws IOException

Returns the next token in the token stream.

Specified by:

next in class TokenStream

Throws:

IOException