org.dice_research.topicmodeling.preprocessing.corpus

Class DocumentFrequencyBasedCorpusPreprocessor

java.lang.Object

org.dice_research.topicmodeling.preprocessing.corpus.DocumentFrequencyBasedCorpusPreprocessor

All Implemented Interfaces:

CorpusPreprocessor

public class DocumentFrequencyBasedCorpusPreprocessor
extends Object
implements CorpusPreprocessor

A preprocessor that removes words that occur either too often or too rare in the corpus from the DocumentTextWordIds of the documents and from the Vocabulary. It is assumed that the corpus has a CorpusVocabulary property.

Author:

Michael Röder (michael.roeder@uni-paderborn.de)

Field Summary

Fields

Modifier and Type	Field and Description
private int	maxDF
private int	minDF

Constructor Summary

Constructors

Constructor and Description
DocumentFrequencyBasedCorpusPreprocessor(int minDF, int maxDF)

Method Summary

Modifier and Type	Method and Description
static int[]	createWordIdMapping(org.dice_research.topicmodeling.utils.vocabulary.Vocabulary vocabulary, AtomicIntegerArray counts, int minDF, int maxDF)
org.dice_research.topicmodeling.utils.corpus.Corpus	preprocess(org.dice_research.topicmodeling.utils.corpus.Corpus corpus)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

minDF

private int minDF

maxDF

private int maxDF

Constructor Detail

DocumentFrequencyBasedCorpusPreprocessor

public DocumentFrequencyBasedCorpusPreprocessor(int minDF,
                                                int maxDF)

Method Detail

preprocess

public org.dice_research.topicmodeling.utils.corpus.Corpus preprocess(org.dice_research.topicmodeling.utils.corpus.Corpus corpus)

Specified by:

preprocess in interface CorpusPreprocessor

createWordIdMapping

public static int[] createWordIdMapping(org.dice_research.topicmodeling.utils.vocabulary.Vocabulary vocabulary,
                                        AtomicIntegerArray counts,
                                        int minDF,
                                        int maxDF)