org.dice_research.topicmodeling.preprocessing.docsupplier.decorator

Class HtmlCharsetExtractingSupplierDecorator

java.lang.Object

org.dice_research.topicmodeling.preprocessing.docsupplier.decorator.AbstractDocumentSupplierDecorator

org.dice_research.topicmodeling.preprocessing.docsupplier.decorator.HtmlCharsetExtractingSupplierDecorator

All Implemented Interfaces:

Function<org.dice_research.topicmodeling.utils.doc.Document,org.dice_research.topicmodeling.utils.doc.Document>, DocumentSupplierDecorator, org.dice_research.topicmodeling.preprocessing.docsupplier.DocumentSupplier

public class HtmlCharsetExtractingSupplierDecorator
extends AbstractDocumentSupplierDecorator

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HtmlCharsetExtractingSupplierDecorator.StringWithCharset

Field Summary

Fields

Modifier and Type	Field and Description
private static Charset	DEFAULT_CHARSET
private static org.slf4j.Logger	LOGGER

Fields inherited from class org.dice_research.topicmodeling.preprocessing.docsupplier.decorator.AbstractDocumentSupplierDecorator

documentSource

Constructor Summary

Constructors

Constructor and Description
HtmlCharsetExtractingSupplierDecorator(org.dice_research.topicmodeling.preprocessing.docsupplier.DocumentSupplier documentSource)

Method Summary

Modifier and Type	Method and Description
private HtmlCharsetExtractingSupplierDecorator.StringWithCharset	checkEncoding(HtmlCharsetExtractingSupplierDecorator.StringWithCharset text)
private Charset	extractCharset(String html)
private String	extractCharsetFromMetaTag(String html, int metaStart, int metaEnd)
private String	extractLowercasedHead(String html)
org.dice_research.topicmodeling.utils.doc.Document	prepareDocument(org.dice_research.topicmodeling.utils.doc.Document document)

Methods inherited from class org.dice_research.topicmodeling.preprocessing.docsupplier.decorator.AbstractDocumentSupplierDecorator

apply, getDecoratedDocumentSupplier, getNextDocument, setDecoratedDocumentSupplier, setDocumentStartId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dice_research.topicmodeling.preprocessing.docsupplier.DocumentSupplier

convertToStream

Methods inherited from interface java.util.function.Function

andThen, compose, identity

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

DEFAULT_CHARSET

private static final Charset DEFAULT_CHARSET

Constructor Detail

HtmlCharsetExtractingSupplierDecorator

public HtmlCharsetExtractingSupplierDecorator(org.dice_research.topicmodeling.preprocessing.docsupplier.DocumentSupplier documentSource)

Method Detail

prepareDocument

public org.dice_research.topicmodeling.utils.doc.Document prepareDocument(org.dice_research.topicmodeling.utils.doc.Document document)

Specified by:

prepareDocument in class AbstractDocumentSupplierDecorator

checkEncoding

private HtmlCharsetExtractingSupplierDecorator.StringWithCharset checkEncoding(HtmlCharsetExtractingSupplierDecorator.StringWithCharset text)

extractCharset

private Charset extractCharset(String html)

extractLowercasedHead

private String extractLowercasedHead(String html)

extractCharsetFromMetaTag

private String extractCharsetFromMetaTag(String html,
                                         int metaStart,
                                         int metaEnd)