Package net.sansa_stack.hadoop.format.gson.json

Class RecordReaderJsonSequence

java.lang.Object

org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.LongWritable,T>

net.sansa_stack.hadoop.core.RecordReaderGenericBase<com.google.gson.JsonElement,com.google.gson.JsonElement,com.google.gson.JsonElement,com.google.gson.JsonElement>

net.sansa_stack.hadoop.format.gson.json.RecordReaderJsonSequence

All Implemented Interfaces:

Closeable, AutoCloseable

public class RecordReaderJsonSequence
extends RecordReaderGenericBase<com.google.gson.JsonElement,com.google.gson.JsonElement,com.google.gson.JsonElement,com.google.gson.JsonElement>

Nested Class Summary

Nested classes/interfaces inherited from class net.sansa_stack.hadoop.core.RecordReaderGenericBase

RecordReaderGenericBase.ReadTooFarException

Field Summary

Fields

Modifier and Type	Field	Description
protected com.google.gson.Gson	gson
protected static final CustomPattern	jsonFwdPattern
static final String	RECORD_MAXLENGTH_KEY
static final String	RECORD_MINLENGTH_KEY
static final String	RECORD_PROBECOUNT_KEY

Fields inherited from class net.sansa_stack.hadoop.core.RecordReaderGenericBase

accumulating, codec, currentKey, currentValue, datasetFlow, decompressor, EMPTY_BYTE_ARRAY, enableStats, isEncoded, isFirstSplit, maxExtraByteCount, maxRecordLength, maxRecordLengthKey, minRecordLength, minRecordLengthKey, postambleBytes, preambleBytes, probeElementCount, probeElementCountKey, probeRecordCount, probeRecordCountKey, rawStream, recordFlowCloseable, recordStartPattern, regionStartSearchReadOverRegionEnd, regionStartSearchReadOverSplitEnd, skipRecordCount, split, splitEnd, splitId, splitLength, splitName, splitStart, stream, tailByteBuffer, tailBytes, tailEltBuffer, tailElts, tailEltsTime, tailRecordOffset, totalEltCount, totalRecordCount

Constructor Summary

Constructors

Constructor	Description
RecordReaderJsonSequence()
RecordReaderJsonSequence(com.google.gson.Gson gson)
RecordReaderJsonSequence(RecordReaderConf conf, com.google.gson.Gson gson)

Method Summary

Modifier and Type	Method	Description
void	initialize(org.apache.hadoop.mapreduce.InputSplit inputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext context)	Read out config paramaters (prefixes, length thresholds, ...) and examine the codec in order to set an internal flag whether the stream will be encoded or not.
protected Stream<com.google.gson.JsonElement>	parse(InputStream in, boolean isProbe)	Create a flowable from the input stream.

Methods inherited from class net.sansa_stack.hadoop.core.RecordReaderGenericBase

abbreviate, abbreviate, abbreviateAsUTF8, aggregate, aggregate, close, convert, createMatcherFactory, createRecordFlow, detectTail, didHitSplitBound, effectiveInputStream, effectiveInputStreamSupp, findFirstPositionWithProbeSuccess, findNextRegion, getCurrentKey, getCurrentValue, getPos, getPosition, getProgress, getStats, initRecordFlow, lines, logClose, logUnexpectedClose, nextKeyValue, parseFromSeekable, prober, setStreamToInterval, unbufferedStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

RECORD_MINLENGTH_KEY

public static final String RECORD_MINLENGTH_KEY

See Also:

• Constant Field Values

RECORD_MAXLENGTH_KEY

public static final String RECORD_MAXLENGTH_KEY

See Also:

• Constant Field Values

RECORD_PROBECOUNT_KEY

public static final String RECORD_PROBECOUNT_KEY

See Also:

• Constant Field Values

jsonFwdPattern

protected static final CustomPattern jsonFwdPattern

gson

protected com.google.gson.Gson gson

Constructor Details

RecordReaderJsonSequence

public RecordReaderJsonSequence()

RecordReaderJsonSequence

public RecordReaderJsonSequence(com.google.gson.Gson gson)

RecordReaderJsonSequence

public RecordReaderJsonSequence(RecordReaderConf conf,
 com.google.gson.Gson gson)

Method Details

initialize

public void initialize(org.apache.hadoop.mapreduce.InputSplit inputSplit,
 org.apache.hadoop.mapreduce.TaskAttemptContext context)
                throws IOException

Description copied from class: RecordReaderGenericBase

Read out config paramaters (prefixes, length thresholds, ...) and examine the codec in order to set an internal flag whether the stream will be encoded or not.

Overrides:

initialize in class RecordReaderGenericBase<com.google.gson.JsonElement,com.google.gson.JsonElement,com.google.gson.JsonElement,com.google.gson.JsonElement>

Throws:

IOException

parse

protected Stream<com.google.gson.JsonElement> parse(InputStream in,
 boolean isProbe)

Description copied from class: RecordReaderGenericBase

Create a flowable from the input stream. The input stream may be incorrectly positioned in which case the Flowable is expected to indicate this by raising an error event.

Specified by:

parse in class RecordReaderGenericBase<com.google.gson.JsonElement,com.google.gson.JsonElement,com.google.gson.JsonElement,com.google.gson.JsonElement>

Parameters:

in - A supplier of input streams. May supply the same underlying stream on each call hence only at most a single stream should be taken from the supplier. Supplied streams are safe to use in try-with-resources blocks (possibly using CloseShieldInputStream). Taken streams should be closed by the client code.

isProbe - Whether the parser should be configured for probing. For example, it is desireable to suppress porse errors during probing. Also, for probing the parser may optimize itself for minimizing latency of yielding items rather than overall throughput.

Returns: