net.sansa_stack.hadoop.util

Class FileSplitUtils

java.lang.Object

net.sansa_stack.hadoop.util.FileSplitUtils

public class FileSplitUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
FileSplitUtils()

Method Summary

Modifier and Type	Method and Description
static List<org.apache.hadoop.mapreduce.InputSplit>	generateFileSplits(org.apache.hadoop.fs.Path testHadoopPath, long fileLengthTotal, int numSplits) Utility method to create a specific number of splits for a file.
static InputStream	getDecodedStreamFromSplit(org.apache.hadoop.mapreduce.lib.input.FileSplit split, org.apache.hadoop.conf.Configuration job) Util method to open a decoded stream from a split.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileSplitUtils

public FileSplitUtils()

Method Detail

getDecodedStreamFromSplit

public static InputStream getDecodedStreamFromSplit(org.apache.hadoop.mapreduce.lib.input.FileSplit split,
                                                    org.apache.hadoop.conf.Configuration job)
                                             throws IOException

Util method to open a decoded stream from a split. Useful to read out header information from the first split in order to put it into the hadoop configuration before starting parallel processing.

Parameters:

split -

job -

Returns:

Throws:

IOException

generateFileSplits

public static List<org.apache.hadoop.mapreduce.InputSplit> generateFileSplits(org.apache.hadoop.fs.Path testHadoopPath,
                                                                              long fileLengthTotal,
                                                                              int numSplits)
                                                                       throws IOException

Utility method to create a specific number of splits for a file.

Parameters:

testHadoopPath -

fileLengthTotal -

numSplits -

Returns:

Throws:

IOException