org.dice_research.squirrel.sink.impl.mem

Class InMemorySink

java.lang.Object

org.dice_research.squirrel.sink.impl.mem.InMemorySink

All Implemented Interfaces:

Closeable, AutoCloseable, Sink, SinkBase, TripleBasedSink, UnstructuredDataSink

public class InMemorySink
extends Object
implements Sink

This is a simple in-memory implementation of a sink that can be used for testing purposes. The data that is written to this sink can be accessed using the getCrawledRdfData() method. If the sink encounters a problem during its usage, e.g., data is written using a URI for which the stream already has been closed before, the sink becomes unhealthy. Note that this status does not influence the functionality of the sink. The status of the sink, i.e., whether it is healthy or not, can be accessed using the isSinkHealthy() method.

Author:

Michael Röder (roeder@informatik.uni-leipzig.de)

Field Summary

Fields

Modifier and Type	Field and Description
private Set<String>	closedSinks Set of URIs for which the sink has already been closed.
private boolean	healthyness The healthyness of the sink that is set to false if an error is encountered.
private static org.slf4j.Logger	LOGGER
private Map<String,org.apache.jena.rdf.model.Model>	rdfData In-memory map used to store the RDF data that is written to the sink.
private Map<String,List<byte[]>>	unstrcuturedData In-memory map used to store the unstructured data that is written to the sink.

Constructor Summary

Constructors

Constructor and Description
InMemorySink()

Method Summary

Modifier and Type	Method and Description
void	addData(CrawleableUri uri, byte[] data) Stores the given data for the given URI.
void	addData(CrawleableUri uri, InputStream stream) Stores the data from the given stream for the given URI.
void	addTriple(CrawleableUri uri, org.apache.jena.graph.Triple triple) Add a triple for the given uri.
void	closeSinkForUri(CrawleableUri uri) Closes the resources necessary for storing the data of the given URI.
Map<String,org.apache.jena.rdf.model.Model>	getCrawledRdfData() Returns the data written to the sink as a map with the crawled URI as key and the RDF data as value.
Map<String,List<byte[]>>	getCrawledUnstructuredData() Returns the data written to the sink as a map with the crawled URI as key and the unstructured data as value.
boolean	isSinkHealthy() Returns the status of the sink.
void	openSinkForUri(CrawleableUri uri) Opens the sink to process data for the given URI.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dice_research.squirrel.sink.Sink

addMetaData, close

Methods inherited from interface org.dice_research.squirrel.sink.UnstructuredDataSink

addData

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

rdfData

private Map<String,org.apache.jena.rdf.model.Model> rdfData

In-memory map used to store the RDF data that is written to the sink.

unstrcuturedData

private Map<String,List<byte[]>> unstrcuturedData

In-memory map used to store the unstructured data that is written to the sink.

closedSinks

private Set<String> closedSinks

Set of URIs for which the sink has already been closed.

healthyness

private boolean healthyness

The healthyness of the sink that is set to false if an error is encountered.

Constructor Detail

InMemorySink

public InMemorySink()

Method Detail

addTriple

public void addTriple(CrawleableUri uri,
                      org.apache.jena.graph.Triple triple)

Description copied from interface: TripleBasedSink

Add a triple for the given uri.

Specified by:

addTriple in interface TripleBasedSink

Parameters:

uri - The given uri.

triple - The triple to add.

openSinkForUri

public void openSinkForUri(CrawleableUri uri)

Description copied from interface: SinkBase

Opens the sink to process data for the given URI.

Specified by:

openSinkForUri in interface SinkBase

Parameters:

uri - the URI for which data should be stored.

closeSinkForUri

public void closeSinkForUri(CrawleableUri uri)

Description copied from interface: SinkBase

Closes the resources necessary for storing the data of the given URI.

Specified by:

closeSinkForUri in interface SinkBase

Parameters:

uri - the URI for which data has been stored and for which the resources should be freed.

getCrawledRdfData

public Map<String,org.apache.jena.rdf.model.Model> getCrawledRdfData()

Returns the data written to the sink as a map with the crawled URI as key and the RDF data as value.

Returns:

the data written to the sink.

getCrawledUnstructuredData

public Map<String,List<byte[]>> getCrawledUnstructuredData()

Returns the data written to the sink as a map with the crawled URI as key and the unstructured data as value.

Returns:

the data written to the sink.

isSinkHealthy

public boolean isSinkHealthy()

Returns the status of the sink.

Returns:

the status of the sink.

addData

public void addData(CrawleableUri uri,
                    byte[] data)

Description copied from interface: UnstructuredDataSink

Stores the given data for the given URI.

Specified by:

addData in interface UnstructuredDataSink

Parameters:

uri - the URI for which the data should be stored

data - the data that should be stored

addData

public void addData(CrawleableUri uri,
                    InputStream stream)

Description copied from interface: UnstructuredDataSink

Stores the data from the given stream for the given URI.

Specified by:

addData in interface UnstructuredDataSink

Parameters:

uri - the URI for which the data should be stored

stream - the stream from which the data will be read that should be stored