org.aksw.limes.core.evaluation.evaluator

Class Evaluator

java.lang.Object

org.aksw.limes.core.evaluation.evaluator.Evaluator

public class Evaluator
extends Object

This evaluator is responsible for evaluating set of datasets that have source, target, gold standard and mappings against set of measures

Since:

1.0

Version:

1.0

Author:

Mofeed Hassan (mounir@informatik.uni-leipzig.de), Daniel Obraczka (obraczka@studserv.uni-leipzig.de)

Field Summary

Fields

Modifier and Type	Field and Description
List<EvaluationRun>	runsList
Map<String,Map<String,Map<String,Double>>>	statisticalTestResults
Map<String,Map<String,int[]>>	successesAndFailures

Constructor Summary

Constructors

Constructor and Description
Evaluator()

Method Summary

Modifier and Type	Method and Description
ACache	cacheUnion(ACache a, ACache b)
Set<List<LearningParameter>>	createParameterGrid(Map<LearningParameter,List<Object>> parameters)
List<EvaluationRun>	crossValidate(AMLAlgorithm algorithm, List<LearningParameter> parameter, Set<TaskData> datasets, int foldNumber, Set<EvaluatorType> qlMeasures, Set<IQuantitativeMeasure> qnMeasures)
Summary	crossValidateWithTuningAndStatisticalTest(List<TaskAlgorithm> TaskAlgorithms, Set<TaskData> datasets, Set<EvaluatorType> qlMeasures, int foldNumber)
List<EvaluationRun>	evaluate(List<TaskAlgorithm> TaskAlgorithms, Set<TaskData> datasets, Set<EvaluatorType> QlMeasures, Set<IQuantitativeMeasure> QnMeasures)
List<FoldData>	generateFolds(EvaluationData data, int foldNumber, boolean withNegativeExamples)
List<AMapping>	generateMappingFolds(AMapping refMap, ACache source, ACache target, int foldNumber, boolean withNegativeExamples)
static String	getRandomTargetInstance(ACache source, ACache target, List<String> values, Random random, HashMap<String,HashMap<String,Double>> refMap, String sourceInstance, int previousRandom)
static void	main(String[] args)
AMapping	removeSubMap(AMapping mainMap, AMapping subMap)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

runsList

public List<EvaluationRun> runsList

successesAndFailures

public Map<String,Map<String,int[]>> successesAndFailures

statisticalTestResults

public Map<String,Map<String,Map<String,Double>>> statisticalTestResults

Constructor Detail

Evaluator

public Evaluator()

Method Detail

evaluate

public List<EvaluationRun> evaluate(List<TaskAlgorithm> TaskAlgorithms,
                                    Set<TaskData> datasets,
                                    Set<EvaluatorType> QlMeasures,
                                    Set<IQuantitativeMeasure> QnMeasures)

Parameters:

TaskAlgorithms - the set of algorithms used to generate the predicted mappings

datasets - the set of the datasets to apply the algorithms on them. The should include source Cache, target Cache, goldstandard and predicted mapping

QlMeasures - the set of qualitative measures

QnMeasures - the set of quantitative measures

Returns:

List - contains list of multiple runs evaluation results corresponding to the algorithms, its implementation and used dataset

crossValidate

public List<EvaluationRun> crossValidate(AMLAlgorithm algorithm,
                                         List<LearningParameter> parameter,
                                         Set<TaskData> datasets,
                                         int foldNumber,
                                         Set<EvaluatorType> qlMeasures,
                                         Set<IQuantitativeMeasure> qnMeasures)

Parameters:

algorithm - the algorithm used to generate the predicted mappings

datasets - the set of the datasets to apply the algorithms on them. The should include source Cache, target Cache, goldstandard and predicted mapping

parameter - the parameters of the algorithm (will be set to default if this is null)

foldNumber - the number of subsamples to divide the data (k)

qlMeasures - the set of qualitative measures

qnMeasures - the set of quantitative measures

Returns:

List - contains list of multiple runs evaluation results corresponding to the algorithms, its implementation and used dataset

crossValidateWithTuningAndStatisticalTest

public Summary crossValidateWithTuningAndStatisticalTest(List<TaskAlgorithm> TaskAlgorithms,
                                                         Set<TaskData> datasets,
                                                         Set<EvaluatorType> qlMeasures,
                                                         int foldNumber)

createParameterGrid

public Set<List<LearningParameter>> createParameterGrid(Map<LearningParameter,List<Object>> parameters)

main

public static void main(String[] args)
                 throws UnsupportedMLImplementationException

Throws:

UnsupportedMLImplementationException

cacheUnion

public ACache cacheUnion(ACache a,
                         ACache b)

generateFolds

public List<FoldData> generateFolds(EvaluationData data,
                                    int foldNumber,
                                    boolean withNegativeExamples)

generateMappingFolds

public List<AMapping> generateMappingFolds(AMapping refMap,
                                           ACache source,
                                           ACache target,
                                           int foldNumber,
                                           boolean withNegativeExamples)

getRandomTargetInstance

public static String getRandomTargetInstance(ACache source,
                                             ACache target,
                                             List<String> values,
                                             Random random,
                                             HashMap<String,HashMap<String,Double>> refMap,
                                             String sourceInstance,
                                             int previousRandom)

removeSubMap

public AMapping removeSubMap(AMapping mainMap,
                             AMapping subMap)