Package org.aksw.limes.core.evaluation.evaluator

Class Evaluator

java.lang.Object

org.aksw.limes.core.evaluation.evaluator.Evaluator

public class Evaluator
extends Object

This evaluator is responsible for evaluating set of datasets that have source, target, gold standard and mappings against set of measures

Since:

1.0

Version:

1.0

Author:

Mofeed Hassan (mounir@informatik.uni-leipzig.de), Daniel Obraczka (obraczka@studserv.uni-leipzig.de), Tommaso Soru (tsoru@informatik.uni-leipzig.de)

Field Summary

Fields

Modifier and Type	Field	Description
List<EvaluationRun>	runsList
Map<String,Map<String,Map<String,Double>>>	statisticalTestResults
Map<String,Map<String,int[]>>	successesAndFailures

Constructor Summary

Constructors

Constructor	Description
Evaluator()

Method Summary

Modifier and Type	Method	Description
ACache	cacheUnion(ACache a, ACache b)
Set<List<LearningParameter>>	createParameterGrid(Map<LearningParameter,List<Object>> parameters)
List<EvaluationRun>	crossValidate(List<TaskAlgorithm> algorithms, List<TaskData> datasets, int foldNumber, Set<EvaluatorType> qlMeasures)
Summary	crossValidateWithTuningAndMcNemarsTest(List<TaskAlgorithm> TaskAlgorithms, TaskData dataset, Set<EvaluatorType> qlMeasures, int foldNumber)	Performs crossvalidation, enables the tuning of hyperparameters via grid search on part of the training fold, performs pairwise McNemars Test on successes and failures between algorithms, i.e.
List<EvaluationRun>	evaluate(List<TaskAlgorithm> TaskAlgorithms, List<TaskData> datasets, Set<EvaluatorType> QlMeasures, Set<IQuantitativeMeasure> QnMeasures)
List<FoldData>	generateFolds(EvaluationData data, int foldNumber, boolean withNegativeExamples)
List<AMapping>	generateMappingFolds(AMapping refMap, ACache source, ACache target, int foldNumber, boolean withNegativeExamples)
static String	getRandomTargetInstance(ACache source, ACache target, List<String> values, Random random, HashMap<String,HashMap<String,Double>> refMap, String sourceInstance, int previousRandom)
AMapping	removeSubMap(AMapping mainMap, AMapping subMap)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

runsList

public List<EvaluationRun> runsList

successesAndFailures

public Map<String,Map<String,int[]>> successesAndFailures

statisticalTestResults

public Map<String,Map<String,Map<String,Double>>> statisticalTestResults

Constructor Detail

Evaluator

public Evaluator()

Method Detail

evaluate

public List<EvaluationRun> evaluate(List<TaskAlgorithm> TaskAlgorithms,
                                    List<TaskData> datasets,
                                    Set<EvaluatorType> QlMeasures,
                                    Set<IQuantitativeMeasure> QnMeasures)

Parameters:

TaskAlgorithms - the set of algorithms used to generate the predicted mappings

datasets - the set of the datasets to apply the algorithms on them. The should include source Cache, target Cache, goldstandard and predicted mapping

QlMeasures - the set of qualitative measures

QnMeasures - the set of quantitative measures

Returns:

List - contains list of multiple runs evaluation results corresponding to the algorithms, its implementation and used dataset

crossValidate

public List<EvaluationRun> crossValidate(List<TaskAlgorithm> algorithms,
                                         List<TaskData> datasets,
                                         int foldNumber,
                                         Set<EvaluatorType> qlMeasures)

Parameters:

algorithms - the algorithm used to generate the predicted mappings

datasets - the set of the datasets to apply the algorithms on them. The should include source Cache, target Cache, goldstandard and predicted mapping

foldNumber - the number of subsamples to divide the data (k)

qlMeasures - the set of qualitative measures

Returns:

List - contains list of multiple runs evaluation results corresponding to the algorithms, its implementation and used dataset

crossValidateWithTuningAndMcNemarsTest

public Summary crossValidateWithTuningAndMcNemarsTest(List<TaskAlgorithm> TaskAlgorithms,
                                                      TaskData dataset,
                                                      Set<EvaluatorType> qlMeasures,
                                                      int foldNumber)

Performs crossvalidation, enables the tuning of hyperparameters via grid search on part of the training fold, performs pairwise McNemars Test on successes and failures between algorithms, i.e. the times algorithm A got a link right and algorithm B got it wrong and vice versa

This is an implementation of the recommended approach of this paper

Parameters:

TaskAlgorithms - to use here

dataset - dataset on which the crossvalidation should be performed

qlMeasures - measures that will be used

foldNumber - number of folds that the crossvalidation should use

Returns:

Summary containing results with statistical evaluation

createParameterGrid

public Set<List<LearningParameter>> createParameterGrid(Map<LearningParameter,List<Object>> parameters)

cacheUnion

public ACache cacheUnion(ACache a,
                         ACache b)

generateFolds

public List<FoldData> generateFolds(EvaluationData data,
                                    int foldNumber,
                                    boolean withNegativeExamples)

generateMappingFolds

public List<AMapping> generateMappingFolds(AMapping refMap,
                                           ACache source,
                                           ACache target,
                                           int foldNumber,
                                           boolean withNegativeExamples)

getRandomTargetInstance

public static String getRandomTargetInstance(ACache source,
                                             ACache target,
                                             List<String> values,
                                             Random random,
                                             HashMap<String,HashMap<String,Double>> refMap,
                                             String sourceInstance,
                                             int previousRandom)

removeSubMap

public AMapping removeSubMap(AMapping mainMap,
                             AMapping subMap)