gate.util
Class ClassificationMeasures

java.lang.Object
  gate.util.ClassificationMeasures

public class ClassificationMeasures
extends Object

Given two annotation sets, a type and a feature, compares the feature values. It finds matching annotations and treats the feature values as classifications. Its purpose is to calculate the extent of agreement between the feature values in the two annotation sets. It computes observed agreement and Kappa measures.

Field Summary

private float[][]	confusionMatrix Array of dimensions categories * categories.
private TreeSet<String>	featureValues List of feature values that are the labels of the confusion matrix
private boolean	isCalculatedKappas
private float	kappaCohen Cohen's kappa.
private float	kappaPi Scott's pi or Siegel & Castellan's kappa

Constructor Summary

ClassificationMeasures()

ClassificationMeasures(Collection<ClassificationMeasures> tables)
Given a list of ClassificationMeasures, this will combine to make a megatable.

Method Summary

void	calculateConfusionMatrix(AnnotationSet aS1, AnnotationSet aS2, String type, String feature, boolean verbose) Create a confusion matrix in which annotations of identical span bearing the specified feature name are compared in terms of feature value.
protected void	computeKappaPairwise() Compute Cohen's and Pi kappas for two annotators.
private float[][]	convert2DHashTo2DFloatArray(HashMap<String,HashMap<String,Float>> countMap, TreeSet<String> featureValues) Convert between two formats of confusion matrix.
float	getAgreedTrials() Gets the number of annotations for which the two annotation sets are in agreement with regards to the annotation type.
float[][]	getConfusionMatrix() To understand exactly which types are being confused with which other types you will need to view this array in conjunction with featureValues, which gives the class labels (annotation types) in the correct order.
List<List<String>>	getConfusionMatrix(String title)
SortedSet<String>	getFeatureValues() This is necessary to make sense of the confusion matrix.
float	getKappaCohen() Kappa is defined as the observed agreements minus the agreement expected by chance.
float	getKappaPi() Kappa is defined as the observed agreements minus the agreement expected by chance.
List<String>	getMeasuresRow(Object[] measures, String documentName)
float	getObservedAgreement() Portion of the instances on which the annotators agree.
float	getTotalTrials() Gets the total number of annotations in the two sets.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

confusionMatrix

private float[][] confusionMatrix

Array of dimensions categories * categories.

kappaCohen

private float kappaCohen

Cohen's kappa.

kappaPi

private float kappaPi

Scott's pi or Siegel & Castellan's kappa

isCalculatedKappas

private boolean isCalculatedKappas

featureValues

private TreeSet<String> featureValues

List of feature values that are the labels of the confusion matrix

Constructor Detail

ClassificationMeasures

public ClassificationMeasures()

ClassificationMeasures

public ClassificationMeasures(Collection<ClassificationMeasures> tables)

Given a list of ClassificationMeasures, this will combine to make a megatable. Then you can use kappa getters to get micro average figures for the entire set.

Parameters:

tables - tables to combine

Method Detail

getObservedAgreement

public float getObservedAgreement()

Portion of the instances on which the annotators agree.

Returns:

a number between 0 and 1. 1 means perfect agreements.

getKappaCohen

public float getKappaCohen()

Kappa is defined as the observed agreements minus the agreement expected by chance. The Cohen’s Kappa is based on the individual distribution of each annotator.

Returns:

a number between -1 and 1. 1 means perfect agreements.

getKappaPi

public float getKappaPi()

Kappa is defined as the observed agreements minus the agreement expected by chance. The Siegel & Castellan’s Kappa is based on the assumption that all the annotators have the same distribution.

Returns:

a number between -1 and 1. 1 means perfect agreements.

getConfusionMatrix

public float[][] getConfusionMatrix()

To understand exactly which types are being confused with which other types you will need to view this array in conjunction with featureValues, which gives the class labels (annotation types) in the correct order.

Returns:

confusion matrix describing how annotations in one set are classified in the other and vice versa

getFeatureValues

public SortedSet<String> getFeatureValues()

This is necessary to make sense of the confusion matrix.

Returns:

list of annotation types (class labels) in the order in which they appear in the confusion matrix

calculateConfusionMatrix

public void calculateConfusionMatrix(AnnotationSet aS1,
                                     AnnotationSet aS2,
                                     String type,
                                     String feature,
                                     boolean verbose)

Create a confusion matrix in which annotations of identical span bearing the specified feature name are compared in terms of feature value. Compiles list of classes (feature values) on the fly.

Parameters:

aS1 - annotation set to compare to the second

aS2 - annotation set to compare to the first

type - annotation type containing the features to compare

feature - feature name whose values will be compared

verbose - message error output when ignoring annotations

computeKappaPairwise

protected void computeKappaPairwise()

Compute Cohen's and Pi kappas for two annotators.

getAgreedTrials

public float getAgreedTrials()

Gets the number of annotations for which the two annotation sets are in agreement with regards to the annotation type.

Returns:

Number of agreed trials

getTotalTrials

public float getTotalTrials()

Gets the total number of annotations in the two sets. Note that only matched annotations (identical span) are considered.

Returns:

Number of trials

getConfusionMatrix

public List<List<String>> getConfusionMatrix(String title)

Parameters:

title - matrix title

Returns:

confusion matrix as a list of list of String

getMeasuresRow

public List<String> getMeasuresRow(Object[] measures,
                                   String documentName)

convert2DHashTo2DFloatArray

private float[][] convert2DHashTo2DFloatArray(HashMap<String,HashMap<String,Float>> countMap,
                                              TreeSet<String> featureValues)

Convert between two formats of confusion matrix. A hashmap of hashmaps is easier to populate but an array is better for matrix computation.

Parameters:

countMap - count for each label as in confusion matrix

featureValues - sorted set of labels that will define the dimensions

Returns:

converted confusion matrix as an 2D array