Task 1: Multilingual question answering

The open challenge on multilingual question answering over linked data provides a benchmark for comparing different approaches and systems that mediate between a user, expressing his or her information need in natural language, and semantic data.

Task

Given a RDF dataset and a natural language question or set of keywords in one of six languages (English, Spanish, German, Italian, French, Dutch), either return the correct answers, or a SPARQL query that retrieves these answers.

Full description: qald3_openchallenge.pdf (Last updated: March 25, 2013)

Datasets

SPARQL endpoint

We do not maintain a SPARQL endpoint once the challenge is finished. Please load the data into a local triple store.

Training data

The training data comprises 100 natural language questions for English DBpedia and MusicBrainz each, annotated with keywords as well as corresponding SPARQL queries and the answers that these queries retrieve. The questions are of different complexity and are available in six languages: English, Spanish, German, Italian, French, and Dutch.

Furthermore, 50 similar natural language questions are provided for Spanish DBpedia. They are available in English and Spanish, and contain the same annotations. Special thanks to Mariano Rico from the esDBpedia team for providing us with these queries!

Participant's challenge:


Test data

The test data consists of similar questions and keywords, without annotations, for which answers or SPARQL queries have to be submitted.

Here you can check the gold standard (i.e. with annotations, SPARQL queries and answers):

Evaluation

For both training and test phase, results can be uploaded with the following evaluation form:

Evaluation results: qald3_results.pdf