Spoken CALL Shared Task - 1st Edition

In the speech version of the CALL shared task, each item consists of

• an identifier
• a German text prompt
• an audio file containing an English language response

The data was collected from an online CALL tool used to help young Swiss German students improve their English fluency.

The task is to create software that will decide whether each response is appropriate (accept) or inappropriate (reject) in the context of the prompt. This will presumably require a combination of speech recognition and text processing methods. A response is considered appropriate if it both responds to the prompt in terms of meaning and is also correct English. For example, if the prompt is

"Frag: rote Stiefel"

("Ask for: red boots"), then "I would like some red boots" or "Red boots, please" are appropriate responses. "Give me brown boots" is inappropriate because it has the wrong meaning. "I wants red boots" is inappropriate because it is incorrect English.

The task is open-ended; there are many potentially appropriate responses to each prompt.

In this version of the task, no explicit attention is paid to quality of pronunciation.

Format of training data

The speech task training release directory contains the following resources:

1. A set of 5248 audio files
2. A CSV file of metadata in the format (headers and example lines)

Id	Prompt	Wavfile	Transcription	language	meaning
11336	Frag: rote Stiefel	11336.wav	i'd like red boots	correct	correct
7068	Frag: Wie viel kostet es?	7068.wav	how many is it	incorrect	incorrect
8774	Frag: Ich möchte die Rechnung	8774.wav	i want the bills	incorrect	correct

There is one line for each audio file. The specific CSV format is UTF-8, tab separated.

The 'language' and 'meaning' columns give judgements derived from three native speakers. The 'language' column contains the word "correct" if the response has been judged fully correct in terms of both language and meaning. The 'meaning' column contains the word "correct" if the response has been judged correct in terms of meaning, but not necessarily language.

Format of test data

The speech task test release directory, which will be released on Mar 13 2017, will contain the following resources:

1. A set of 1000 audio files
2. A CSV file of metadata in the format (headers and example lines)

Id	Prompt	Wavfile
11336	Frag: rote Stiefel	11336.wav
7068	Frag: Wie viel kostet es?	7068.wav
8774	Frag: Ich möchte die Rechnung	8774.wav

i.e. like the training data but without transcriptions or judgements. The specific CSV format is UTF-8, tab separated.

Format of answers

Groups who wish to submit an entry to the shared task should submit a CSV file, produced by running their system over the test data. The format should be the same as the training data, but with an extra column called Judgement added in which the possible values are 'accept' and 'reject'. For example:

Id	Prompt	Wavfile	Judgement
11336	Frag: rote Stiefel	11336.wav	accept
7068	Frag: Wie viel kostet es?	7068.wav	reject
8774	Frag: Ich möchte die Rechnung	8774.wav	reject

There should be one line for each line in the test data. The specific CSV format is once more UTF-8, tab separated.

Answer spreadsheets will be submitted by email to johanna.gerlach@unige.ch and emmanuel.rayner@unige.ch.

Scoring metric

The metric used to score the results is based on three intuitions:

• The system should reject incorrect answers as often as possible, and reject correct answers as seldom as possible.
• The more pronounced the difference between the system's response to incorrect as opposed to correct answers, the more useful it will be.
• Some system errors are more serious than others. In particular, it is worse for the system to accept a sentence which is incorrect in terms of meaning than it is to accept one which is correct in terms of meaning but incorrect in terms of language.

The metric is defined as follows (there is further discussion in §4.1 of the LREC 2016 paper):

Each system response falls into one of five categories:

1. Correct Reject: the student's answer is incorrect, the system rejects.
2. Correct Accept: the student's answer is correct, the system accepts.
3. False Reject: the student's answer is correct, the system rejects.
4. Plain False Accept: the student's answer is correct in meaning but incorrect English, the system accepts.
5. Gross False Accept: the student's answer is incorrect in meaning, the system accepts.

Define CR, CA, FR, PFA, GFA to be the number of utterances in each of the above categories, and put FA = PFA + k.GFA where k is a weighting factor that makes gross false accepts relatively more important. Then we define the differential response score, D, to be the ratio of the reject rate on incorrect answers to the reject rate on correct utterances:

D = ( CR/(CR + FA) ) / ( FR/(FR + CA) ) = CR(FR + CA) / FR(CR + FA)

We will use D as the metric for evaluating the quality of systems competing in the shared task, with the weighting factor k set equal to 3.

Recognition resources

A baseline recogniser for the task, built using the popular KALDI platform, will soon be available from the downloads tab.

Grammar resources

A sample grammar, based on the one in the app used to collect the data, is provided as part of the release. The grammar is in XML format, and associates each possible prompt with

1. a translation of the prompt into English
2. a set of possible responses.

A typical entry looks like this:

<prompt_unit>
 <prompt>Sag: Ich möchte am Montagmorgen abreisen</prompt>
 <translated_prompt>Ask for: I want to leave on monday morning</translated_prompt>
 <response>i need to leave on monday morning</response>
 <response>i need to leave on monday morning please</response>
 <response>i should like to leave on monday morning</response>
 <response>i should like to leave on monday morning please</response>
 <response>i want to leave on monday morning</response>
 <response>i want to leave on monday morning please</response>
 <response>i would like to leave on monday morning</response>
 <response>i would like to leave on monday morning please</response>
 <response>i'd like to leave on monday morning</response>
 <response>i'd like to leave on monday morning please</response>
</prompt_unit>

In the text version of the CALL shared task, each item consists of

• an identifier
• a German text prompt
• an audio file containing an English language response
• a text string representing the 1-best result of performing speech recognition on the audio file

The data was collected from an online CALL tool used to help young Swiss German students improve their English fluency.

The task is to create software that will decide whether each response is appropriate (accept) or inappropriate (reject) in the context of the prompt. This will require some kind of text processing method. A response is considered appropriate if it both responds to the prompt in terms of meaning and is also correct English. For example, if the prompt is

"Frag: rote Stiefel"

("Ask for: red boots"), then "I would like some red boots" or "Red boots, please" are appropriate responses. "Give me brown boots" is inappropriate because it has the wrong meaning. "I wants red boots" is inappropriate because it is incorrect English.

The task is open-ended; there are many potentially appropriate responses to each prompt.

Format of training data

The speech task training release directory contains the following resources:

1. A set of 5248 audio files
2. A CSV file of metadata in the format (headers and example lines)

Id	Prompt	Wavfile	RecResult	Transcription	language	meaning
11336	Frag: rote Stiefel	11336.wav	i'd like red boots	i'd like red boots	correct	correct
7068	Frag: Wie viel kostet es?	7068.wav	how many is it	how many is it	incorrect	incorrect
8774	Frag: Ich möchte die Rechnung	8774.wav	i want the bill	i want the bills	incorrect	correct

There is one line for each audio file. The specific CSV format is UTF-8, tab separated.

The 'language' and 'meaning' columns give judgements derived from three native speakers. The 'language' column contains the word "correct" if the response has been judged fully correct in terms of both language and meaning. The 'meaning' column contains the word "correct" if the response has been judged correct in terms of meaning, but not necessarily language.

The fact that speech recognition is often inaccurate means that there may not always be sufficient information to make a correct decision. For example, the third utterance should be rejected, since the student has replied with a grammatically incorrect sentence, but since the recogniser has corrected the error there is no way to determine this.

Format of test data

The text task test release directory, which will be released on Mar 13 2017, will contain the following resources:

1. A set of 1000 audio files
2. A CSV file of metadata in the format (headers and example lines)

Id	Prompt	Wavfile	RecResult
11336	Frag: rote Stiefel	11336.wav	i'd like red boots
7068	Frag: Wie viel kostet es?	7068.wav	how many is it
8774	Frag: Ich möchte die Rechnung	8774.wav	i want the bill

i.e. like the training data but without transcriptions or judgements. The specific CSV format is UTF-8, tab separated.

Format of answers

Groups who wish to submit an entry to the shared task should upload a CSV file, produced by running their system over the test data. The format should be the same as the test data, but with an extra column called Judgement added in which the possible values are 'accept' and 'reject'. For example:

Id	Prompt	Wavfile	RecResult	Judgement
11336	Frag: rote Stiefel	11336.wav	i'd like red boots	accept
7068	Frag: Wie viel kostet es?	7068.wav	how many is it	reject
8774	Frag: Ich möchte die Rechnung	8774.wav	i want the bill	reject

There should be one line for each line in the test data. The specific CSV format is once more UTF-8, tab separated.

Answer spreadsheets will be submitted by email to johanna.gerlach@unige.ch and emmanuel.rayner@unige.ch.

Scoring metric

The metric used to score the results is based on three intuitions:

• The system should reject incorrect answers as often as possible, and reject correct answers as seldom as possible.
• The more pronounced the difference between the system's response to incorrect as opposed to correct answers, the more useful it will be.
• Some system errors are more serious than others. In particular, it is worse for the system to accept a sentence which is incorrect in terms of meaning than it is to accept one which is correct in terms of meaning but incorrect in terms of language.

The metric is defined as follows (there is further discussion in §4.1 of the LREC 2016 paper):

Each system response falls into one of five categories:

1. Correct Reject: the student's answer is incorrect, the system rejects.
2. Correct Accept: the student's answer is correct, the system accepts.
3. False Reject: the student's answer is correct, the system rejects.
4. Plain False Accept: the student's answer is correct in meaning but incorrect English, the system accepts.
5. Gross False Accept: the student's answer is incorrect in meaning, the system accepts.

Define CR, CA, FR, PFA, GFA to be the number of utterances in each of the above categories, and put FA = PFA + k.GFA where k is a weighting factor that makes gross false accepts relatively more important. Then we define the differential response score, D, to be the ratio of the reject rate on incorrect answers to the reject rate on correct utterances:

D = ( CR/(CR + FA) ) / ( FR/(FR + CA) ) = CR(FR + CA) / FR(CR + FA)

We will use D as the metric for evaluating the quality of systems competing in the shared task, with the weighting factor k set equal to 3.

Grammar resources

A sample grammar, based on the one in the app used to collect the data, is provided as part of the release. The grammar is in XML format, and associates each possible prompt with

1. a translation of the prompt into English
2. a set of possible responses.

A typical entry looks like this:

<prompt_unit>
 <prompt>Sag: Ich möchte am Montagmorgen abreisen</prompt>
 <translated_prompt>Ask for: I want to leave on monday morning</translated_prompt>
 <response>i need to leave on monday morning</response>
 <response>i need to leave on monday morning please</response>
 <response>i should like to leave on monday morning</response>
 <response>i should like to leave on monday morning please</response>
 <response>i want to leave on monday morning</response>
 <response>i want to leave on monday morning please</response>
 <response>i would like to leave on monday morning</response>
 <response>i would like to leave on monday morning please</response>
 <response>i'd like to leave on monday morning</response>
 <response>i'd like to leave on monday morning please</response>
</prompt_unit>

Baseline system resource

A Python3 script which carries out a baseline version of the text task is provided as part of the release. The script reads the sample XML grammar and a training data spreadsheet, then scores each item in the spreadsheet by matching the prompt and recognition result against the appropriate record in the grammar. If the recognition result is listed in the grammar as a possible response for the prompt, it is accepted, otherwise it is rejected. The results are written out as a new spreadsheet.

The files used (resource grammar, input spreadsheet and output spreadsheet) are defined at the top of the script.