About the task

The shared task is based on data collected from a speech-enabled online tool which has been used to help young Swiss German teens practise skills in English conversation. Items are prompt-response pairs, where the prompt is a piece of German text and the response is a recorded English audio file. The task is to label pairs as “accept” or “reject”, accepting responses which are grammatically and linguistically correct to match a set of hidden gold standard answers as closely as possible. Initial resources are provided so that a scratch system can be constructed with a minimal investment of effort, and in particular without necessarily using a speech recogniser.

The task is now closed, results will be presented at the SLaTE 2017 workshop.

The annotated test data is now available in the downloads tab

New! The anonymised task results are now available in the results tab

New! A browsable annotated version of the test data is now available in the test data tab


Schedule

July 13, 2016
Release of training material
March 13, 2017
Release of test material
March 20, 2017
Deadline for submission of results
May 5, 2017
SLaTE 2017 submission deadline
August 25-26, 2017
SLaTE 2017: Presentation and discussion of results

Shared task results

Results for 20 anonymised submissions and three baseline systems (BaselineKaldi = system with baseline Kaldi recogniser and baseline XML grammar; BaselineNuance = system with baseline Nuance recogniser and baseline XML grammar; BaselinePerfectRec = system with input from transcriptions and baseline XML grammar.)

IdRecPrRecFSAIRejCRejDResources
KKKCustom0.8810.8450.8620.8400.7390.1554.766
JJJCustom0.8710.8480.8590.8380.7170.1524.710ASR output
BaselinePerfectRec n/a0.9950.7810.8750.8390.9890.2194.512
CCCKaldi0.6870.9330.7910.8010.3000.0674.468
IIICustom0.7520.9040.8210.8050.4210.0964.371
MMMKaldi0.7320.9050.8090.8180.4130.0954.353
OOOKaldi0.7390.8990.8120.8180.4300.1014.273
GGGKaldi0.6390.9570.7660.7690.1730.0433.998
AAAKaldi0.6060.9730.7470.7490.0980.0273.678
FFFKaldi0.6310.9510.7590.7620.1640.0493.352
BBBKaldi0.7480.8620.8010.7980.4610.1383.335
HHHKaldi0.6020.9710.7430.7470.0960.0293.289
PPPCustom0.8380.7950.8160.7860.6600.2053.217
NNNKaldi0.7130.8800.7880.7900.3830.1203.188
QQQCustom0.8540.7530.8000.7700.7130.2472.882
DDDKaldi0.7770.8110.7940.7790.5390.1892.857
LLLNuance0.8140.7540.7830.7460.6230.2462.533
BaselineNuance Nuance0.8220.7230.7700.7310.6520.2772.358
EEENuance0.8160.7290.7700.7310.6360.2712.347
SSSNuance0.7370.8200.7760.7430.4230.1802.346
RRRCustom0.8840.5840.7030.6680.8180.4161.965
BaselineKaldi Kaldi0.9570.4390.6020.5880.9510.5611.694
TTTNuance0.6220.8970.7350.7130.1480.1031.437
  • Rec = recogniser used
  • Pr = precision,
  • R = recall
  • F = F-measure
  • SA = scoring average
  • IRej = rejections on incorrect responses,
  • CRej = rejections on correct responses
  • D = D-measure.
  • Resources = resources shared by participant

The scores have been calculated using the scoring script

Test data

Id Prompt File Transcription Language Meaning NBad
4596 Frag: mein Steak durchgebraten i would like my steak well down incorrect correct 20
3709 Sag: Ich habe 2 jüngere Brüder i have two young brothers incorrect incorrect 20
3691 Frag: Junior Suite on the junior suite incorrect incorrect 20
4535 Sag: Ich habe nur grosse Noten i have only big notes correct correct 19
4271 Frag: ein T-Shirt i will like a t-shirt incorrect correct 18
4223 Frag: Gibt es einen Coiffeur? gives here a hairdresser incorrect correct 18
3944 Frag: Hosen i want pants correct correct 18
4546 Frag: Tickets für das Naturhistorische Museum can i have tickets for the natural museum incorrect incorrect 17
4461 Sag: Dies ist zu eng is it too tight correct correct 17
4244 Sag: Ich habe keine Reservation i don't haven't a reservation incorrect correct 17
4146 Sag: Ich möchte mit Visa bezahlen i would to pay by visa incorrect correct 17
4087 Frag: weisse Stiefel would like white boots incorrect correct 17
3801 Sag: Ich möchte mit Postkarte bezahlen i'd like to change with post card incorrect incorrect 17
3688 Sag: Ich möchte mit Schweizer Franken bezahlen can i pay with swiss friends incorrect incorrect 17
4699 Sag: Ich bin aus der Schweiz i am from swiss incorrect correct 16
4372 Frag: Einzelzimmer can i you have a single room please incorrect correct 16
4062 Sag: Ich möchte mit Visa bezahlen can i pay with a visa incorrect correct 16
4030 Frag: Zimmer für 5 Nächte want a room for five nights incorrect correct 16
3803 Sag: Mir geht es gut i'm thanks incorrect incorrect 16
3790 Frag: Wo ist das Hotelrestaurant? where is the hotel restaurant correct correct 16
3657 Sag: Ich möchte mit Mastercard bezahlen i would like to play with my master card incorrect incorrect 16
4653 Frag: Jeans can i have teen incorrect incorrect 15
4621 Sag: Ich möchte mit Kreditkarte bezahlen i would like to pay with credit card incorrect correct 15
4483 Sag: Ich bin aus Spanien i am from espania incorrect correct 15
4115 Frag: Gibt es einen Lebensmittelladen? does it have a sup supermarket correct correct 15
4104 Frag: Wo ist der Lift? there is the lift incorrect incorrect 15
4072 Sag: Ich möchte hinten sitzen can i sit in the back correct correct 15
3899 Frag: 1 Musical-Ticket a ticket for wednesday evening incorrect incorrect 15
3817 Sag: Dies ist mein Pass these is my passport incorrect correct 15
3640 Sag: Ich möchte mit Postkarte bezahlen i would like to play with my post card incorrect correct 15
3618 Frag: 2 Tickets nach London two ticket to london incorrect correct 15
4640 Frag: Wo ist Big Ben? where is the big ben incorrect correct 14
4527 Sag: Ich möchte mit Postkarte bezahlen i like to pay by post card incorrect correct 14
4519 Frag: ein Ticket zum Green Park can i have a ticket to the green park incorrect correct 14
4456 Frag: Grösse 43 i am a forty-two incorrect incorrect 14
4362 Frag: Grösse 43 forty three correct correct 14
4289 Frag: Wo ist Big Ben? where is the big ben incorrect correct 14
4257 Frag: Wo ist das Fitnessstudio? where is the fitness studio correct correct 14
4209 Sag: Mir geht es gut i am good correct correct 14
4097 Frag: Wie viel kostet die Jeans? how much do the jeans cost correct correct 14
4066 Sag: Ich möchte heute Abend gehen can i go this evening correct correct 14
3891 Sag: Ich möchte mit Mastercard bezahlen i would like pay with mastercard incorrect correct 14
3819 Frag: 3 Musical-Tickets musical tickets incorrect incorrect 14
3623 Frag: 2 Musical-Tickets two mitch tickets please incorrect incorrect 14
4656 Frag: Wie viel kostet die Jeans? how much are these jeans correct correct 13
4526 Frag: ein Ticket zum Trafalgar Square can i have a ticket to the travel squares incorrect incorrect 13
4513 Sag: Ich habe eine ältere Schwester i have three older sisters incorrect incorrect 13
4492 Sag: Ich bin aus Spanien i'm from espania incorrect correct 13
4293 Frag: ein Ticket nach Notting Hill a billet to notting hill incorrect incorrect 13
4251 Frag: Suite can i have a suite correct correct 13
4196 Frag: ein Ticket nach London i would have a ticket to london incorrect correct 13
4181 Sag: Im Zimmer hat es keinen Haarföhn i could not find the ha hairdryer correct correct 13
4116 Frag: Gibt es einen Lebensmittelladen? is there a grocery store correct correct 13
4063 Sag: Ich möchte mit Pfund bezahlen can i pay with pfund incorrect incorrect 13
4007 Frag: Twinzimmer i want a double room incorrect incorrect 13
3893 Frag: Zimmer für 1 Woche i would like a room for once a week incorrect incorrect 13
3836 Frag: ein Ticket für heute Abend a ticket for night incorrect incorrect 13
3802 Sag: Ich möchte mit Pfund bezahlen i'd like to change with pounds incorrect incorrect 13
3800 Frag: ein Ticket zum Piccadilly Circus i want a ticket to the piccadilly circus incorrect correct 13
3757 Frag: Wo ist der Tower? where is the tour i'm incorrect incorrect 13
3610 Frag: Ich möchte die Dessertkarte could i see the dessert menu correct correct 13
3605 Frag: mein Steak rare like my steak rare incorrect correct 13
3582 Sag: Ich möchte vorne sitzen i would like to sit in the front correct correct 13
4599 Sag: Dies ist nicht was ich bestellt habe that is not what i order incorrect correct 12
4486 Sag: Die Hauptstadt von Italien ist Rom the capital of italian is rome incorrect correct 12
4465 Sag: Ich habe keine Reservation i have not a reservation incorrect correct 12
4354 Frag: eine S little eh small correct correct 12
4287 Frag: Wo ist das Naturhistorische Museum? i am looking for the natury history museum incorrect correct 12
4262 Frag: ein Ticket für Mittwochabend can i have a ticket for midday night incorrect incorrect 12
4261 Frag: 4 Musical-Tickets i will pay four musical tickets incorrect incorrect 12
4252 Frag: Suite suite correct correct 12
4231 Frag: Wo ist der Tower? can you give me the directions to the tower of london correct correct 12
4134 Sag: Ich möchte mit Kreditkarte bezahlen i would to pay by credit card incorrect correct 12
4127 Sag: Ich möchte mit Kreditkarte bezahlen i want to pay with the credit card correct correct 12
4081 Frag: mein Steak rare i would like to have a steak rare incorrect correct 12
3810 Sag: Ich möchte mit Kreditkarte bezahlen i want pay with credit card incorrect correct 12
3789 Frag: Wo ist das Hotelrestaurant? where is the restaurant correct correct 12
3771 Sag: Ich habe keine Reservation i have no reservation correct correct 12
3764 Frag: Twinzimmer a twin zimmer incorrect incorrect 12
3762 Frag: Zimmer für 6 Nächte a dorm for six nights incorrect incorrect 12
3708 Sag: Die Hauptstadt von Portugal ist Lissabon the capital of portugal is lisabon incorrect correct 12
3668 Sag: Ich möchte in der achten Reihe sitzen i want seat in the eight row incorrect correct 12
3655 Frag: ein Ticket zum Piccadilly Circus like a ticket to piccadilly circus incorrect correct 12
3594 Sag: Ich möchte mit Pfund bezahlen can i pay by pound incorrect correct 12
3593 Sag: Ich möchte mit Pfund bezahlen can i pay by pound incorrect correct 12
3580 Sag: Ich möchte vorne sitzen i would like to sit in the john incorrect incorrect 12
4697 Sag: Ich möchte am Sonntag abreisen i like to leave at sunday incorrect correct 11
4683 Frag: Gibt es einen Lift? is there a ascenseur incorrect incorrect 11
4672 Sag: Ich möchte mit Kreditkarte bezahlen can i pay with credit card incorrect correct 11
4611 Frag: etwas Knoblauchbrot can i have a bit toast bread incorrect incorrect 11
4586 Frag: Bohnen i would like bones incorrect incorrect 11
4512 Sag: Die Hauptstadt von Österreich ist Wien the capital of austria is wien incorrect correct 11
4487 Sag: Ich habe 4 Schwestern i have four siblings incorrect incorrect 11
4472 Frag: 4 Tickets nach London i would like four tickets to london correct correct 11
4457 Frag: ein Hemd i want a shirt correct correct 11
4442 Frag: Grösse 42 question forty-two incorrect incorrect 11
4358 Frag: Stiefel i would like some boo incorrect incorrect 11
4323 Frag: Tickets für das London Eye tickets for the london eye incorrect correct 11
4305 Sag: Ich bin aus Österreich i am from austrish incorrect incorrect 11
4266 Frag: Stiefel i want to have shoes incorrect incorrect 11
4186 Sag: Ich habe keine Reservation no i haven't a incorrect incorrect 11
4083 Frag: etwas Kartoffelstock i would like to have some of the smashed potatoes incorrect incorrect 11
4080 Frag: mein Steak rare i want a rare steak correct correct 11
3876 Frag: Erbsen i want peas correct correct 11
3875 Frag: Erbsen i want peas correct correct 11
3797 Sag: Ich möchte mit Schweizer Franken bezahlen i would like to pay by swiss francs incorrect correct 11
3788 Frag: Wo ist das Hotelrestaurant? where is the restaurant correct correct 11
3751 Frag: ein Steak kusvich incorrect incorrect 11
3674 Frag: Zimmer für 6 Nächte can i have six rooms for six nights incorrect incorrect 11
3664 Frag: 4 Tickets für Wicked i would like four tickets for the wick incorrect incorrect 11
3641 Sag: Ich möchte mit Postkarte bezahlen i would like to play with incorrect incorrect 11
3627 Frag: 2 Tickets für König der Löwen two tickets for lion king incorrect correct 11
3583 Frag: Tickets für die National Gallery i would like tickets for the national gallery correct correct 11
3570 Frag: Zimmer für 1 Woche can i have a hotel room for one night incorrect incorrect 11
4677 Frag: Zimmer für 1 Woche can i have a room for one night incorrect incorrect 10
4642 Sag: Saint Pauls Cathedral ist nahe der Bank of England the saint pauls cathedral is near to bank of england incorrect correct 10
4618 Frag: Zitronenkuchen i would like the cake of citro incorrect incorrect 10
4559 Sag: Ich möchte am Dienstagabend gehen i like to go at tuesday night incorrect correct 10
4538 Sag: Kann ich mit Kreditkarte bezahlen can i pay with credit card incorrect correct 10
4536 Sag: Ich habe nur grosse Noten i only have big notes correct correct 10
4493 Sag: Die Hauptstadt von Spanien ist Madrid the capital of espania is madrid incorrect correct 10
4385 Frag: Gibt es einen Lebensmittelladen? there a grocery sto store incorrect correct 10
4352 Frag: eine S little incorrect correct 10
4338 Frag: Ich möchte die Rechnung i would like to pay correct correct 10
4332 Frag: Auberginen i would like aubergines correct correct 10
4322 Frag: Tickets für das London Eye tickets for the london eye correct correct 10
4255 Frag: Gibt es ein Fitnessstudio? gives here a fitness studio incorrect incorrect 10
4227 Frag: Wo ist das Science Museum? where is the single ience museum incorrect incorrect 10
4220 Frag: Junior Suite can i have a junior suite correct correct 10
4213 Frag: Gibt es ein Fitnessstudio? gives here a fitness studio incorrect correct 10
4185 Sag: Ich habe keine Reservation abi du dubbel incorrect incorrect 10
4183 Frag: Wo kann ich einen Haarföhn kaufen? where i can buy a hairdryer correct correct 10
4176 Frag: Gibt es einen Lebensmittelladen? is there a supermarket correct correct 10
4168 Frag: Zimmer für 1 Woche don't you hahaha i would have a room for one week incorrect correct 10
4162 Frag: Zimmer für 2 Nächte i stay for two nights incorrect correct 10
4147 Sag: Ich möchte mit Dollars bezahlen i would to pay by dollars incorrect correct 10
4136 Sag: Ich möchte mit Kreditkarte bezahlen i would pay by credit card incorrect correct 10
4135 Sag: Ich möchte mit Kreditkarte bezahlen i would to play pay by credit card incorrect correct 10
4114 Sag: Ich möchte mit Mastercard bezahlen i want to pay with master card correct correct 10
4070 Sag: Ich möchte hinten sitzen can i sit for behind incorrect incorrect 10
4040 Frag: Wo ist der Haarföhn? where is the hair blower incorrect correct 10
4015 Sag: Ich habe keine Reservation no i don't no i do not have a reservation correct correct 10
4001 Frag: Ticket für dieses Musical i need tickets a ticket for this musical correct correct 10
3804 Frag: Stiefel i'd like stiefel incorrect incorrect 10
3750 Frag: ein Steak i would like a steetch incorrect incorrect 10
3730 Frag: Gibt es einen Lift? is it a lift incorrect incorrect 10
3671 Sag: Ich habe keine Reservation no i do not have a reservation correct correct 10
3589 Sag: Ich möchte mit Visa bezahlen i would like to pay be visa incorrect correct 10
3581 Sag: Ich möchte vorne sitzen i would like to sit in the front correct correct 10
4678 Frag: Doppelzimmer can i have a double roo double room correct correct 9
4632 Sag: Ich muss gehen i'd like to go correct correct 9
4609 Frag: etwas Knoblauchbrot can i have a cursy bread incorrect incorrect 9
4605 Sag: Ich möchte mit Kreditkarte bezahlen i like to pay with credit card incorrect correct 9
4582 Sag: Ich möchte mit Kreditkarte bezahlen i like to pay by credit card incorrect correct 9
4576 Frag: Pilze can i have some mushrooms correct correct 9
4573 Frag: den Sonntagsbraten i would like the sunday roast correct correct 9
4551 Frag: ein Ticket für Freitagabend i couldn't understand you incorrect incorrect 9
4540 Sag: Kann ich mit Kreditkarte bezahlen i like to pay with credit card incorrect correct 9
4515 Sag: Ich habe eine ältere Schwester i have three older sisters incorrect incorrect 9
4514 Sag: Ich habe eine ältere Schwester i have three older sisters incorrect incorrect 9
4494 Sag: Ich habe 4 ältere Schwestern i have four older sisters correct correct 9
4485 Sag: Ich bin aus Italien i am from italia incorrect incorrect 9
4482 Sag: Ich bin aus Spanien i am from espanien incorrect correct 9
4476 Frag: 2 Tickets nach London i would like two tickets to london correct correct 9
4470 Frag: 4 Tickets nach London i want four tickets to london correct correct 9
4462 Sag: Dies ist zu eng this is too tight correct correct 9
4449 Sag: Dies ist zu teuer this is too cher incorrect incorrect 9
4436 Sag: Es ist nahe der Bank of England it's near the bank of england near the bank england incorrect correct 9
4412 Sag: Ich möchte mit Visa bezahlen pay with visa incorrect correct 9
4404 Sag: Die Hauptstadt von der Schweiz ist Bern the capital city of the swiss is bern incorrect correct 9
4389 Sag: Ich habe keine Reservation i don't a reservation incorrect correct 9
4361 Frag: Grösse 43 forty three correct correct 9
4343 Sag: Mir geht es gut eh i'm fine correct correct 9
4308 Frag: ein Ticket zum Piccadilly Circus tickets to the pissadilly circus incorrect correct 9
4299 Frag: 5 Tickets nach London i would like boy five tickets to london incorrect correct 9
4296 Sag: Ich möchte mit Mastercard bezahlen can i pay be mastercard incorrect correct 9
4248 Sag: Ich habe keine Reservation grüessech frau käser incorrect incorrect 9
4247 Sag: Ich habe keine Reservation i don't haven't a reservation i don't haven't a reservation incorrect correct 9
4189 Frag: Zimmer für 2 Nächte a room for two nights correct correct 9
4175 Frag: Gibt es einen Lebensmittelladen? do you have a supermarket correct correct 9
4121 Sag: Ich möchte um 12 Uhr morgen abreisen i want to leave at twelve o'clock tomorrow correct correct 9
4071 Sag: Ich möchte hinten sitzen i want to sit in the back correct correct 9
4068 Sag: Ich möchte heute Abend gehen can i go this evening correct correct 9
4053 Frag: ein Ticket zum Trafalgar Square a ticket to trafalgar square correct correct 9
4052 Frag: ein Ticket zum Trafalgar Square a ticket to trafalgar square correct correct 9
4011 Frag: Gibt es ein Hotelrestaurant? are there a hotel restaurant incorrect correct 9
4010 Sag: Ich möchte mit Euros bezahlen i will pay with euros incorrect correct 9
3901 Frag: ein Ticket für morgen Abend a ticket for morning night incorrect incorrect 9
3892 Sag: Ich möchte mit Mastercard bezahlen are there a coiffeur incorrect incorrect 9
3862 Sag: Marble Arch ist gegenüber dem Hyde Park marble arch is across hyde street park incorrect incorrect 9
3858 Sag: Big Ben ist neben den Houses of Parliament the big ben is beside the houses of parliament incorrect correct 9
3796 Sag: Ich möchte mit Schweizer Franken bezahlen i would like to pay by swiss francs incorrect correct 9
3784 Frag: Doppelzimmer a double room correct correct 9
3777 Sag: Ich habe keine Reservation i do not have a reservation correct correct 9
3776 Sag: Ich habe keine Reservation i do not have a reservation correct correct 9
3759 Frag: Welcher Bus fährt dorthin? which bus goes there correct correct 9
3749 Frag: ein Steak single room please a single r incorrect incorrect 9
3663 Sag: Ich möchte am Freitagabend gehen i would like to go on free day night incorrect correct 9
3611 Sag: Mir geht es gut i am good correct correct 9
3608 Frag: Pilze i would like pilz incorrect incorrect 9
3592 Sag: Ich möchte mit Visa bezahlen i would like to bay with visa incorrect correct 9
4682 Frag: Gibt es einen Lift? is there a lift correct correct 8
4681 Sag: Ich möchte mit Mastercard bezahlen can i pay with mastercard correct correct 8
4649 Frag: Sandalen i would like sandals correct correct 8
4647 Sag: Saint Pauls Cathedral ist nahe der Bank of England the saint pauls cathedral is near to the bank of england incorrect correct 8
4629 Frag: Welcher Bus fährt dorthin? which bus can i take correct correct 8
4589 Frag: Ich möchte die Dessertkarte can i have the dessert menu correct correct 8
4580 Frag: Custard Tart i would like the custard tart correct correct 8
4534 Sag: Ich möchte mit Euros bezahlen can i pay with euros correct correct 8
4528 Sag: Ich möchte mit Schweizer Franken bezahlen i like to pay by swiss francs incorrect correct 8
4524 Frag: Wo ist der Geldautomat? euh where is the atm correct correct 8
4521 Sag: Ich möchte mit Pfund bezahlen i like to buy with pounds incorrect correct 8
4377 Frag: Zimmer für 6 Nächte can i have a room for six nights correct correct 8
4360 Frag: weisse Stiefel i would like white boots correct correct 8
4353 Frag: eine S little incorrect correct 8
4347 Frag: Sandalen i would like sandals correct correct 8
4346 Frag: Sandalen i would like sandals jetzt chani incorrect incorrect 8
4334 Frag: Ich möchte die Dessertkarte i would like the dessert card incorrect correct 8
4312 Sag: Ich möchte mit Mastercard bezahlen can i pay diner's card incorrect incorrect 8
4309 Frag: ein Ticket zum Piccadilly Circus i want a ticket to piccasarticircus incorrect incorrect 8
4291 Sag: Marble Arch ist gegenüber dem Hyde Park marble arch is vis à vis the hyde park incorrect incorrect 8
4269 Frag: Grösse 39 thirty-nine correct correct 8
4250 Frag: Zimmer für 3 Nächte can i have a room for three nights please can i have a room for three nights correct correct 8
4238 Sag: Ich habe keine Reservation i don't have a reservation correct correct 8
4222 Sag: Ich möchte mit Postkarte bezahlen i like pay with post card incorrect correct 8
4217 Sag: Das WC ist dreckig the wc is sale incorrect incorrect 8
4212 Sag: Ich möchte mit Kreditkarte bezahlen i will pay with credit card incorrect correct 8
4211 Sag: Ich möchte mit Kreditkarte bezahlen i will pay with credit card incorrect correct 8
4191 Frag: Doppelzimmer a room for two person incorrect incorrect 8
4174 Sag: Ich möchte mit Kreditkarte bezahlen i want to pay with credit card incorrect correct 8
4128 Sag: Ich möchte mit Kreditkarte bezahlen would i play with my credit card incorrect incorrect 8
4079 Frag: mein Steak rare i want steak rare incorrect correct 8
4064 Sag: Ich möchte mit Pfund bezahlen can i pay with pounds correct correct 8
4039 Frag: Wo ist der Haarföhn? where is the hair blower incorrect correct 8
3955 Sag: Ich habe 1 Bruder i have a brother correct correct 8
3950 Sag: Die Hose ist zu teuer the trousers is too expensive incorrect correct 8
3948 Frag: Wie viel kostet die Hose? how much are the croasts incorrect incorrect 8
3947 Frag: eine S small correct correct 8
3900 Frag: ein Ticket für morgen Abend a ticket for incorrect incorrect 8
3871 Frag: einen Orangensaft i want a an orange juice correct correct 8
3864 Frag: Wo ist Big Ben? where is the big ben incorrect correct 8
3852 Frag: Wo ist die Oxford Street? where is the oxford street incorrect correct 8
3769 Sag: Ich habe keine Reservation i've no a reservation incorrect correct 8
3647 Sag: Ich habe 1 Bruder i have a brother correct correct 8
3642 Frag: Gibt es eine Hotelbar? is there the hotel bar incorrect correct 8
3639 Sag: Ich möchte mit Postkarte bezahlen i would to play with my post card incorrect incorrect 8
3626 Frag: ein Ticket für Montagabend tickets for the monday avenue incorrect incorrect 8
3599 Sag: Ich möchte mit Dollars bezahlen can a pay by dollars incorrect correct 8
3597 Sag: Ich möchte mit Mastercard bezahlen can i i pay by mastercard correct correct 8
3588 Sag: Ich möchte mit Visa bezahlen i would like to pay with visa correct correct 8
3587 Frag: ein Ticket zum Piccadilly Circus i would like a ticket to piccadilly circus correct correct 8
3584 Frag: Tickets für die National Gallery i would like i would like nationaly gal i would ah mann incorrect incorrect 8
3575 Frag: 3 Musical-Tickets i would like to buy three musical tickets correct correct 8
4674 Frag: Wo ist der Coiffeur? where is the hairdresser correct correct 7
4646 Frag: Wo ist der Tower? where is the tower correct correct 7
4644 Frag: Wo ist der Tower? where is the tower correct correct 7
4639 Frag: Wo ist das British Museum? where is the british museum correct correct 7
4620 Sag: Ich möchte mit Kreditkarte bezahlen i like to pay with credit card incorrect correct 7
4614 Frag: Ich möchte die Dessertkarte can i have the dessert menu please correct correct 7
4610 Frag: etwas Knoblauchbrot can i have a bit percy bread incorrect incorrect 7
4601 Frag: Ich möchte die Dessertkarte can i have the dessert menu please can i have the dessert menu please correct correct 7
4593 Frag: ein Steak can i have the steak correct correct 7
4554 Sag: Ich möchte vorne sitzen i like to sit in the first row incorrect correct 7
4550 Frag: ein Ticket für Freitagabend can i have a ticket for friday night correct correct 7
4525 Frag: Wo ist der Geldautomat? where is the a t m correct correct 7
4471 Frag: 4 Tickets nach London i would like incorrect incorrect 7
4468 Frag: ein Ticket zum Piccadilly Circus i want a ticket to piccadilly circus correct correct 7
4455 Frag: Grösse 43 question forty three incorrect incorrect 7
4453 Frag: blaue Sandalen i would like blue sandals correct correct 7
4451 Sag: Mir geht es gut i'm good correct correct 7
4435 Frag: Wo ist Piccadilly Circus? where is the piccadilly bill incorrect incorrect 7
4418 Frag: ein Ticket für Mamma Mia a ticket for mamma mia correct correct 7
4359 Frag: Stiefel i would like some boots correct correct 7
4335 Frag: Ich möchte die Dessertkarte i would like the dessert plan incorrect incorrect 7
4314 Frag: 4 Musical-Tickets i want pay four musical tickets incorrect correct 7
4310 Frag: ein Ticket zum Piccadilly Circus i want a ticket to piccadilly circus correct correct 7
4284 Sag: Die Hose ist zu teuer this trousers is too teuer incorrect incorrect 7
4281 Frag: pinkige Hosen i would like pink trousers correct correct 7
4237 Sag: Ich habe keine Reservation i haven't a reservation incorrect correct 7
4228 Frag: Wo ist das Science Museum? where is the science museum correct correct 7
4224 Frag: Wo ist der Coiffeur? where is the hairdresser correct correct 7
4214 Frag: Wo ist das Fitnessstudio? where is the fitness studio correct correct 7
4206 Sag: Ich habe 2 ältere Schwestern i have two older sisters correct correct 7
4197 Frag: ein Ticket nach London i would one ticket to incorrect incorrect 7
4188 Sag: Ich habe keine Reservation no i haven't got a reservation correct correct 7
4180 Sag: Im Zimmer hat es keinen Haarföhn i could not i could not find the incorrect incorrect 7
4166 Frag: Gibt es ein Schwimmbad? do you have a swimming pool correct correct 7
4122 Sag: Ich möchte um 12 Uhr morgen abreisen i would like to leave at twelve o'clock tomorrow correct correct 7
4118 Frag: Wo ist der Lebensmittelladen? where is the grocery store correct correct 7
4049 Sag: Im Zimmer hat es keinen Haarföhn in the room isn't a hair dryer incorrect correct 7
4038 Frag: Wo ist der Haarföhn? where is the hair blower incorrect correct 7
4035 Frag: Gibt es einen Coiffeur? do you have a hairdresser correct correct 7
3949 Frag: Wie viel kostet die Hose? how much does the trousers cost incorrect correct 7
3904 Frag: ein Ticket für Starlight Express a ticket for starlight express correct correct 7
3902 Frag: ein Ticket für morgen Abend a ticket for tomorrow night correct correct 7
3799 Frag: Gibt es einen Coiffeur? is there a hairdresser correct correct 7
3779 Sag: Ich habe keine Reservation no i have not have incorrect incorrect 7
3761 Frag: Zimmer für 6 Nächte a dorm for six nights incorrect incorrect 7
3740 Frag: Gibt es ein Fitnessstudio? have you got a fitness studio correct correct 7
3669 Frag: Tickets für das Naturhistorische Museum i want tickets for the natural history museum please correct correct 7
3661 Sag: Ich möchte am Freitagabend gehen i would go by free day avenue incorrect incorrect 7
3654 Frag: ein Ticket zum Piccadilly Circus a ticket from piccadilly circus please incorrect incorrect 7
3648 Sag: Ich habe 1 Bruder i have a brother correct correct 7
3630 Sag: Ich möchte in der zweiten Reihe sitzen i would like to sit in incorrect incorrect 7
3625 Frag: ein Ticket für Montagabend tickets for the monday evening incorrect incorrect 7
3595 Sag: Ich möchte mit Pfund bezahlen can i pay by pounds incorrect correct 7
4686 Sag: Ich möchte um 22 Uhr morgen abreisen i like to leave tomorrow at ten o'clock pm correct correct 6
4676 Frag: Wo ist der Coiffeur? can you give me the direction to the hairdresser incorrect correct 6
4657 Frag: Wie viel kostet die Jeans? how much does the jeans cost incorrect correct 6
4624 Frag: Wo ist Piccadilly Circus? could you give me directions to piccadilly circus correct correct 6
4615 Frag: Ich möchte die Dessertkarte can i have the dessert menu please correct correct 6
4598 Sag: Dies ist nicht was ich bestellt habe that's not what i ordered correct correct 6
4587 Frag: Bohnen i would like some beans correct correct 6
4539 Sag: Kann ich mit Kreditkarte bezahlen can i pay with credit card incorrect correct 6
4518 Sag: Dies ist mein Pass yes this is my passport correct correct 6
4433 Frag: Wo ist Piccadilly Circus? where is the piccadilly circus incorrect correct 6
4426 Sag: Ich möchte mit Visa bezahlen i want pay with visa card incorrect correct 6
4390 Sag: Ich habe keine Reservation i do not a reservation incorrect correct 6
4340 Sag: Ich möchte mit Kreditkarte bezahlen i would like pay with the credit card incorrect correct 6
4311 Sag: Ich möchte mit Mastercard bezahlen i want buy with the master card incorrect correct 6
4292 Frag: ein Ticket nach Notting Hill a billet to nothing hill incorrect incorrect 6
4286 Frag: Wo ist das Naturhistorische Museum? i am looking for the naturally naturally history museum incorrect correct 6
4268 Frag: weisse Stiefel i would like white boots correct correct 6
4260 Frag: 4 Musical-Tickets i would like buy four musical tickets incorrect correct 6
4256 Frag: Wo ist das Fitnessstudio? where is the fitness studio correct correct 6
4239 Sag: Ich habe keine Reservation i haven't reservation incorrect correct 6
4236 Sag: Ich habe keine Reservation i don't have a reservation correct correct 6
4233 Sag: Marble Arch ist gegenüber dem Hyde Park the marble arch is nei on the other side of hyde park incorrect incorrect 6
4226 Frag: Wo ist das Science Museum? where is the science museum correct correct 6
4215 Frag: Wo ist das Fitnessstudio? where is the fitness studio correct correct 6
4204 Sag: Die Hauptstadt von Deutschland ist Berlin the capital of my country is berlin incorrect incorrect 6
4190 Frag: Doppelzimmer a double room correct correct 6
4155 Frag: Wo kann ich ein Shampoo kaufen? where i can buy shampoo incorrect correct 6
4109 Sag: Ich bin aus Australien i am from australia correct correct 6
4106 Sag: Ich bin aus Deutschland i am from germany correct correct 6
4100 Sag: Marble Arch ist gegenüber dem Hyde Park marble arch is in front of the hyde park incorrect correct 6
4095 Frag: eine L big incorrect correct 6
4092 Frag: Jeans i would like to have some jeans correct correct 6
4067 Sag: Ich möchte heute Abend gehen i want to go this evening correct correct 6
4048 Sag: Im Zimmer hat es keinen Haarföhn the room has not a hair dryer incorrect correct 6
3986 Sag: Ich habe keine Reservation no i do no i do not have a reservation correct correct 6
3840 Sag: Ich möchte vorne sitzen i want sit up front incorrect correct 6
3791 Frag: Zimmer für 2 Nächte a room for two nights correct correct 6
3782 Frag: Zimmer für 7 Nächte a room for seven nights correct correct 6
3728 Sag: Ich möchte mit Kreditkarte bezahlen i incorrect incorrect 6
3716 Sag: Ich möchte mit Postkarte bezahlen i want pay incorrect incorrect 6
3713 Sag: Dies ist mein Pass that's is my passport incorrect correct 6
3693 Frag: Gibt es einen Coiffeur? is there a hairdresser correct correct 6
3689 Sag: Ich habe keine Reservation no i do not have a reservation correct correct 6
3679 Frag: Gibt es ein Schwimmbad? have you a pool correct correct 6
3662 Sag: Ich möchte am Freitagabend gehen i would like to go a priday night incorrect incorrect 6
3652 Sag: Dies ist mein Pass this is my pass incorrect incorrect 6
3645 Sag: Die Hauptstadt von Österreich ist Wien the capital of austria is wien incorrect correct 6
3628 Frag: 2 Tickets für König der Löwen two tickets for the ke incorrect incorrect 6
3621 Sag: Ich heisse James i am james correct correct 6
3579 Frag: 3 Tickets für König der Löwen i want three tickets for the lion king correct correct 6
4695 Sag: Ich bin aus Italien i'm from italy correct correct 5
4694 Sag: Ich bin aus Italien i'm from italy correct correct 5
4693 Sag: Ich heisse Janet my name is janet correct correct 5
4691 Sag: Ich möchte am Mittwochabend abreisen i like to leave at wednesday evening incorrect correct 5
4690 Sag: Ich möchte am Mittwochabend abreisen i like to leave at wednesday night incorrect correct 5
4641 Frag: Welcher Bus fährt dorthin? which bus can i take correct correct 5
4636 Sag: Marble Arch ist gegenüber dem Hyde Park the marble parch nei incorrect incorrect 5
4631 Sag: Ich weiss es nicht i don't know correct correct 5
4625 Frag: Welcher Bus fährt dorthin? which bus can i take correct correct 5
4623 Frag: Wo ist Piccadilly Circus? where is the piccadilly circus incorrect correct 5
4606 Frag: eine Limonade can i have a lemonade correct correct 5
4591 Frag: eine Cola can i have a coke correct correct 5
4577 Sag: Dies ist nicht was ich bestellt habe this is not what i ordered correct correct 5
4575 Frag: meinen Sonntagsbraten medium i like my sunday roast medium correct correct 5
4569 Frag: Ich möchte die Dessertkarte can i have the dessert card incorrect correct 5
4564 Frag: einen Tee can i have a tea correct correct 5
4553 Sag: Ich möchte vorne sitzen i like to sat on the first row incorrect correct 5
4529 Sag: Ich möchte mit Schweizer Franken bezahlen i like pay with swiss francs incorrect correct 5
4522 Sag: Ich habe nur grosse Noten i have only great note incorrect incorrect 5
4480 Sag: Ich möchte am Samstagabend abreisen i like to leave at sat saturday evening incorrect correct 5
4466 Sag: Ich habe keine Reservation you can't ask help at the silver level incorrect incorrect 5
4460 Frag: eine L big incorrect correct 5
4440 Frag: weisse Stiefel i would like white boots correct correct 5
4411 Frag: ein Ticket nach Notting Hill can i have a ticket to notting hill correct correct 5
4399 Sag: Ich möchte um 18 Uhr morgen abreisen i want to leave at six pm tomorrow correct correct 5
4395 Sag: Ich habe keine Reservation you can't ask help at the silver level incorrect incorrect 5
4394 Sag: Ich habe keine Reservation i can't ask for help at the silver level incorrect incorrect 5
4373 Sag: Ich möchte mit Kreditkarte bezahlen i want pay with credit card incorrect correct 5
4333 Frag: Auberginen could i have some aubergines correct correct 5
4288 Frag: Wo ist Big Ben? where are the big ben incorrect correct 5
4283 Frag: Wie viel kostet die Hose? how much does the trousers cost incorrect correct 5
4276 Frag: rote Sandalen i would like red sandals correct correct 5
4240 Sag: Ich habe keine Reservation i haven't reservation incorrect correct 5
4202 Sag: Ich heisse Andy my name is andy correct correct 5
4184 Frag: Wo kann ich einen Haarföhn kaufen? where can i buy a hairdryer correct correct 5
4179 Sag: Das WC ist dreckig the toilet is dirty correct correct 5
4178 Frag: Wo ist der Lebensmittelladen? where is the grocery store correct correct 5
4165 Sag: Ich möchte mit Postkarte bezahlen i would pay with post card incorrect correct 5
4152 Sag: Im Zimmer hat es kein Shampoo in the room there's no shampoo correct correct 5
4132 Frag: Zimmer für 7 Nächte a room for seven nights correct correct 5
4108 Sag: Ich möchte um 16 Uhr morgen abreisen i would like to leave tomorrow at four p m correct correct 5
4103 Frag: Wo ist der Lift? and where is the lift correct correct 5
4094 Frag: eine L big incorrect correct 5
4091 Frag: Jeans i want some jeans correct correct 5
4061 Frag: ein Ticket zum Trafalgar Square can i have a ticket to trafalgar square correct correct 5
4047 Sag: Im Zimmer hat es keinen Haarföhn in the room is incorrect incorrect 5
3927 Frag: eine L great incorrect incorrect 5
3926 Frag: eine L great incorrect incorrect 5
3916 Frag: eine L great incorrect incorrect 5
3915 Frag: eine L big incorrect correct 5
3878 Frag: Ich möchte die Dessertkarte yes can i have the dessert card incorrect correct 5
3877 Frag: Erbsen i would like some peas correct correct 5
3867 Frag: Wo ist der Zoo? where is the zoo correct correct 5
3860 Frag: Wo ist der Tower? you can't use help at the silver level incorrect incorrect 5
3848 Sag: Ich möchte vorne sitzen i want to sit in the front correct correct 5
3842 Frag: Tickets für das British Museum i need tickets for the british museum correct correct 5
3798 Sag: Ich möchte mit Schweizer Franken bezahlen can i pay with swiss francs correct correct 5
3794 Frag: Doppelzimmer a twin room correct correct 5
3712 Sag: Dies ist mein Pass that is my passport correct correct 5
3666 Sag: Ich möchte in der achten Reihe sitzen in the eight row incorrect correct 5
3637 Frag: Einzelzimmer a single room correct correct 5
3634 Frag: 5 Tickets five tickets please correct correct 5
3577 Sag: Ich möchte am Montagabend gehen i would like to go on monday night correct correct 5
3576 Sag: Ich möchte am Montagabend gehen i would like to go on monday night correct correct 5
4692 Sag: Ich möchte am Mittwochabend abreisen i like to leave at wednesday night incorrect correct 4
4688 Sag: Ich bin aus Italien i am from italy correct correct 4
4684 Frag: Gibt es einen Lift? is there a lift correct correct 4
4668 Frag: Wie viel kostet es? how much does it cost correct correct 4
4645 Frag: Wo ist der Tower? where is the tower correct correct 4
4638 Frag: Wo ist das British Museum? where is the british museum correct correct 4
4637 Sag: Marble Arch ist gegenüber dem Hyde Park the marble arch is across to hyde park incorrect correct 4
4628 Frag: Wo ist das London Eye? where is the london eye correct correct 4
4613 Frag: Ich möchte die Dessertkarte can i have the dessert menu correct correct 4
4612 Sag: Dies ist nicht was ich bestellt habe this is not what what i ordered correct correct 4
4588 Frag: Ich möchte die Dessertkarte can i have the dessert menu correct correct 4
4579 Frag: Ich möchte die Dessertkarte can i have the dessert menu please correct correct 4
4578 Frag: Ich möchte die Dessertkarte can i have the dessert menu correct correct 4
4549 Frag: ein Ticket für Freitagabend i like to go at friday night incorrect correct 4
4520 Sag: Ich möchte mit Kreditkarte bezahlen i like to pay with credit card incorrect correct 4
4498 Sag: Ich bin aus Australien i am from australia correct correct 4
4490 Sag: Dies ist mein Pass yes this is my passport correct correct 4
4484 Sag: Ich heisse James my name is james correct correct 4
4473 Sag: Ich möchte am Dienstagabend abreisen i want to leave on tuesday evening correct correct 4
4437 Sag: Es ist nahe der Bank of England it is near the bank of england correct correct 4
4420 Sag: Ich möchte in der fünften Reihe sitzen i would like to sit in the fifth row correct correct 4
4409 Sag: Dies ist mein Pass this is my passport correct correct 4
4384 Sag: Ich möchte mit Kreditkarte bezahlen i want pay with credit card incorrect correct 4
4364 Frag: Zimmer für 2 Nächte i want a room for two nights correct correct 4
4357 Frag: Stiefel i would like some boots correct correct 4
4355 Frag: Wie viel kostet das T-Shirt? how much does the t-shirt cost correct correct 4
4348 Frag: blaue Sandalen i would like blue sandals correct correct 4
4327 Frag: eine Limonade i would like a lemonade correct correct 4
4259 Frag: 4 Musical-Tickets can i have four musical tickets correct correct 4
4258 Frag: Wo ist das Fitnessstudio? where is the fitness studio correct correct 4
4254 Sag: Ich möchte mit Visa bezahlen i would pay with visa incorrect correct 4
4246 Sag: Ich habe keine Reservation kuckuck incorrect incorrect 4
4216 Sag: Das WC ist dreckig the wc is sale incorrect incorrect 4
4210 Sag: Mir geht es gut i am good correct correct 4
4199 Sag: Ich möchte um 16 Uhr morgen abreisen i would like to leave tomorrow at four p m correct correct 4
4171 Frag: Twinzimmer twin room correct correct 4
4153 Sag: Im Zimmer hat es kein Shampoo there is no shampoo in the room correct correct 4
4148 Frag: Gibt es einen Coiffeur? is there a coiffeur incorrect correct 4
4129 Sag: Ich möchte mit Kreditkarte bezahlen i want to pay by credit card correct correct 4
4120 Frag: Wo ist der Lebensmittelladen? where is the grocery store correct correct 4
4099 Frag: Welcher Bus fährt dorthin? which bus drives there incorrect correct 4
3980 Frag: billigeres Ticket i want a cheaper ticket correct correct 4
3971 Sag: Ich heisse Silvia my name is silvia correct correct 4
3958 Sag: Dies ist mein Pass this is my passport correct correct 4
3888 Sag: Ich habe keine Reservation no i do not have a reservation correct correct 4
3846 Frag: ein Ticket für Samstagabend i need a ticket for saturday night correct correct 4
3837 Frag: ein Ticket für heute Abend i want tickets for this evening correct correct 4
3831 Frag: Ticket für dieses Musical i want tickets for this musical correct correct 4
3830 Frag: Ticket für dieses Musical i want a ticket for this musical correct correct 4
3828 Frag: ein Ticket für Donnerstagabend i need a ticket for thursday night correct correct 4
3816 Sag: Dies ist mein Pass this is my passport correct correct 4
3807 Sag: Ich habe keine Reservation no i don't have a reservation correct correct 4
3787 Frag: Gibt es ein Hotelrestaurant? is there a restaurant correct correct 4
3768 Frag: Gibt es einen Lift? is there a lift correct correct 4
3727 Frag: Einzelzimmer single room correct correct 4
3726 Frag: Einzelzimmer single room correct correct 4
3714 Sag: Dies ist mein Pass this is my passport correct correct 4
3659 Frag: 4 Musical-Tickets four musical tickets correct correct 4
3658 Frag: 4 Musical-Tickets four musical tickets please correct correct 4
3651 Sag: Dies ist mein Pass this is my passport correct correct 4
3638 Frag: Einzelzimmer a single room please correct correct 4
3632 Sag: Ich möchte in der zweiten Reihe sitzen i would like to sit in the second row correct correct 4
3631 Sag: Ich möchte in der zweiten Reihe sitzen i would sit in the incorrect incorrect 4
3629 Frag: 2 Tickets für König der Löwen two tickets for the king of lion incorrect correct 4
3620 Sag: Ich möchte um 21 Uhr morgen abreisen i want to leave at nine o'clock tomorrow correct correct 4
3604 Frag: ein Steak i want a steak correct correct 4
3596 Frag: ein Ticket nach Notting Hill i would like a ticket to notting hill correct correct 4
4701 Sag: Ich bin aus Portugal i am from portugal correct correct 3
4679 Frag: Doppelzimmer can i have a double room correct correct 3
4667 Frag: eine L i am a large correct correct 3
4662 Frag: Grösse 37 thirty-seven correct correct 3
4635 Sag: Marble Arch ist gegenüber dem Hyde Park the marble arch is across the hyde park incorrect incorrect 3
4603 Frag: Custard Tart i would like the custard tart correct correct 3
4597 Frag: die Tagessuppe can i have the soup of the day correct correct 3
4595 Frag: mein Steak durchgebraten i would like my steak well done correct correct 3
4592 Frag: ein Steak can i have a steak correct correct 3
4590 Frag: Ich möchte die Dessertkarte the dessert menu please correct correct 3
4571 Frag: Ich möchte die Dessertkarte the dessert menu please correct correct 3
4532 Frag: ein Ticket nach Notting Hill can i have a ticket to notting hill correct correct 3
4517 Sag: Dies ist mein Pass this is my passport correct correct 3
4488 Sag: Ich habe 4 Schwestern i have four sisters correct correct 3
4477 Sag: Ich möchte am Sonntag abreisen i want to leave on sunday correct correct 3
4475 Sag: Ich bin aus Österreich i am from austria correct correct 3
4446 Frag: eine S small correct correct 3
4443 Frag: ein T-Shirt i would like a t-shirt correct correct 3
4434 Frag: Wo ist Piccadilly Circus? where is the piccadilly circus incorrect correct 3
4432 Frag: Wo ist die National Gallery? where is the national gallery correct correct 3
4424 Frag: Zimmer für 4 Nächte i want a room for four nights correct correct 3
4417 Sag: Ich möchte am Dienstagabend gehen i want to go on tuesday evening correct correct 3
4392 Sag: Ich habe keine Reservation i have not a reservation incorrect correct 3
4388 Sag: Ich habe keine Reservation you can't ask help at silver level incorrect incorrect 3
4336 Frag: Ich möchte die Dessertkarte the dessert menu correct correct 3
4329 Frag: meinen Hamburger rare i would like my hamburger rare correct correct 3
4325 Frag: 2 Tickets two tickets correct correct 3
4315 Frag: 4 Musical-Tickets i would like to buy four musical tickets correct correct 3
4313 Frag: 4 Musical-Tickets four musical tickets correct correct 3
4304 Sag: Ich heisse Sarah my name is sarah correct correct 3
4272 Frag: ein T-Shirt i would like a t-shirt correct correct 3
4270 Frag: ein T-Shirt i would like a t-shirt correct correct 3
4264 Sag: Ich habe keine Reservation i don't have a reservation correct correct 3
4249 Sag: Ich habe keine Reservation funktioniert nid jo guet muesch nüt meh säge incorrect incorrect 3
4245 Sag: Ich habe keine Reservation you are a dubbel incorrect incorrect 3
4243 Sag: Ich habe keine Reservation no you are a dubbel incorrect incorrect 3
4234 Sag: Marble Arch ist gegenüber dem Hyde Park the arch is at the other side opposite hyde park incorrect incorrect 3
4229 Frag: Wo ist der Tower? where is the tower correct correct 3
4203 Sag: Ich bin aus Deutschland i am from germany correct correct 3
4187 Sag: Ich habe keine Reservation no i have no reservation correct correct 3
4159 Frag: 4 Tickets nach London four tickets to london correct correct 3
4158 Frag: 4 Tickets nach London four tickets for london correct correct 3
4157 Frag: 4 Tickets nach London four tickets to london correct correct 3
4145 Sag: Ich möchte mit Visa bezahlen i want to pay by visa correct correct 3
4142 Frag: Einzelzimmer single room correct correct 3
4137 Frag: Gibt es ein Hotelrestaurant? is there a hotel restaurant correct correct 3
4130 Frag: Gibt es eine Hotelbar? is there a hotel bar correct correct 3
4119 Frag: Wo ist der Lebensmittelladen? where is the grocery store correct correct 3
4098 Sag: Dies ist zu teuer no this is too expensive correct correct 3
4093 Frag: weisse Jeans i want white jeans correct correct 3
4073 Frag: Tickets für das London Eye can i have tickets for the london eye correct correct 3
4069 Frag: 3 Tickets für das Phantom der Oper i want three tickets for the phantom of the opera correct correct 3
4058 Sag: Ich möchte mit Mastercard bezahlen can i pay by master card correct correct 3
4051 Frag: Twinzimmer a twin room correct correct 3
4034 Sag: Ich möchte mit Kreditkarte bezahlen i want to pay with a credit card correct correct 3
4014 Sag: Ich habe keine Reservation i have no reservation correct correct 3
3988 Frag: Doppelzimmer a double room correct correct 3
3932 Frag: Stiefel i want boots correct correct 3
3919 Sag: Dies ist zu teuer that's too expensive correct correct 3
3912 Frag: Grösse 36 thirty-six correct correct 3
3910 Frag: Stiefel i want boots correct correct 3
3905 Frag: ein Ticket für Starlight Express a ticket for starlight express correct correct 3
3874 Frag: meinen Hamburger rare i want my hamburger rare correct correct 3
3868 Frag: Wo ist Madame Tussauds? where is madame tussauds correct correct 3
3861 Frag: Wo ist der Tower? where is the tower correct correct 3
3833 Frag: Tickets für Madame Tussauds i want tickets for madame tussauds correct correct 3
3823 Sag: Ich möchte in der vierten Reihe sitzen i would like in the fourth serious sit incorrect incorrect 3
3792 Frag: Doppelzimmer a double room correct correct 3
3783 Frag: Doppelzimmer a double room correct correct 3
3778 Sag: Ich habe keine Reservation no i do not have a reservation correct correct 3
3773 Sag: Ich habe keine Reservation i haven't got a reservation correct correct 3
3770 Sag: Ich habe keine Reservation i have no reservation correct correct 3
3744 Sag: Ich möchte um 12 Uhr morgen abreisen i want to leave tomorrow at noon correct correct 3
3743 Frag: ein Ticket nach London i want a ticket to london correct correct 3
3741 Frag: Gibt es ein Fitnessstudio? is there a gym correct correct 3
3729 Sag: Ich möchte mit Kreditkarte bezahlen ich verschtands nid incorrect incorrect 3
3725 Frag: Einzelzimmer a single room correct correct 3
3724 Frag: Zimmer für 2 Nächte can i have a room for two nights correct correct 3
3710 Sag: Ich gehe in die Ferien i going to holidays incorrect incorrect 3
3660 Frag: 4 Musical-Tickets i would like to buy four musical tickets correct correct 3
3636 Frag: Einzelzimmer a single room correct correct 3
3606 Frag: mein Steak rare i would like my steak rare correct correct 3
3602 Frag: eine Cola i would like a coke correct correct 3
4675 Frag: Wo ist der Coiffeur? where is the hairdresser correct correct 2
4671 Frag: Suite can i have a suite correct correct 2
4654 Frag: pinkige Jeans i would like pink jeans correct correct 2
4652 Frag: Grösse 39 thirty-nine correct correct 2
4608 Frag: mein Steak rare i like my steak rare correct correct 2
4594 Frag: ein Steak i would like the steak correct correct 2
4585 Frag: mein Steak medium i like my steak medium correct correct 2
4574 Frag: den Sonntagsbraten can i have the sunday roast correct correct 2
4533 Sag: Ich möchte mit Visa bezahlen can i pay with vi visa correct correct 2
4510 Sag: Das ist okay that is okay correct correct 2
4503 Sag: Ich heisse Jack my name is jack correct correct 2
4489 Sag: Ich gehe in die Ferien i go on the holiday incorrect correct 2
4464 Sag, dass du es nicht möchtest no i do not want it correct correct 2
4454 Frag: blaue Sandalen blue sandals please correct correct 2
4450 Sag: Dies ist zu teuer it's too expensive correct correct 2
4447 Frag: Wie viel kostet das T-Shirt? how much is this tee shirt correct correct 2
4430 Frag: Wo ist das Hotelrestaurant? where is the hotel restaurant correct correct 2
4428 Sag: Ich möchte mit Visa bezahlen i want pay with visa incorrect correct 2
4427 Sag: Ich möchte mit Visa bezahlen i want pay with visa incorrect correct 2
4406 Sag: Ich habe 3 ältere Brüder i have three older brothers correct correct 2
4400 Sag: Ich heisse Thomas my name is thomas correct correct 2
4396 Sag: Ich heisse Thomas my name is thomas correct correct 2
4387 Sag: Ich habe keine Reservation i have not a reservation incorrect correct 2
4380 Frag: Gibt es ein Hotelrestaurant? is there a hotel restaurant correct correct 2
4351 Frag: ein rotes T-Shirt i want a red t-shirt correct correct 2
4344 Sag: Mir geht es gut i'm good correct correct 2
4337 Frag: Custard Tart i would like the custard tart correct correct 2
4330 Frag: meinen Hamburger rare i would like my hamburger rare correct correct 2
4324 Frag: Tickets für das London Eye can i also have tickets for the london eye correct correct 2
4317 Frag: 4 Tickets für Mittwochabend four tickets for wednesday evening correct correct 2
4294 Frag: ein Ticket nach Notting Hill i would like a ticket to notting hill correct correct 2
4279 Frag: Hosen i would like trousers correct correct 2
4274 Frag: Sandalen i would like sandals correct correct 2
4273 Sag: Mir geht es gut i'm fine correct correct 2
4241 Sag: Ich habe keine Reservation i have not a reservation incorrect correct 2
4230 Frag: Wo ist der Tower? where is the tower correct correct 2
4219 Frag: Zimmer für 3 Nächte can i have a room for three nights correct correct 2
4208 Sag: Dies ist mein Pass this is my passport correct correct 2
4205 Sag: Die Hauptstadt von Deutschland ist Berlin the capital of germany is berlin correct correct 2
4195 Frag: Wo ist das Hotelrestaurant? where is the hotel restaurant correct correct 2
4182 Sag: Im Zimmer hat es keinen Haarföhn i could not find the hairdryer correct correct 2
4161 Sag: Ich habe keine Reservation i don't have a reservation correct correct 2
4144 Frag: Einzelzimmer single room correct correct 2
4141 Frag: Zimmer für 3 Nächte a room for three nights correct correct 2
4133 Frag: Einzelzimmer a single room correct correct 2
4090 Frag: Grösse 41 forty one correct correct 2
4089 Frag: Grösse 41 forty one correct correct 2
4065 Frag: 3 Musical-Tickets can i have three musical tickets correct correct 2
4046 Sag: Mein WC ist dreckig my toilet is dirty correct correct 2
4037 Sag: Mein Badezimmer ist dreckig my bathroom is dirty correct correct 2
4004 Frag: Tickets für das British Museum i need tickets for the british museum correct correct 2
4003 Sag: Ich möchte hinten sitzen i want to sit in the back correct correct 2
3997 Frag: 1 Ticket i need one ticket correct correct 2
3996 Frag: Tickets für das Science Museum i need tickets for the science museum correct correct 2
3979 Frag: Wie viel kostet es? how much is it correct correct 2
3974 Sag: Ich möchte am Freitagabend abreisen i want to leave on friday night correct correct 2
3966 Sag: Dies ist mein Pass yes it is my passport correct correct 2
3890 Frag: Suite i'd like a suite correct correct 2
3889 Frag: Zimmer für 2 Nächte a room for two nights correct correct 2
3856 Frag: Wo ist Piccadilly Circus? where is piccadilly circus correct correct 2
3841 Sag: Ich möchte vorne sitzen i would like to sit in the front correct correct 2
3822 Frag: 3 Tickets für Starlight Express i need three tickets for stairlight express correct correct 2
3815 Sag: Ich gehe in die Ferien i go in the holidays incorrect correct 2
3812 Sag: Ich bin aus Frankreich i am from france correct correct 2
3774 Sag: Ich habe keine Reservation i have not a reservation incorrect correct 2
3763 Frag: Zimmer für 6 Nächte i want a room for six nights correct correct 2
3738 Sag: Ich möchte mit Postkarte bezahlen i want to pay with credit card incorrect correct 2
3717 Sag: Ich möchte mit Postkarte bezahlen i would like to pay by post card correct correct 2
3703 Sag: Ich bin aus Spanien i am from spain correct correct 2
3698 Sag: Ich heisse Sarah my name is sarah correct correct 2
3697 Sag: Ich heisse Sarah my name is sarah correct correct 2
3667 Sag: Ich möchte in der achten Reihe sitzen i want to sit in the eighth row correct correct 2
3650 Sag: Ich gehe in die Ferien i going in the holiday incorrect correct 2
4685 Frag: ein Ticket nach London can i have a ticket to london correct correct 1
4664 Frag: ein gelbes Hemd i would like a yellow shirt correct correct 1
4630 Sag: Der Tower ist nahe der Tower Bridge the tower is near the tower bridge correct correct 1
4619 Frag: Ich möchte die Rechnung can i have the bill correct correct 1
4617 Frag: Zitronenkuchen i would like a citro cake incorrect incorrect 1
4607 Frag: ein Steak can i have the steak correct correct 1
4604 Frag: Ich möchte die Rechnung can i have the bill correct correct 1
4602 Frag: Ich möchte die Dessertkarte can i have the dessert menu correct correct 1
4600 Sag: Dies ist nicht was ich bestellt habe this is not what i ordered correct correct 1
4581 Frag: Ich möchte die Rechnung can i have the bill correct correct 1
4570 Frag: Ich möchte die Dessertkarte i would like the dessert card incorrect correct 1
4548 Frag: 1 Musical-Ticket can i have one musical ticket correct correct 1
4531 Sag: Ich habe kleinere Note i have smaller notes correct correct 1
4509 Sag: Das ist okay that is okay correct correct 1
4506 Sag: Ich bin aus Österreich i am from austria correct correct 1
4504 Sag: Ich bin aus Schottland i am from scotland correct correct 1
4502 Sag: Ich bin aus England i am from england correct correct 1
4500 Sag: Ich bin aus Amerika i am from america correct correct 1
4497 Sag: Ich heisse Erica my name is erica correct correct 1
4496 Sag: Dies ist mein Pass yes this is my passport correct correct 1
4491 Sag: Ich heisse Silvia my name is silvia correct correct 1
4478 Sag: Ich heisse Patrick my name is patrick correct correct 1
4474 Sag: Ich heisse Nick my name is nick correct correct 1
4458 Frag: ein Hemd i want a shirt correct correct 1
4452 Frag: Sandalen i am looking for sandals correct correct 1
4439 Frag: Stiefel i would like some boots correct correct 1
4438 Sag: Mir geht es gut i'm good correct correct 1
4425 Frag: Doppelzimmer can i have a double room please correct correct 1
4419 Frag: ein Ticket für Mamma Mia i want one ticket for mamma mia correct correct 1
4415 Frag: 1 Musical-Ticket can i have one musical ticket correct correct 1
4410 Sag: Dies ist mein Pass yes it is my passport correct correct 1
4408 Sag: Ich gehe in die Ferien i going on holidays incorrect correct 1
4403 Sag: Ich bin aus der Schweiz i am from switzerland switzerland correct correct 1
4401 Sag: Ich bin aus Spanien i am from spain correct correct 1
4397 Sag: Ich bin aus Irland i am from ireland correct correct 1
4382 Frag: Zimmer für 5 Nächte can i have a room for five nights correct correct 1
4374 Frag: Gibt es ein Hotelrestaurant? is there a hotel restaurant correct correct 1
4349 Frag: Grösse 41 forty-one correct correct 1
4339 Frag: Ich möchte die Rechnung i would like the check correct correct 1
4278 Frag: Grösse 42 forty-two correct correct 1
4265 Sag: Mir geht es gut i'm fine correct correct 1
4225 Frag: Wo ist der Coiffeur? where is the hairdresser correct correct 1
4201 Sag: Ich bin aus Kanada i am from canada correct correct 1
4156 Frag: Wo kann ich ein Shampoo kaufen? where can i buy shampoo correct correct 1
4154 Sag: Im Zimmer hat es kein Shampoo there is no shampoo in the room correct correct 1
4125 Frag: Einzelzimmer a single room correct correct 1
4123 Sag: Ich bin aus Schottland i am from scotland correct correct 1
4111 Sag: Ich bin aus Österreich i am from austria correct correct 1
4110 Sag: Ich heisse Janet my name is janet correct correct 1
4107 Sag: Ich bin aus Spanien i am from spain correct correct 1
4105 Frag: Wo ist der Lift? where is the lift correct correct 1
4102 Frag: Gibt es einen Lift? is there a lift correct correct 1
4044 Frag: Gibt es eine Hotelbar? do you have a hotel bar correct correct 1
4031 Frag: Wo ist der Coiffeur? where is the hairdresser correct correct 1
4020 Frag: Gibt es einen Lift? is there a a lift correct correct 1
4002 Frag: Ticket für dieses Musical i need a ticket for this musical correct correct 1
3999 Frag: 2 Tickets für heute Abend i need two tickets for tonight correct correct 1
3998 Frag: 2 Musical-Tickets i need two musical tickets correct correct 1
3983 Sag: Ich bin aus Australien i am from australia correct correct 1
3978 Sag: Ich möchte um 8 Uhr morgen abreisen i want to leave at eight o'clock tomorrow correct correct 1
3976 Sag: Ich bin aus Deutschland i am from germany correct correct 1
3972 Sag: Ich bin aus England i am from england correct correct 1
3962 Sag: Die Hauptstadt von Deutschland ist Berlin the capital of germany is berlin correct correct 1
3961 Sag: Ich bin aus Deutschland i am from germany correct correct 1
3953 Sag: Ich bin aus Italien i am from italy correct correct 1
3952 Sag: Ich heisse Sarah my name is sarah correct correct 1
3941 Frag: Sandalen i want sandals correct correct 1
3940 Sag: Mir geht es gut i'm fine correct correct 1
3930 Sag: Das T-Shirt ist zu teuer the t-shirt is too expensive correct correct 1
3929 Frag: Wie viel kostet es? how much is it correct correct 1
3928 Frag: eine L i am a large correct correct 1
3921 Frag: Sandalen i want sandals correct correct 1
3920 Sag: Mir geht es gut i am fine correct correct 1
3918 Frag: Wie viel kostet es? how much is it correct correct 1
3917 Frag: eine L i am a large correct correct 1
3909 Sag: Mir geht es gut i'm fine correct correct 1
3903 Frag: ein Ticket für Starlight Express i would like a ticket for starlight express correct correct 1
3898 Frag: 1 Musical-Ticket can i have one musical ticket correct correct 1
3883 Frag: Ich möchte die Rechnung i would like the bill correct correct 1
3882 Frag: Karrotenkuchen can i have the carrot cake correct correct 1
3879 Frag: Ich möchte die Dessertkarte can i have the dessert card incorrect correct 1
3873 Frag: einen Hamburger i want a hamburger correct correct 1
3854 Sag: Der Tower ist nahe der Tower Bridge the tower is near the tower bridge correct correct 1
3835 Frag: 1 Musical-Ticket i need one musical ticket correct correct 1
3827 Frag: 3 Musical-Tickets i need three musical tickets correct correct 1
3795 Sag: Ich möchte mit Postkarte bezahlen i would like to pay by post card correct correct 1
3746 Sag: Ich bin aus Deutschland i am from germany correct correct 1
3720 Frag: 2 Tickets für Montagabend i want two tickets for monday evening correct correct 1
3695 Frag: 3 Tickets nach London i would like three tickets to london correct correct 1
3680 Frag: Wo ist das Schwimmbad? where is the pool correct correct 1
3676 Frag: Doppelzimmer i want a double room correct correct 1
3675 Frag: Zimmer für 6 Nächte i want a room for six nights correct correct 1
3598 Sag: Ich möchte mit Mastercard bezahlen can i pay by master card correct correct 1
3574 Frag: Gibt es ein Fitnessstudio? have a fitness studio incorrect correct 1
4700 Sag: Ich heisse Sarah my name is sarah correct correct 0
4698 Sag: Ich heisse Paolo my name is paolo correct correct 0
4696 Frag: 3 Tickets nach London can i have three tickets to london correct correct 0
4689 Frag: 4 Tickets nach London can i have four tickets to london correct correct 0
4687 Sag: Ich heisse John my name is john correct correct 0
4680 Frag: Doppelzimmer can i have a double room correct correct 0
4673 Frag: Gibt es einen Coiffeur? is there a hairdresser correct correct 0
4670 Frag: Zimmer für 5 Nächte can i have a room for five nights correct correct 0
4669 Sag: Das Hemd ist zu teuer the shirt is too expensive correct correct 0
4663 Frag: ein Hemd can i have a shirt correct correct 0
4661 Frag: blaue Stiefel i would like blue boots correct correct 0
4660 Frag: Stiefel i would like boots correct correct 0
4659 Sag: Mir geht es gut i'm fine correct correct 0
4658 Sag: Dies ist zu teuer this is too expensive correct correct 0
4651 Frag: rote Sandalen i would like red sandals correct correct 0
4643 Frag: Wo ist das Science Museum? where is the science museum correct correct 0
4634 Frag: Wo ist das Science Museum? where is the science museum correct correct 0
4627 Frag: Wo ist das Science Museum? where is the science museum correct correct 0
4622 Frag: Wo ist die National Gallery? where is the national gallery correct correct 0
4584 Frag: ein Steak can i have a steak correct correct 0
4583 Frag: eine Fanta can i have a fanta correct correct 0
4572 Frag: ein Wasser ohne Kohlensäure can i have a still water correct correct 0
4568 Frag: etwas Knoblauchbrot i would like some garlic bread correct correct 0
4567 Frag: meinen Hamburger medium i would like my hamburger medium correct correct 0
4566 Frag: einen Hamburger i would like a hamburger correct correct 0
4565 Frag: einen Tee i would like a tea correct correct 0
4563 Frag: 3 Tickets can i have three tickets correct correct 0
4562 Frag: Tickets für die National Gallery can i have tickets for the national gallery correct correct 0
4560 Frag: 4 Tickets für Wicked can i have four tickets for wicked correct correct 0
4558 Frag: 4 Musical-Tickets can i have four musical tickets correct correct 0
4557 Frag: 4 Tickets can i have four tickets correct correct 0
4556 Frag: Tickets für die National Gallery can i have tickets for the national gallery correct correct 0
4552 Frag: ein Ticket für Billy Elliot can i have a ticket for billy elliot correct correct 0
4547 Frag: 2 Tickets can i have two tickets correct correct 0
4545 Sag: Ich möchte in der vierten Reihe sitzen i would like to sit in the fourth row correct correct 0
4544 Frag: 3 Tickets für Wicked can i have three tickets for wicked correct correct 0
4542 Frag: ein Ticket für Donnerstagabend i like to go at tuesday evening incorrect incorrect 0
4541 Frag: 3 Musical-Tickets can i have three musical tickets correct correct 0
4530 Sag: Ich möchte mit Schweizer Franken bezahlen can i pay with swiss francs correct correct 0
4523 Sag: Ich habe nur grosse Noten i only have large notes correct correct 0
4511 Sag: Das ist okay that is okay correct correct 0
4508 Sag: Österreich ist in Europa austria is in europe correct correct 0
4507 Sag: Österreich ist nicht in Afrika austria isn't in africa correct correct 0
4505 Sag: Ich heisse Jack my name is jack correct correct 0
4501 Sag: Ich heisse Michelle my name is michelle correct correct 0
4499 Sag: Ich heisse Mona my name is mona correct correct 0
4481 Sag: Ich heisse Barbara my name is barbara correct correct 0
4479 Frag: 3 Tickets nach London can i have three tickets to london correct correct 0
4463 Sag du möchtest nicht: dieses Hemd i do not want this shirt correct correct 0
4459 Frag: ein pinkiges Hemd i would like a pink shirt correct correct 0
4445 Frag: ein gelbes T-Shirt i would like a yellow t-shirt correct correct 0
4444 Frag: ein T-Shirt i would like a t-shirt correct correct 0
4441 Frag: weisse Stiefel i would like white boots correct correct 0
4431 Frag: Wo ist das Hotelrestaurant? where is the hotel restaurant correct correct 0
4429 Frag: Gibt es ein Hotelrestaurant? is there a hotel restaurant correct correct 0
4423 Frag: 2 Tickets can i have two tickets please correct correct 0
4422 Frag: Tickets für die National Gallery i would like tickets for the national gallery correct correct 0
4421 Sag: Ich möchte in der fünften Reihe sitzen i want to sit in the fifth row correct correct 0
4413 Sag: Ich möchte mit Visa bezahlen can i pay with visa correct correct 0
4405 Sag: Die Hauptstadt von der Schweiz ist Bern the capital of switzerland is bern correct correct 0
4402 Sag: Ich heisse Patrick my name is patrick correct correct 0
4398 Frag: ein Ticket nach London i want one ticket to london correct correct 0
4386 Frag: Wo ist der Lebensmittelladen? where is the grocery store correct correct 0
4383 Frag: Doppelzimmer can i have a double room correct correct 0
4381 Frag: Wo ist das Hotelrestaurant? where is the hotel restaurant correct correct 0
4375 Frag: Wo ist das Hotelrestaurant? where is the hotel restaurant correct correct 0
4371 Frag: Zimmer für 4 Nächte i want a room for four nights correct correct 0
4370 Sag: Ich habe keine Reservation no i do not have a reservation correct correct 0
4369 Frag: Wo ist das Fitnessstudio? where is the gym correct correct 0
4368 Frag: Gibt es ein Fitnessstudio? is there a gym correct correct 0
4366 Sag: Ich möchte mit Mastercard bezahlen can i pay with mastercard correct correct 0
4365 Frag: Einzelzimmer can i have a single room correct correct 0
4363 Sag: Ich habe keine Reservation no i do not have a reservation correct correct 0
4350 Frag: ein T-Shirt i want a t-shirt correct correct 0
4331 Frag: meinen Hamburger rare i would like my hamburger rare correct correct 0
4328 Frag: einen Hamburger i would like a hamburger correct correct 0
4326 Frag: 2 Tickets can i have two tickets correct correct 0
4321 Sag: Ich möchte vorne sitzen i would like to sit in the front correct correct 0
4319 Frag: 4 Tickets für das Phantom der Oper i want four tickets for the phantom of the opera correct correct 0
4318 Frag: 4 Tickets für Mittwochabend i want four tickets for wednesday night correct correct 0
4307 Sag: Ich bin aus England i am from england correct correct 0
4306 Sag: Ich heisse Silvia my name is silvia correct correct 0
4302 Sag: Ich möchte am Sonntagnachmittag abreisen i would like to leave on sunday afternoon correct correct 0
4300 Frag: 5 Tickets nach London i would like five tickets to london correct correct 0
4297 Sag: Ich möchte mit Mastercard bezahlen can i pay with mastercard correct correct 0
4290 Frag: Welcher Bus fährt dorthin? which bus goes there correct correct 0
4280 Frag: Hosen i am looking for trousers correct correct 0
4277 Frag: rote Sandalen red sandals correct correct 0
4267 Frag: Stiefel i would like some boots correct correct 0
4232 Frag: Welcher Bus fährt dorthin? which bus goes there correct correct 0
4207 Sag: Ich gehe in die Ferien i am on holiday correct correct 0
4200 Sag: Ich heisse Michelle my name is michelle correct correct 0
4194 Frag: Gibt es ein Hotelrestaurant? is there a hotel restaurant correct correct 0
4192 Frag: Doppelzimmer i would like a double room correct correct 0
4177 Frag: Gibt es einen Lebensmittelladen? is there a grocery store correct correct 0
4172 Frag: Twinzimmer a twin room please correct correct 0
4167 Frag: Wo ist das Schwimmbad? where is the swimming pool correct correct 0
4164 Frag: Doppelzimmer i would like a double room correct correct 0
4151 Sag: Mein WC ist dreckig my toilet is dirty correct correct 0
4150 Frag: Wo ist der Coiffeur? where is the hairdresser correct correct 0
4149 Frag: Gibt es einen Coiffeur? is there a hairdresser correct correct 0
4143 Frag: Einzelzimmer a single room correct correct 0
4140 Frag: Zimmer für 3 Nächte a room for three nights correct correct 0
4139 Frag: Zimmer für 3 Nächte a room for three nights correct correct 0
4138 Frag: Wo ist das Hotelrestaurant? where is the hotel restaurant correct correct 0
4131 Sag: Ich habe keine Reservation i don't have a reservation correct correct 0
4126 Frag: Einzelzimmer a single room please correct correct 0
4117 Frag: Gibt es einen Lebensmittelladen? is there a grocery store correct correct 0
4113 Frag: Wo ist der Coiffeur? where is the hairdresser correct correct 0
4112 Sag: Ich heisse Erica my name is erica correct correct 0
4088 Frag: weisse Stiefel i would like white boots correct correct 0
4085 Frag: Ich möchte die Dessertkarte i would like the dessert card incorrect correct 0
4075 Frag: 4 Tickets für morgen Abend i would like four tickets for tomorrow evening correct correct 0
4050 Frag: Zimmer für 6 Nächte a room for six nights correct correct 0
4045 Frag: Wo ist die Hotelbar? where is the hotel bar correct correct 0
4043 Sag: Ich möchte mit Mastercard bezahlen i would like to pay with mastercard correct correct 0
4042 Frag: Twinzimmer a twin room correct correct 0
4041 Frag: Zimmer für 3 Nächte a room for three nights correct correct 0
4036 Frag: Wo ist der Coiffeur? where is the hairdresser correct correct 0
4033 Frag: Twinzimmer a twin room correct correct 0
4032 Frag: Zimmer für 5 Nächte a room for five nights correct correct 0
4029 Frag: Wo ist der Coiffeur? where is the hairdresser correct correct 0
4028 Frag: Gibt es einen Coiffeur? is there a hairdresser correct correct 0
4026 Frag: Einzelzimmer i would like a single room correct correct 0
4025 Frag: Wo ist der Lebensmittelladen? where is the grocery store correct correct 0
4024 Frag: Gibt es einen Lebensmittelladen? is there a grocery store correct correct 0
4023 Sag: Ich möchte mit Visa bezahlen can i pay with visa correct correct 0
4022 Frag: Junior Suite i'd like a junior suite correct correct 0
4021 Frag: Zimmer für 3 Nächte a room for three nights correct correct 0
4019 Sag: Ich möchte mit Visa bezahlen can i pay with visa correct correct 0
4018 Frag: Suite i'd like a suite correct correct 0
4017 Frag: Zimmer für 2 Nächte a room for two nights correct correct 0
4016 Sag: Ich habe keine Reservation no i don't have a reservation correct correct 0
4012 Frag: Gibt es ein Hotelrestaurant? is there a hotel restaurant correct correct 0
4009 Sag: Ich möchte mit Visa bezahlen can i pay with visa correct correct 0
4008 Frag: Twinzimmer a twin room please correct correct 0
4006 Frag: Zimmer für 5 Nächte i want a room for five nights correct correct 0
4005 Frag: 3 Tickets i need three tickets correct correct 0
4000 Frag: 2 Tickets für Mamma Mia i need two tickets for mamma mia correct correct 0
3995 Sag: Ich möchte in der fünften Reihe sitzen i would like to sit in the fifth row correct correct 0
3993 Frag: 2 Tickets für Sonntagabend i need two tickets for sunday night correct correct 0
3992 Frag: 2 Musical-Tickets i need two musical tickets correct correct 0
3991 Frag: Gibt es einen Lift? is there a lift correct correct 0
3989 Frag: Doppelzimmer i would like a double room correct correct 0
3987 Frag: Zimmer für 6 Nächte a room for six nights correct correct 0
3982 Sag: Ich heisse Paul my name is paul correct correct 0
3981 Sag: Das ist okay this is okay correct correct 0
3977 Frag: 2 Tickets nach London i need two tickets to london correct correct 0
3975 Sag: Ich heisse Stephanie my name is stephanie correct correct 0
3973 Frag: 6 Tickets nach London i need six tickets to london correct correct 0
3970 Sag: Ich möchte am Freitagmorgen abreisen i want to leave on friday morning correct correct 0
3969 Frag: 2 Tickets nach London i need two tickets to london correct correct 0
3968 Sag: Ich möchte mit Kreditkarte bezahlen i would like to pay by credit card correct correct 0
3967 Frag: ein Ticket zum Green Park i need a ticket to green park correct correct 0
3963 Sag: Ich habe 4 ältere Brüder i have four older brothers correct correct 0
3960 Sag: Ich heisse Michelle my name is michelle correct correct 0
3959 Sag: Dies ist mein Pass yes it is my passport correct correct 0
3956 Sag: Ich habe 1 Bruder i have one brother correct correct 0
3954 Sag: Die Hauptstadt von Italien ist Rom the capital of italy is rome correct correct 0
3946 Frag: rote Hosen i want red trousers correct correct 0
3945 Frag: Hosen i'm looking for trousers correct correct 0
3943 Frag: Grösse 44 forty-four correct correct 0
3942 Frag: weisse Sandalen i want white sandals correct correct 0
3938 Frag: Wie viel kostet das T-Shirt? how much is the t-shirt correct correct 0
3936 Frag: ein braunes T-Shirt i want a brown t-shirt correct correct 0
3935 Frag: ein T-Shirt i want a t-shirt correct correct 0
3934 Frag: Grösse 39 thirty-nine correct correct 0
3933 Frag: braune Stiefel i want brown boots correct correct 0
3931 Sag: Mir geht es gut i'm fine correct correct 0
3925 Frag: ein gelbes T-Shirt i want a yellow t-shirt correct correct 0
3924 Frag: ein T-Shirt i want a t-shirt correct correct 0
3923 Frag: Grösse 38 thirty-eight correct correct 0
3922 Frag: rote Sandalen i want red sandals correct correct 0
3914 Frag: ein gelbes T-Shirt i want a yellow t-shirt correct correct 0
3913 Frag: ein T-Shirt i want a t-shirt correct correct 0
3911 Frag: weisse Stiefel i want white boots correct correct 0
3908 Frag: 4 Tickets i need four tickets correct correct 0
3907 Frag: Tickets für das London Eye can i have tickets for the london eye correct correct 0
3906 Sag: Ich möchte in der Mitte sitzen i want to sit in the middle correct correct 0
3897 Frag: Wo ist der Lebensmittelladen? where is the grocery store correct correct 0
3896 Frag: Gibt es einen Lebensmittelladen? is there a grocery store correct correct 0
3894 Frag: Suite i'd like a suite correct correct 0
3880 Frag: Ich möchte die Dessertkarte the dessert menu please correct correct 0
3872 Frag: einen Orangensaft i want an orange juice correct correct 0
3870 Sag: Der Tower ist nahe der Tower Bridge the tower is near tower bridge correct correct 0
3869 Frag: Welcher Bus fährt dorthin? which bus goes there correct correct 0
3866 Sag: Der Tower ist nahe der Tower Bridge the tower is near tower bridge correct correct 0
3865 Frag: Welcher Bus fährt dorthin? which bus goes there correct correct 0
3863 Frag: Wo ist die National Gallery? where is the national gallery correct correct 0
3859 Frag: Wo ist das Naturhistorische Museum? where is the natural history museum correct correct 0
3857 Frag: Welcher Bus fährt dorthin? which bus goes there correct correct 0
3855 Frag: Wo ist das Naturhistorische Museum? where is the natural history museum correct correct 0
3853 Frag: Welcher Bus fährt dorthin? which bus goes there correct correct 0
3851 Frag: Wo ist das Naturhistorische Museum? where is the natural history museum correct correct 0
3850 Frag: 1 Ticket i need one ticket correct correct 0
3849 Frag: Tickets für das Science Museum i want tickets for the science museum correct correct 0
3847 Frag: 2 Tickets für König der Löwen i need two tickets for the lion king correct correct 0
3845 Frag: 2 Musical-Tickets i need two musical tickets correct correct 0
3844 Frag: 2 Tickets i need two tickets correct correct 0
3843 Frag: Tickets für das British Museum can i also have tickets for the british museum correct correct 0
3839 Frag: Ticket für dieses Musical i want a ticket for this musical correct correct 0
3838 Frag: ein Ticket für Mamma Mia i want a ticket for mamma mia correct correct 0
3834 Frag: 1 Ticket i need one ticket correct correct 0
3832 Sag: Ich möchte in der zweiten Reihe sitzen i want to sit in the second row correct correct 0
3829 Frag: 3 Tickets für Wicked i need three tickets for wicked correct correct 0
3826 Frag: 4 Tickets i need four tickets correct correct 0
3825 Frag: Tickets für das London Eye i want tickets for the london eye correct correct 0
3824 Sag: Ich möchte in der vierten Reihe sitzen i would like to sit in the fourth row correct correct 0
3820 Frag: 3 Musical-Tickets can i have three musical tickets correct correct 0
3818 Sag: Dies ist mein Pass yes it is my passport correct correct 0
3814 Sag: Ich habe 2 jüngere Schwestern i have two younger sisters correct correct 0
3813 Sag: Die Hauptstadt von Frankreich ist Paris the capital of france is paris correct correct 0
3811 Sag: Ich heisse Barbara my name is barbara correct correct 0
3809 Frag: Junior Suite a junior suite correct correct 0
3808 Frag: Zimmer für 3 Nächte a room for three nights correct correct 0
3806 Frag: blaue Stiefel i would like blue boots correct correct 0
3805 Frag: Stiefel i would like some boots correct correct 0
3786 Sag: Ich möchte mit Postkarte bezahlen i would like to pay by post card correct correct 0
3781 Sag: Ich habe keine Reservation no i do not have a reservation correct correct 0
3767 Sag: Ich möchte mit Kreditkarte bezahlen i would like to pay by credit card correct correct 0
3745 Sag: Ich heisse Mona my name is mona correct correct 0
3742 Frag: Wo ist das Fitnessstudio? where is the gym correct correct 0
3737 Frag: Einzelzimmer i want a single room correct correct 0
3736 Frag: Zimmer für 3 Nächte i want a room for three nights correct correct 0
3735 Sag: Ich möchte mit Dollars bezahlen can i pay with dollars correct correct 0
3731 Frag: Wo ist der Lift? where is the lift correct correct 0
3723 Frag: Zimmer für 2 Nächte a room for two nights correct correct 0
3719 Frag: 2 Musical-Tickets i want two musical tickets correct correct 0
3718 Sag: Ich möchte mit Postkarte bezahlen i would like to pay by post card correct correct 0
3707 Sag: Ich bin aus Portugal i am from portugal correct correct 0
3706 Sag: Ich heisse Silvia my name is silvia correct correct 0
3705 Sag: Ich bin aus Amerika i am from america correct correct 0
3704 Sag: Ich heisse Brian my name is brian correct correct 0
3702 Sag: Ich heisse John my name is john correct correct 0
3701 Sag: Ich möchte am Sonntag abreisen i would like to leave on sunday correct correct 0
3700 Frag: ein Ticket nach London i want a ticket to london correct correct 0
3696 Sag: Ich möchte am Mittwochabend abreisen i want to leave on wednesday evening correct correct 0
3694 Frag: Wo ist der Coiffeur? where is the hairdresser correct correct 0
3692 Sag: Ich möchte mit Mastercard bezahlen can i pay by mastercard correct correct 0
3690 Frag: Zimmer für 7 Nächte i want a room for seven nights correct correct 0
3686 Sag: Ich möchte mit Visa bezahlen can i pay with visa correct correct 0
3685 Frag: Wo ist das Hotelrestaurant? where is the hotel restaurant correct correct 0
3684 Frag: Gibt es ein Hotelrestaurant? is there a hotel restaurant correct correct 0
3683 Sag: Ich möchte mit Visa bezahlen can i pay with visa correct correct 0
3682 Frag: Doppelzimmer i want a double room correct correct 0
3681 Frag: Zimmer für 4 Nächte i want a room for four nights correct correct 0
3673 Sag: Ich habe keine Reservation no i do not have a reservation correct correct 0
3670 Frag: 2 Tickets i want two tickets please correct correct 0
3665 Frag: 4 Tickets für Wicked i want four tickets for wicked correct correct 0
3656 Frag: ein Ticket zum Piccadilly Circus i would like a ticket to piccadilly circus correct correct 0
3649 Sag: Ich habe 1 Bruder i have one brother correct correct 0
3644 Sag: Ich bin aus Österreich i am from austria correct correct 0
3643 Sag: Ich heisse Nick my name is nick correct correct 0
3622 Sag: Ich bin aus der Türkei i am from turkey correct correct 0
3619 Frag: 2 Tickets nach London two tickets to london correct correct 0
3590 Frag: ein Ticket zum Green Park i need a ticket to green park correct correct 0
3573 Sag: Ich möchte mit Kreditkarte bezahlen can i pay by credit card correct correct 0
3572 Frag: Twinzimmer a twin room please correct correct 0
3571 Frag: Zimmer für 1 Woche i want a room for one week correct correct 0

Task instructions

Speech-processing task

In the speech version of the CALL shared task, each item consists of

  • an identifier
  • a German text prompt
  • an audio file containing an English language response

The data was collected from an online CALL tool used to help young Swiss German students improve their English fluency.

The task is to create software that will decide whether each response is appropriate (accept) or inappropriate (reject) in the context of the prompt. This will presumably require a combination of speech recognition and text processing methods. A response is considered appropriate if it both responds to the prompt in terms of meaning and is also correct English. For example, if the prompt is

"Frag: rote Stiefel"

("Ask for: red boots"), then "I would like some red boots" or "Red boots, please" are appropriate responses. "Give me brown boots" is inappropriate because it has the wrong meaning. "I wants red boots" is inappropriate because it is incorrect English.

The task is open-ended; there are many potentially appropriate responses to each prompt.

In this version of the task, no explicit attention is paid to quality of pronunciation.

Format of training data

The speech task training release directory contains the following resources:

  1. A set of 5248 audio files
  2. A CSV file of metadata in the format (headers and example lines)
Id Prompt Wavfile Transcription language meaning
11336 Frag: rote Stiefel 11336.wav i'd like red boots correct correct
7068 Frag: Wie viel kostet es? 7068.wav how many is it incorrect incorrect
8774 Frag: Ich möchte die Rechnung 8774.wav i want the bills incorrect correct

There is one line for each audio file. The specific CSV format is UTF-8, tab separated.

The 'language' and 'meaning' columns give judgements derived from three native speakers. The 'language' column contains the word "correct" if the response has been judged fully correct in terms of both language and meaning. The 'meaning' column contains the word "correct" if the response has been judged correct in terms of meaning, but not necessarily language.

Format of test data

The speech task test release directory, which will be released on Mar 13 2017, will contain the following resources:

  1. A set of 1000 audio files
  2. A CSV file of metadata in the format (headers and example lines)
Id Prompt Wavfile
11336 Frag: rote Stiefel 11336.wav
7068 Frag: Wie viel kostet es? 7068.wav
8774 Frag: Ich möchte die Rechnung 8774.wav

i.e. like the training data but without transcriptions or judgements. The specific CSV format is UTF-8, tab separated.

Format of answers

Groups who wish to submit an entry to the shared task should submit a CSV file, produced by running their system over the test data. The format should be the same as the training data, but with an extra column called Judgement added in which the possible values are 'accept' and 'reject'. For example:

Id Prompt Wavfile Judgement
11336 Frag: rote Stiefel 11336.wav accept
7068 Frag: Wie viel kostet es? 7068.wav reject
8774 Frag: Ich möchte die Rechnung 8774.wav reject

There should be one line for each line in the test data. The specific CSV format is once more UTF-8, tab separated.

Answer spreadsheets will be submitted by email to johanna.gerlach@unige.ch and emmanuel.rayner@unige.ch.

Scoring metric

The metric used to score the results is based on three intuitions:

  • The system should reject incorrect answers as often as possible, and reject correct answers as seldom as possible.
  • The more pronounced the difference between the system's response to incorrect as opposed to correct answers, the more useful it will be.
  • Some system errors are more serious than others. In particular, it is worse for the system to accept a sentence which is incorrect in terms of meaning than it is to accept one which is correct in terms of meaning but incorrect in terms of language.

The metric is defined as follows (there is further discussion in §4.1 of the LREC 2016 paper):

Each system response falls into one of five categories:

  1. Correct Reject: the student's answer is incorrect, the system rejects.
  2. Correct Accept: the student's answer is correct, the system accepts.
  3. False Reject: the student's answer is correct, the system rejects.
  4. Plain False Accept: the student's answer is correct in meaning but incorrect English, the system accepts.
  5. Gross False Accept: the student's answer is incorrect in meaning, the system accepts.

Define CR, CA, FR, PFA, GFA to be the number of utterances in each of the above categories, and put FA = PFA + k.GFA where k is a weighting factor that makes gross false accepts relatively more important. Then we define the differential response score, D, to be the ratio of the reject rate on incorrect answers to the reject rate on correct utterances:

D = ( CR/(CR + FA) ) / ( FR/(FR + CA) ) = CR(FR + CA) / FR(CR + FA)

We will use D as the metric for evaluating the quality of systems competing in the shared task, with the weighting factor k set equal to 3.

Recognition resources

A baseline recogniser for the task, built using the popular KALDI platform, will soon be available from the downloads tab.

Grammar resources

A sample grammar, based on the one in the app used to collect the data, is provided as part of the release. The grammar is in XML format, and associates each possible prompt with

  1. a translation of the prompt into English
  2. a set of possible responses.

A typical entry looks like this:

<prompt_unit>
<prompt>Sag: Ich möchte am Montagmorgen abreisen</prompt>
<translated_prompt>Ask for: I want to leave on monday morning</translated_prompt>
<response>i need to leave on monday morning</response>
<response>i need to leave on monday morning please</response>
<response>i should like to leave on monday morning</response>
<response>i should like to leave on monday morning please</response>
<response>i want to leave on monday morning</response>
<response>i want to leave on monday morning please</response>
<response>i would like to leave on monday morning</response>
<response>i would like to leave on monday morning please</response>
<response>i'd like to leave on monday morning</response>
<response>i'd like to leave on monday morning please</response>
</prompt_unit>

Important: the sample grammar is NOT INTENDED TO BE COMPLETE. As already noted, the task is open-ended.

Text-processing task

In the text version of the CALL shared task, each item consists of

  • an identifier
  • a German text prompt
  • an audio file containing an English language response
  • a text string representing the 1-best result of performing speech recognition on the audio file

The data was collected from an online CALL tool used to help young Swiss German students improve their English fluency.

The task is to create software that will decide whether each response is appropriate (accept) or inappropriate (reject) in the context of the prompt. This will require some kind of text processing method. A response is considered appropriate if it both responds to the prompt in terms of meaning and is also correct English. For example, if the prompt is

"Frag: rote Stiefel"

("Ask for: red boots"), then "I would like some red boots" or "Red boots, please" are appropriate responses. "Give me brown boots" is inappropriate because it has the wrong meaning. "I wants red boots" is inappropriate because it is incorrect English.

The task is open-ended; there are many potentially appropriate responses to each prompt.

Format of training data

The speech task training release directory contains the following resources:

  1. A set of 5248 audio files
  2. A CSV file of metadata in the format (headers and example lines)
Id Prompt Wavfile RecResult Transcription language meaning
11336 Frag: rote Stiefel 11336.wav i'd like red boots i'd like red boots correct correct
7068 Frag: Wie viel kostet es? 7068.wav how many is it how many is it incorrect incorrect
8774 Frag: Ich möchte die Rechnung 8774.wav i want the bill i want the bills incorrect correct

There is one line for each audio file. The specific CSV format is UTF-8, tab separated.

The 'language' and 'meaning' columns give judgements derived from three native speakers. The 'language' column contains the word "correct" if the response has been judged fully correct in terms of both language and meaning. The 'meaning' column contains the word "correct" if the response has been judged correct in terms of meaning, but not necessarily language.

The fact that speech recognition is often inaccurate means that there may not always be sufficient information to make a correct decision. For example, the third utterance should be rejected, since the student has replied with a grammatically incorrect sentence, but since the recogniser has corrected the error there is no way to determine this.

Format of test data

The text task test release directory, which will be released on Mar 13 2017, will contain the following resources:

  1. A set of 1000 audio files
  2. A CSV file of metadata in the format (headers and example lines)
Id Prompt Wavfile RecResult
11336 Frag: rote Stiefel 11336.wav i'd like red boots
7068 Frag: Wie viel kostet es? 7068.wav how many is it
8774 Frag: Ich möchte die Rechnung 8774.wav i want the bill

i.e. like the training data but without transcriptions or judgements. The specific CSV format is UTF-8, tab separated.

Format of answers

Groups who wish to submit an entry to the shared task should upload a CSV file, produced by running their system over the test data. The format should be the same as the test data, but with an extra column called Judgement added in which the possible values are 'accept' and 'reject'. For example:

Id Prompt Wavfile RecResult Judgement
11336 Frag: rote Stiefel 11336.wav i'd like red boots accept
7068 Frag: Wie viel kostet es? 7068.wav how many is it reject
8774 Frag: Ich möchte die Rechnung 8774.wav i want the bill reject

There should be one line for each line in the test data. The specific CSV format is once more UTF-8, tab separated.

Answer spreadsheets will be submitted by email to johanna.gerlach@unige.ch and emmanuel.rayner@unige.ch.

Scoring metric

The metric used to score the results is based on three intuitions:

  • The system should reject incorrect answers as often as possible, and reject correct answers as seldom as possible.
  • The more pronounced the difference between the system's response to incorrect as opposed to correct answers, the more useful it will be.
  • Some system errors are more serious than others. In particular, it is worse for the system to accept a sentence which is incorrect in terms of meaning than it is to accept one which is correct in terms of meaning but incorrect in terms of language.

The metric is defined as follows (there is further discussion in §4.1 of the LREC 2016 paper):

Each system response falls into one of five categories:

  1. Correct Reject: the student's answer is incorrect, the system rejects.
  2. Correct Accept: the student's answer is correct, the system accepts.
  3. False Reject: the student's answer is correct, the system rejects.
  4. Plain False Accept: the student's answer is correct in meaning but incorrect English, the system accepts.
  5. Gross False Accept: the student's answer is incorrect in meaning, the system accepts.

Define CR, CA, FR, PFA, GFA to be the number of utterances in each of the above categories, and put FA = PFA + k.GFA where k is a weighting factor that makes gross false accepts relatively more important. Then we define the differential response score, D, to be the ratio of the reject rate on incorrect answers to the reject rate on correct utterances:

D = ( CR/(CR + FA) ) / ( FR/(FR + CA) ) = CR(FR + CA) / FR(CR + FA)

We will use D as the metric for evaluating the quality of systems competing in the shared task, with the weighting factor k set equal to 3.

Grammar resources

A sample grammar, based on the one in the app used to collect the data, is provided as part of the release. The grammar is in XML format, and associates each possible prompt with

  1. a translation of the prompt into English
  2. a set of possible responses.

A typical entry looks like this:

<prompt_unit>
<prompt>Sag: Ich möchte am Montagmorgen abreisen</prompt>
<translated_prompt>Ask for: I want to leave on monday morning</translated_prompt>
<response>i need to leave on monday morning</response>
<response>i need to leave on monday morning please</response>
<response>i should like to leave on monday morning</response>
<response>i should like to leave on monday morning please</response>
<response>i want to leave on monday morning</response>
<response>i want to leave on monday morning please</response>
<response>i would like to leave on monday morning</response>
<response>i would like to leave on monday morning please</response>
<response>i'd like to leave on monday morning</response>
<response>i'd like to leave on monday morning please</response>
</prompt_unit>

Important: the sample grammar is NOT INTENDED TO BE COMPLETE. As already noted, the task is open-ended.

Baseline system resource

A Python3 script which carries out a baseline version of the text task is provided as part of the release. The script reads the sample XML grammar and a training data spreadsheet, then scores each item in the spreadsheet by matching the prompt and recognition result against the appropriate record in the grammar. If the recognition result is listed in the grammar as a possible response for the prompt, it is accepted, otherwise it is rejected. The results are written out as a new spreadsheet.

The files used (resource grammar, input spreadsheet and output spreadsheet) are defined at the top of the script.

Note that the script does not run under Python 2.x

Further information

The ideas behind the shared task are elaborated further in the paper: Baur, Claudia, Johanna Gerlach, Emmanuel Rayner, Martin Russell, and Helmer Strik. (2016). "A Shared Task for Spoken CALL?". Proc LREC 2016, Portoroz, Slovenia.

If you have questions, please contact us at johanna.gerlach@unige.ch

Please enter a valid email address to get acces to the data download.

Your email address will never be shared with any 3rd parties and we will use it only to keep you updated on shared task resources, notifications and deadlines.

Available downloads

Annotated test data

Annotated test data

Test data with language and meaning annotations (same as training data).

Speech-processing task test data

Test data will be released on Mar 13 2017
Speech processing task test data

Test data for the speech task. See task instructions tab for more info about the data format.

Text-processing task test data

Test data will be released on Mar 13 2017
Nuance text processing task test data

Test data for the text processing task using Nuance recognition results. See task instructions tab for more info about the data format.

Kaldi text processing task test data

Test data for the text processing task using Kaldi recognition results. See task instructions tab for more info about the data format.

Speech-processing task downloads

{{l.name}} ({{l.size}})

{{l.name}} coming soon

Kaldi resources

Speech recognition resources include acoustic and language models as well as scripts to construct a minimal baseline system to produce recognition results for the speech-processing task

{{l.name}} ({{l.size}})

{{l.name}} coming soon

Text-processing task downloads

Two data sets are available for the text-processing task: one with recognition results produced by Nuance with the original CALL-SLT grammar, another with recognition results produced by a baseline Kaldi system. Either one can be used for the task.

{{l.name}} ({{l.size}})

{{l.name}} coming soon

Common downloads (resources usable for both tasks)

{{l.name}} ({{l.size}})

{{l.name}} coming soon

2 Sep 2016: Minor correction to training data metadata sheet (1 line): line 284, item with Id 6156: language is 'incorrect'

13 Feb 2017: Correction of ~1% of judgments of training data

Submitting task results

Results should be submitted by email to johanna.gerlach@unige.ch and emmanuel.rayner@unige.ch.

Please submit a csv result file in the format specified in the task instructions tab.

Groups may submit up to three entries for each task. There is only one text task: entrants may use the Nuance or Kaldi versions of the data as they prefer, or combine them. When ranking the results, only the best entry from each group will be included.

Submission deadline: March 20, 2017, 23:59 CET.

Data annotation

In detail, the data was annotated as follows.

Three native speakers of English performed the annotation independently, using a web tool; for each utterance, the tool presented the original German prompt, an English translation of the prompt, and the response. The annotator filled in both the Language and Meaning fields together. Annotators did not refer to the sample grammar when annotating. The specific instructions given to the annotators are reproduced below.

A total of 6000 utterances were annotated. We had hoped that this would yield 5000 utterances where judgements from the three annotators were unanimous, but this proved slightly optimistic: in fact, only 4873 such utterances were found. In order to make up the shortfall, one of the authors, who had also previously served as an annotator, carefully reviewed the remaining utterances to select a subset where they judged that there was a strong probability the majority judgement was correct. The adjudicator tried to be as conservative as possible and exclude any examples where they felt at all uncertain. The utterances obtained using this method were added to the 'unanimous' set.

A Shared Task for Spoken CALL?

Claudia Baur, Johanna Gerlach, Manny Rayner, Martin Russell and Helmer Strik

bibtex hide bibtex

@InProceedings{BAUR16.654,
author = {Claudia Baur and Johanna Gerlach and Manny Rayner and Martin Russell and Helmer Strik},
title = {A Shared Task for Spoken CALL?},
booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)},
year = {2016},
month = {may},
date = {23-28},
location = {Portorož, Slovenia},
editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Sara Goggi and Marko Grobelnik and Bente Maegaard and Joseph Mariani and Helene Mazo and Asuncion Moreno and Jan Odijk and Stelios Piperidis},
publisher = {European Language Resources Association (ELRA)},
address = {Paris, France},
isbn = {978-2-9517408-9-1},
language = {english}
}

Interactive demo

This tab presents a toy interactive demo showing how the shared task works. On each line, you can see a German text prompt, a link to an audio file with the student's response to the prompt and a recognition result. The task is to decide whether the response is appropriate or not. The radio buttons in the "Accept/reject" column let you choose whether to accept or reject the student's response. The final columns (displayed when you press "Show score") give a transcription and gold standard judgements taken from native speakers of English. The "language" column indicates whether the response is fully correct, both in terms of meaning and in terms of being correct English. The "meaning" column only indicates whether the meaning is right, so is a weaker criterion of correctness.

There are two versions of the task:

  • For the speech version you should imagine that you are a system which includes a speech recogniser. You listen to the audio file, and decide on the basis of what you hear whether to accept or reject. For example, the prompt for the first line is "Frag : Ticket zum Trafalgar Square" ("Ask for: ticket to Trafalgar Square"), and the audio file has the response "A ticket to the Trafalgar Square". This is incorrect (the superfluous "the"), so should be rejected. The second line has the prompt "Frag : Zimmer für 7 Nächte" ("Ask for: room for 7 nights"), and the audio prompt has the response "hello". This is completely wrong, and should be rejected. The third line has the same prompt as the second, and the audio file has the response "A room for seven nights". This is completely correct, so should be accepted.
  • For the text version, you should imagine that that the speech recognition has already been done for you. You should NOT listen to the audio file, but make your decision only on the basis of what you see in the "Recognition result" column. If speech recognition was incorrect, you may not have enough information to make a good decision. For example, the second line has the prompt "Frag : Zimmer für 7 Nächte" ("Ask for: room for 7 nights"), and the recognition result is "hello". The obvious decision is to reject, which is correct. However, in the third line, the prompt is the same and the response is "an room for seven nights". This looks slightly wrong, since the first word should be "a" rather than "an". But in fact there has been a recognition error: the student has answered appropriately, and the correct decision is to accept.
Prompt
Audio
Recognition result
Accept/reject
Transcription
Language
Meaninng

{{e.prompt}}
{{e.recResult}}
{{e.transcription}}
correct
incorrect
correct
incorrect

Please complete annotations in Accept/reject column

Correct Reject: {{cr()}} (the student's answer is incorrect, the system rejects)

Correct Accept: {{ca()}} (the student's answer is correct, the system accepts)

False Reject: {{fr()}} (the student's answer is correct, the system rejects)

Plain False Accept: {{pfa()}} (the student's answer is correct in meaning but incorrect English, the system accepts)

Gross False Accept: {{gfa()}} (the student's answer is incorrect in meaning, the system accepts)


Weighted rejection rate on incorrect utterances: {{wRRIU() |textOrNumber:2}} = CR / (CR + PFA + k.GFA) = {{cr()}}/({{cr()}}+{{pfa()}}+3*{{gfa()}})

Rejection rate on correct utterances: {{rRCU() |textOrNumber:2}} = FR / (FR + CA) = {{fr()}}/({{fr()}}+{{ca()}})

Differential response score: {{score() |textOrNumber:2}} = Weighted rejection rate on incorrect utterances / Rejection rate on correct utterances

(or change selections in the Accept/reject column to see impact on the score)