Interesting possibility for crowdsourcing native checks!
NOTE: Original unpaged
Anyone with an Amazon account can either
submit HITS or work on HITS that were submitted by others. Workers are sometimes referred to as “Turkers” and people designing the HITS are “Requesters.” Requesters can specify the amount that they will pay for each item that is completed. Payments are frequently as low as $0.01. Turkers are free to select whichever HITS interest them. Amazon provides three mechanisms to help ensure quality: First, Requesters can have each HIT be completed by multiple Turkers, which allows higher quality labels to be selected, for instance, by taking the majority label. Second, the Requester can require that all workers meet a particular set of qualifications, such as sufficient accuracy on a small test set or a minimum percentage of previously accepted submissions. Finally, the Requester has the option of rejecting the work of individual workers, in which case they are not paid. The level of good-faith participation by Turkers is surprisingly high, given the generally small nature of the payment.
we were skeptical that
Turkers would have sufficient language skills to produce translations. Our translation HIT had the following instructions: Translate these sentences Your task is to translate 10 sentences into English. Please make sure that your English translation: _ Is faithful to the original in both meaning and style _ Is grammatical, fluent, and natural-sounding English _ Does not add or delete information from the original text _ Does not contain any spelling errors When creating your translation, please: _ Do not use any machine translation systems _ You may look up a word on wordreference.com if you do not know its translation Afterwards, we’ll ask you a few quick questions about your language abilities. We solicited translations for 50 sentences in French, German, Spanish, Chinese and Urdu, and designed the HIT so that five Turkers would translate each sentence.
a second group of Turkers cleaned the results.
Detect machine translation Please use two online machine translation systems to translate the text into English, and then copy-and-paste the translations into the boxes below. Finally, look at a list of translations below and click on the ones that look like they came from the online translation services. We automatically excluded Turkers whose translations were flagged 30% of the time or more. Quality of Turkers’ translations Our 50 sentence test sets were selected so that we could compare the translations created by Turkers to translations commissioned by the Linguistics Data Consortium. For the Chinese, French, Spanish, and German translations we used the the Multiple- Translation Chinese Corpus.3 This corpus has 11 reference human translations for each Chinese source sentence. We had bilingual graduate students translate the first 50 English sentences of that corpus into French, German and Spanish, so that we could re-use the multiple English reference