Let me suggest one solution for such cases: Use iterative tasks, in the spirit introduced by TurkIt.
For example, you want to create a caption for an image. You let a user create a caption. Then you take this caption and give it to another user, asking the user to improve it. Take the two versions and ask other workers, "which of the two versions is better?". Iterate until no improvement is possible.
Not a trivial setup, but gets around the binary accept/reject decisions problem and generates results of significantly superior quality.
Good idea. I've also thought about this for data entry/analysis type tasks. It is fully expected that a worker on any level won't be perfect, especially on these types of services. An approach I've considered is to have have 3 workers do the same task then compare results and take the majority answer which should be close to if not 100% accurate. Overtime you could even rate workers for accuracy and recruit/pay more if possible.
Comments
Let me suggest one solution for such cases: Use iterative tasks, in the spirit introduced by TurkIt.
For example, you want to create a caption for an image. You let a user create a caption. Then you take this caption and give it to another user, asking the user to improve it. Take the two versions and ask other workers, "which of the two versions is better?". Iterate until no improvement is possible.
Not a trivial setup, but gets around the binary accept/reject decisions problem and generates results of significantly superior quality.
Good idea. I've also thought about this for data entry/analysis type tasks. It is fully expected that a worker on any level won't be perfect, especially on these types of services. An approach I've considered is to have have 3 workers do the same task then compare results and take the majority answer which should be close to if not 100% accurate. Overtime you could even rate workers for accuracy and recruit/pay more if possible.
On a very selfish note, take a look at these:
http://qmturk.appspot.com/ http://code.google.com/p/get-another-label/
You may find the code useful for what you are trying to do.