GPT-4 shows elite human power in labeling tasks, saving 20,000 hours and over $600,000.

Comparing the labeling work done by experts and crowdworkers with the labeling work using GPT-4, the work content of GPT-4 is closer to the work content of experts than the average work content of crowdworkers. Turns out it was. By leaving it to GPT-4, the time that can be saved is 20,000 hours, and the cost that can be saved is 500,000 dollars (about 66.2 million yen). However, this fact also suggests that the future of crowdworking is grim.

[2304.03279] Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark

GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500,000 and 20,000 hours - Artisana

In order to achieve the task of ``annotating 572,322 text scenarios'', hiring a top-class worker at the world's number one labeling platform Surge AI costs $ 25 per hour (about $ 3,300) increase. For 20,000 hours of work, the cost is $500,000. This is a pretty big investment.

Seeking a cost-effective way to accomplish the task, researchers tested the ability to automate labeling with GPT-4 using custom prompts. We compared 2000 labeling data created by 3 experts and 3 crowdworkers with the content of labeling by GPT-4.

Then, labeling by GPT-4 showed a stronger correlation with labeling by experts than the average labeling performed by crowdworkers. With the help of GPT-4 in achieving the aforementioned tasks, the cost can be kept below $ 5000 (about 660,000 yen) and it is possible to save a lot of time.

This fact makes us feel that the future of crowdworking is not very bright, but Krystall Kuaffman of Turkopticon, an NPO that protects the rights of crowdworkers, still believes in human judgment, saying, ``Lighting is to judge. It's not just about generating sentences, people like Turkers will be needed for job decisions now, and for the foreseeable future. There are too many outstanding issues to have more confidence in GPT's capabilities than workers.'