A story of a failure that resulted in a loss of over 1.5 million yen due to an overlooked bug in the ChatGPT generated code



Ashim Shrestha, co-founder of data extraction startup Reworkd, wrote in a blog post that he had copied and used the code generated by ChatGPT, resulting in many users being unable to sign up for the service and missing out on more than $10,000 in profits.

How a single ChatGPT mistake cost us $10,000+ | Blog

https://asim.bearblog.dev/how-a-single-chatgpt-mistake-cost-us-10000/

The problem dates back to May 2023, when Shrestha and his project team were starting to monetize the startup. While Shrestha was instructed by investors to hurry up with monetization, he also needed to migrate the project from Next.js to Python / FastAPI, so he decided to have ChatGPT do almost all of the migration work.



After confirming that the copy-and-paste ChatGPT code worked without any problems, Shrestha and his team immediately implemented it in a production environment. However, the next day, they discovered that they had received a flood of emails from users complaining that they were unable to register in the middle of the night.

Curiously, the problem didn't occur during working hours and couldn't be reproduced, so Shrestha and his team struggled to identify the cause, and it took them five days to resolve the problem, during which they received 50 emails a day complaining about the problem and lost $10,000 in revenue that would have been made had those users successfully signed up for the service.

What we finally discovered was the bug in the code below on line 56, where we pass in a unique ID string instead of a function to generate an ID.



This meant that when a new user tried to sign up for a subscription, their ID was duplicated with that of a previous user, and they were unable to sign up.

However, Shrestha and his team were running eight Amazon ECS tasks, each with five execution environments, and every time a staff member committed, up to 40 unique IDs were generated, so the problem did not surface during the day.

However, when the business ended, commits stopped and new IDs were no longer generated, meaning that users who tried to sign up for the service in the middle of the night were unable to subscribe.

Shrestha says he spent five unforgettable days dealing with the complaints, but when he published his story about the gaffe on his blog, he was inundated with criticism and had to edit the article.

In the postscript, Shrestha wrote, 'What we did was egregious, shameful, and avoidable, and should have been avoided. It was also nothing more than human error, and in retrospect, a very blatant problem.'

Related Posts:

in Software, Posted by log1l_ks