Mar 21, 2023 18:00:00

An engineer reveals Ubisoft's internal system that sticks to a system that costs more than 100 million yen and does not listen to improvement requests

Yan Harashim, who got a job at Ubisoft, a game studio that produced numerous works such as the `` Assassin's Creed '' series, expressed his feelings about Ubisoft's internal circumstances

that time my manager spent $1M on a backup server that I never used

https://blog.dijit.sh//that-time-my-manager-spend-1m-on-a-backup-server

Harashim, who was tasked with maintaining a system that passes user profile information from game server to game server and demonstrating that the existing data storage that stores that information is high performance and durable. Over the course of months, the MySQL used by Ubisoft was thoroughly analyzed and performance tested to find internal locking bottlenecks and find out under what circumstances data could be lost.

As a result, it turns out that although MySQL can prevent data loss, internal locking slows performance on

many-core systems. The alternative, PostgreSQL, had much better performance, and had the added advantage of being able to cleanly separate write-ahead logs and data onto separate RAID devices, so Harashim decided to create and back up Postgres databases. It sounds like you've started researching ready-made solutions.

Harashim, who found a tool called pgBackRest among them and wanted to test it, requested Ubisoft to purchase the storage necessary to take backups for 90 days. However, this request is denied.

Ubisoft says it already has a standard backup solution, and it keeps it in cold storage in a bank vault in Paris, so there's no need for more.

Reluctantly, Harashim decided to test his solution using several 400GB HDDs. As a result of the test, Mr. Harashim's system seems to be able to operate quickly and transmit data at high speed. However, this system had the problem of slow loading speeds at times. Mr. Harashim's system reads the previously written data and creates an incremental backup, but he said that the reading performance of this data was terrible.

Mr. Harashim, who was dissatisfied with this, consulted with his manager and superiors, but the company's system is that 'even if the current system cannot reliably create backups, we will not purchase dedicated hardware for storing backups.' just be clearer. The reason was that 'I spent 1 million dollars (about 132 million yen), so if I do not use it, the impression will be bad'.

Meanwhile, a system that cost one million dollars malfunctions, and the data of Ubisoft's game ' Division ' is corrupted. At this time, the data was revived thanks to the backup that Mr. Harashim had made during the test, but it was not the latest data, so some progress in the game was lost. It seems that Mr. Harashim was able to obtain permission to purchase hardware safely soon after such experiences improved the internal system.

Through this experience, Mr. Harashim said, 'Even if you spend a million dollars to buy a product because it fits the general case, it doesn't necessarily fit all cases. Buy what you need. 'Sometimes that makes sense, but it's natural to wonder what features should be prioritized for the service or product you're buying.'