Apr 25, 2020 15:00:00

Dropbox has completely improved its sync engine, where is the new engine?

Cloud storage service 'Dropbox' has completely improved the engine for synchronizing local storage and cloud storage. The focus of the improvements is on 'testability,' and Dropbox software engineer

Isaac Goldberg gives an overview of the new engine.

Testing sync at Dropbox-Dropbox
https://dropbox.tech/infrastructure/-testing-our-new-sync-engine

Previous sync engines put all the information in local metadata and synced the information between the local and cloud based on the metadata information.

In the new sync engine ' Nucleus ', there is a 'local tree' that represents local information, a 'remote tree' that represents information on the cloud, and 'final synchronization between the local and cloud'. By creating 3 trees of 'Synchronization tree' and comparing each local tree and remote tree against the synchronization tree, you can easily determine whether the folder or file has been changed locally or in the cloud. It is explained that it has become.

With the previous sync engine, it was difficult to test because the flexibility of synchronization was 'too tight'. For example, when synchronizing the 'cat' file in the 'baz' folder from the cloud, there is a case where the information in the 'cat' file is synchronized before the information in the 'baz' folder. He needed to represent an 'isolated' state like a 'cat' file. These complications made it difficult to tell if you're really having problems with file syncing, or just being isolated.

The new sync engine 'Nucleus' no longer needs to consider the 'isolated' state in applications above the sync engine by reporting an error at the protocol level whenever this happens. In addition, the conventional synchronization engine processes tasks with multiple threads, and its operation tends to be affected by the memory management of the OS and the usage status of the CPU, but Nucleus operates with one thread and affects the effect of the OS. By reducing the amount, it seems that the reproducibility of the test is improved.

Dropbox runs on hundreds of millions of devices and each environment is different. What is necessary to reproduce such an environment is a ' random test ' that inputs a random value into software. Random tests are difficult to ensure repeatability, and the traditional sync engine was one example. Nucleus seems to be able to easily reproduce the error by outputting the value that caused the error when the random test fails.

It is inefficient to enter completely random values for each of the local tree, remote tree, and synchronization tree when executing the random test. Dropbox has independently developed a tool for testing Nucleus synchronization called '

CanopyCheck ', which enables efficient testing of the process of three trees converging from different states to the same state, Goldberg said. I am.

If multiple people use the shared folder, it is possible that multiple people change the same file at the same time. Goldberg says that you should avoid losing the data you were editing because someone deleted the file you were editing. In order to test for such a situation, the same tool as CanopyCheck, 'Trinity', a proprietary tool, monitors Nucleus information and realizes a safe synchronization.

'By focusing on testability from the early stages of the Nucleus project, we were able to successfully rewrite the sync engine, and now the heart of Dropbox sync is more than ever,' said Goldberg. It's solid, 'he said.