A power outage caused NIST's NTP server to deviate from UTC by less than 0.000005 seconds. What's the impact?
On December 17, 2023, the National Institute of Standards and Technology (NIST) announced that its NTP server was able to survive a power outage thanks to its backup power system. However, the outage caused a discrepancy of less than 5 microseconds, and engineer and YouTuber Jeff Gearing explained the impact.
NIST was 5 μs off UTC after last week's power cut | Jeff Geerling
NIST's NTP clock was microseconds from disaster - YouTube
It all began on December 17, 2025, when the power company cut off the supply due to strong winds exceeding 160 kilometers per hour, creating a risk of fire. The entire facility was placed on safety lockdown, with staff access restricted and backup generators running. However, a few days later, the generator powering the main ensemble clock for the NTP server failed.
Fortunately, the remaining staff on-site were able to connect emergency power and buy time with battery backups such as uninterruptible power supplies (UPS), while activating a backup power supply and preventing the NTP server from going down.
NIST power outage puts NTP server time distribution in danger, but the server continues to operate using backup power - GIGAZINE
The resulting deviation from Coordinated Universal Time (UTC) was ultimately reduced to less than 5 microseconds. NIST time is usually 5,000 times more accurate, and the latency for a typical computer communicating with an NTP server over the Internet is about 35 milliseconds (35,000 microseconds), so the problem didn't affect most users. However, certain fields, such as scientific research, financial trading, and broadcasting, require even higher accuracy in time synchronization, so the problem is not entirely unaffected.
Another important consideration is the relationship between GPS and time standards. GPS satellites are equipped with their own atomic clocks and transmit accurate time signals, but the final standard for maintaining this time is controlled to synchronize with Coordinated Universal Time (UTC) managed by the United States Naval Observatory (USNO). Any discrepancy between USNO time and NIST time affects systems that verify time synchronization by referencing both GPS and NIST.
Gearing runs two Raspberry Pi GPS clocks in his studio, one as the main NTP server and the other as a backup for testing. These get their time from an outdoor GPS antenna, but he expressed concern about the over-reliance on GPS for time synchronization infrastructure.
Gearing said the biggest lesson learned from this incident is the vulnerability of time synchronization infrastructure. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) has pointed out the risks inherent in over-reliance on GPS, and in response, the U.S. government has begun exploring alternative methods of positioning, navigation, and timing. One promising solution is the Broadcast Positioning System (BPS), which can ensure time redundancy even if GPS becomes unavailable.
Gearing also owns two GPS synchronized crystal oscillators (GPSDOs) with built-in rubidium atomic clocks. These allow for a holdover period, maintaining nanosecond accuracy even if the GPS signal is disrupted or the antenna fails.
Related Posts:
