How to fix leap second sleep issue without reboot

Question

I found that after the latest leap second insertion (2016-12-31 23:59:60), our CentOS7 application that has worker threads sleeping for 1 second between jobs, started to wake the sleeping threads immediately instead of in a second. In general, all sleeps are awake 1 second ahead of expected wake time.

The simplest and working solution is to reboot the box. But that is not desirable in our case. Is there a way to fix this without rebooting?

PS. For reference, here's a simple program in C++ that reproduces the issue.

#include <boost/date_time.hpp>
#include <boost/thread.hpp>
#include <iostream>

using namespace std;


// this has to be run in a thread to be able to detect the issue
void check_thread()
{
    size_t expected_delay = 1000;
    cout << "Expected delay: " << expected_delay << " ms" << endl;
    boost::posix_time::ptime t1 = boost::posix_time::microsec_clock::universal_time();
    boost::this_thread::sleep(boost::posix_time::milliseconds(1000));
    boost::posix_time::ptime t2 = boost::posix_time::microsec_clock::universal_time();
    size_t actual_delay = (t2 - t1).total_milliseconds();
    cout << "Actual delay: " << actual_delay << " ms" << endl;
    if (abs(expected_delay - actual_delay) > 900) {
        cout << "Too big delay difference: " << (expected_delay - actual_delay) << endl;
        cout << "Possible leap second issue" << endl;
    }
    else {
        cout << "No issues found" << endl;
    }
}

int main()
{
    boost::thread_group g;
    g.create_thread(check_thread);
    g.join_all();
    return 0;
}

Building:

g++ sleep_test.cpp -Wl,-Bstatic -lboost_thread -lboost_system -lboost_date_time -Wl,-Bdynamic -rdynamic -pthread

You'll have to restart the application. In future, apply updates in a timely manner. — Michael Hampton, Jan 13 '17 at 23:28

score 6 · Answer 1 · answered Jan 13 '17 at 23:22

6

Is your system time synched with ntpd or ptp? If not, update your tzdata package.

For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 7 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el7, or a later version, to receive this fix.

Resolve Leap Second Issues in Red Hat Enterprise Linux

answered Jan 13 '17 at 23:22

Troy Osborne

106
1
11

Installing a more recent tzdata-2016j-1.el7 does not fix the issue. Probably, it should be installed prior the leap second is inserted to help. Any other solutions? – amosk Jan 15 '17 at 12:46
I've found another solution: set time 1 second forward, then revert it. – amosk Jan 15 '17 at 13:28
1

@AnatolyMoskovsky If that was your solution, you can create an answer and mark it. – Troy Osborne Jan 16 '17 at 20:10

score 0 · Accepted Answer · answered Jan 26 '17 at 12:36

In addition to what Troy said, on RHEL7 systems that had not updated tzdata by the moment the leap second is applied, and not running ntpd, an extra step is required - manually set time 1 second forward, then revert it:

date -s "+1 sec"
date -s "-1 sec"

How to fix leap second sleep issue without reboot

2 Answers2