[PSST] New test -- check local clock for skew

[bbockelm]

Worker nodes with extreme skew can cause GSI failures that are difficult to debug. We should add this as a new PSST test.

Brian notes there is a timestamp in the TLS v1.2 handshake and this can be used as a lightweight mechanism to determine clock skews. Need to lookup an example link.

[roksys]

@bbockelm @stlammel
What about using ntpdate -q <hostname_of_ntp_server>?
We could use CERN NTP servers.

[bbockelm]

I worry that ntp would be blocked by the site, given its propensity to be used in attacks. Worth testing out.

Alternate ideas:

• Use something like tlsdate to extract the server date from a remote TLS connection.
• HTCondor can report the ServerTime as part of the query response. We could perhaps ask upstream if this is sufficiently lightweight to test from the pilots.

[roksys]

I added check for clock skew.

Local clock is compared with server clock (in this case https://google.com), which is taken from HTTP header. It has precision of ~ 1 sec. I guess it should be enough as only extreme skew can cause some problems.

Currently threshold is set to 5 sec.

[roksys]

What could be the maximum acceptable clock skew?
I was looking at the x509 spec, but did not find anything relevant.

[bbockelm]

A lot of implementations allow for a small number of minutes of clock skew - it's pretty inconsistent.

I would maybe set to critical at 5 minutes and warn at 10s?