\nThe chrony service does not change the time
\nThe often misconception is that the chrony service is setting the time to the one given by the NTP server. This is incorrect \u2013 what actually happens is that based on the answer from the NTP server, chrony just tells the system clock to go faster or slower. For this reason, sometimes even though the time is wrong and the NTP server is working, the time does not get corrected immediately.
\nOnly time when chrony sets time<\/p>\n
When the chrony service starts, there are some settings in the \/etc\/chrony\/chrony.conf file that tells it to actually set the time if specific conditions occur:<\/p>\n
# Force system clock correction at boot time.
\nmakestep 1000 10
\nwhich means that if chrony detects during the first 10 measurements after its start that the time is off by more than 1000 seconds it will set the clock.<\/p>\n
Some useful commands<\/p>\n
Below are some useful commands which can be used for the troubleshooting of chrony related issues.<\/p>\n
# chronyc tracking
\n# chronyc sources
\n# chronyc sourcestats
\n# systemctl status chronyd
\n# chronyc activity
\n# timedatectl
\nCheck chronyd status<\/p>\n
To check the status of the chronyd daemon :<\/p>\n
# systemctl status -l chronyd
\n? chronyd.service – NTP client\/server
\n Loaded: loaded (\/usr\/lib\/systemd\/system\/chronyd.service; enabled; vendor preset: enabled)
\n Active: active (running) since Fri 2016-08-12 13:22:22 IST; 1s ago
\n Process: 33263 ExecStartPost=\/usr\/libexec\/chrony-helper update-daemon (code=exited, status=0\/SUCCESS)
\n Process: 33259 ExecStart=\/usr\/sbin\/chronyd $OPTIONS (code=exited, status=0\/SUCCESS)
\n Main PID: 33261 (chronyd)
\n CGroup: \/system.slice\/chronyd.service
\n ??33261 \/usr\/sbin\/chronyd<\/p>\n
Aug 12 13:22:22 NVMBD1S11BKPMED03 systemd[1]: Starting NTP client\/server…
\nAug 12 13:22:22 NVMBD1S11BKPMED03 chronyd[33261]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
\nAug 12 13:22:22 NVMBD1S11BKPMED03 chronyd[33261]: Frequency 0.000 +\/- 1000000.000 ppm read from \/var\/lib\/chrony\/drift
\nAug 12 13:22:22 NVMBD1S11BKPMED03 systemd[1]: Started NTP client\/server.
\nThe chronyc sources command<\/p>\n
Running chronyc sources -v shows the current state of the NTP server\/s configured in the system. Here is an example output, in which ntp.example.com shows as a valid server which is online:<\/p>\n
# chronyc sources -v
\n210 Number of sources = 1<\/p>\n
.– Source mode ‘^’ = server, ‘=’ = peer, ‘#’ = local clock.
\n \/ .- Source state ‘*’ = current synced, ‘+’ = OK for sync, ‘?’ = unreachable,
\n| \/ ‘x’ = time may be in error, ‘~’ = time is too variable.
\n|| .- xxxx [ yyyy ] +\/- zzzz
\n|| \/ xxxx = adjusted offset,
\n|| Log2(Polling interval) -. | yyyy = measured offset,
\n|| \\ | zzzz = estimated error.
\n|| | |
\nMS Name\/IP address Stratum Poll LastRx Last sample
\n============================================================================
\n^* ntp.example.com 3 6 40 +31us[ -98us] +\/- 118ms
\nNote that a Source state different than \u2018*\u2019 usually indicates a problem with the NTP server.<\/p>\n
Source state \u2018~\u2019 means that the time is too variable
\nIf the Source state is \u2018~\u2018, it probably means that the server is accessible but the time is too variable. This can happen if the server responds too slow or responds sometimes slower and sometimes faster. You could check the response time of the pings to the server to see if they are slow or variable. This state has also been noticed when the server is running on virtual machines which are too slow causing timing issues.<\/p>\n
Chrony check and restart every hour<\/p>\n
Once an hour, the chrony service checks the output of the chronyc sources -v command, by running script \/usr\/sbin\/palladion_chrony_healthcheck which runs \/usr\/sbin\/palladion_check_chrony and checks its output:<\/p>\n
if \/usr\/sbin\/palladion_check_chrony returns 1 \u2013 it means there was no online source (no source with Source state = \u2018*\u2019) , so chrony restarts in an attempt to re-initialize the server status
\nif \/usr\/sbin\/palladion_check_chrony returns 0 \u2013 this means everything is ok, chrony does not need to be restarted because it already has a valid online source
\n# cat \/etc\/cron.d\/chrony
\nSHELL=\/bin\/sh
\nPATH=\/usr\/local\/sbin:\/usr\/local\/bin:\/sbin:\/bin:\/usr\/sbin:\/usr\/bin
\n#
\n# Check chrony every hour and restart if necessary.
\n#
\n16 * * * * root \/usr\/sbin\/palladion_chrony_healthcheck
\nChrony logs<\/p>\n
There are several chrony logs that can be used to troubleshoot. Most of them are located in \/var\/log\/chrony\/. Note that the latest file is not always the *.log one. Sometimes it happens that even the *.log.2 or *.log.3 file are the ones that are more recent. Here is an example of listing the files with sorting by the most recent:<\/p>\n
# ls -lisaht \/var\/log\/chrony\/
\ntotal 1.5M
\n3801115 580K -rw-r–r– 1 root root 574K Oct 21 14:56 measurements.log.3
\n3801131 544K -rw-r–r– 1 root root 540K Oct 21 14:56 statistics.log.3
\n3801166 356K -rw-r–r– 1 root root 350K Oct 21 14:56 tracking.log.3
\n3801089 4.0K drwxr-xr-x 16 root root 4.0K Oct 21 00:01 ..
\n3801114 4.0K drwxr-xr-x 2 root root 4.0K Oct 21 00:01 .
\n3801128 0 -rw-r–r– 1 root root 0 Oct 21 00:01 tracking.log
\n3801110 0 -rw-r–r– 1 root root 0 Oct 21 00:01 measurements.log
\n3801120 0 -rw-r–r– 1 root root 0 Oct 21 00:01 statistics.log
\n3801167 0 -rw-r–r– 1 root root 0 Oct 20 00:01 tracking.log.1
\n3801165 0 -rw-r–r– 1 root root 0 Oct 20 00:01 statistics.log.1
\n3801159 0 -rw-r–r– 1 root root 0 Oct 20 00:01 measurements.log.1
\n…………
\nTry setting only one NTP server by entering its IP address<\/p>\n
If until now you have been using two or more NTP servers (either because they were set or because you entered an FQDN that resolves in different IP addresses), try to set one single NTP server by entering only one IP address. This may solve your NTP related issue.<\/p>\n
Tracing the communication with the NTP server<\/p>\n
To double check if the NTP server is answering or not, it is possible to trace the traffic between chrony and the NTP server for a period of time while monitoring the server:
\n1. Start a pcap trace with tcpdump on the NTP port 123 and leave it running until the issue appears (run it in \u2018screen\u2019 or with \u2018nohup\u2019 to avoid it from being stopped if you disconnect from the shell command)
\n2. As soon as the issue re-appears, get a System Diagnostics covering the entire history since you have set the server to DNS name until the gap reoccurred. If this produces a file that is too big, just get the System Diagnostics for Current data and in addition copy all the files from \/var\/log\/chrony\/, and all files called \/var\/log\/syslog* . Remember to stop the trace you started at step 1<\/p>\n","protected":false},"excerpt":{"rendered":"
The chrony service does not change the time The often misconception is that the chrony service is setting the time to the one given by the NTP server. This is incorrect \u2013 what actually happens is that based on the answer from the NTP server, chrony just tells the system clock to go faster […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[73],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6726"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6726"}],"version-history":[{"count":1,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6726\/revisions"}],"predecessor-version":[{"id":6727,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6726\/revisions\/6727"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6726"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6726"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6726"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}