Time
hwclock
hwclock is the “backup battery” on the mainboard. Before it checks against ntp servers, this hwclock is checked so you can get roughly close to real time. At the minimum, see if the times are close enough:
(1:885)# /bin/date && hwclock
Fri Sep 2 11:43:43 CDT 2022
2022-09-02 11:43:44.733264-0500
Note, you cannot change hwclock on a virtual instance like a cloud VPS. It is literally the “hardware” clock, which is emulated on a VPS.
Keep in mind that timezones might make logs look weird too. UTC +6 vs EST. Best practice is to have all servers on the same timezone setting.
ntpq/timedatectl
ntpq (redhat) and timedatectl (debian) are the main programs you’ll use to investigate. Most logs are dumped via syslog to /var/log/messages or /var/log/syslogs and can be seen with:
grep ntp /var/log/syslogs
Sep 2 11:09:23 someserver ntpd[24731]: proto: precision = 0.071 usec (-24)
Sep 2 11:09:23 someserver ntpd[24731]: restrict: ignoring line 40, mask '::' unusable.
Sep 2 11:09:23 someserver ntpd[24731]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): good hash signature
Sep 2 11:09:23 someserver ntpd[24731]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): loaded, expire=2022-12-28T00:00:00Z last=2017-01-01T00:00:00Z ofs=37
Sep 2 11:09:23 someserver ntpd[24731]: Listen and drop on 0 v4wildcard 0.0.0.0:123
Sep 2 11:09:23 someserver ntpd[24731]: Listen normally on 1 lo 127.0.0.1:123
Sep 2 11:09:23 someserver ntpd[24731]: Listen normally on 2 eth0 x.x.x.x:123
Sep 2 11:09:23 someserver ntpd[24731]: Listen normally on 3 eth0 10.13.0.7:123
Sep 2 11:09:23 someserver ntpd[24731]: Listening on routing socket on fd #20 for interface updates
Sep 2 11:09:24 someserver ntpd[24731]: Soliciting pool server 216.232.132.102
Sep 2 11:09:25 someserver ntpd[24731]: Soliciting pool server 203.57.51.100
Sep 2 11:09:25 someserver ntpd[24731]: Soliciting pool server 209.51.161.238
Sep 2 11:09:26 someserver ntpd[24731]: Soliciting pool server 142.147.88.111
Sep 2 11:09:26 someserver ntpd[24731]: Soliciting pool server 208.81.1.244
Sep 2 11:09:26 someserver ntpd[24731]: Soliciting pool server 185.35.202.197
Sep 2 11:09:27 someserver ntpd[24731]: Soliciting pool server 69.89.207.199
Sep 2 11:09:27 someserver ntpd[24731]: Soliciting pool server 38.100.216.142
Sep 2 11:09:27 someserver ntpd[24731]: Soliciting pool server 84.16.67.12
Sep 2 11:09:27 someserver ntpd[24731]: Soliciting pool server 216.229.4.66
Sep 2 11:09:28 someserver ntpd[24731]: Soliciting pool server 66.220.9.122
Sep 2 11:09:28 someserver ntpd[24731]: Soliciting pool server 178.33.101.170
Sep 2 11:09:28 someserver ntpd[24731]: Soliciting pool server 91.189.94.4
Sep 2 11:09:29 someserver ntpd[24731]: Soliciting pool server 185.125.190.58
Sep 2 11:09:29 someserver ntpd[24731]: Soliciting pool server 162.159.200.1
Sep 2 11:09:29 someserver ntpd[24731]: Soliciting pool server 185.120.22.12
Sep 2 11:09:30 someserver ntpd[24731]: Soliciting pool server 91.189.91.157
Sep 2 11:09:30 someserver ntpd[24731]: Soliciting pool server 50.205.244.36
Sep 2 11:09:30 someserver ntpd[24731]: Soliciting pool server 2600:3c00::f03c:91ff:fe96:cbca
Sep 2 11:30:50 someserver ntpd[24731]: 216.232.132.102 local addr 162.243.14.186 -> <null>
Sep 2 11:34:08 someserver ntpd[24731]: 216.229.4.66 local addr 162.243.14.186 -> <null>
Sep 2 11:34:12 someserver ntpd[24731]: 185.35.202.197 local addr 162.243.14.186 -> <null>
Sep 2 11:34:14 someserver ntpd[24731]: 208.81.1.244 local addr 162.243.14.186 -> <null>
Sep 2 11:34:18 someserver ntpd[24731]: 178.33.101.170 local addr 162.243.14.186 -> <null>
Sep 2 11:34:22 someserver ntpd[24731]: 203.57.51.100 local addr 162.243.14.186 -> <null>
Sep 2 11:35:21 someserver ntpd[24731]: 142.147.88.111 local addr 162.243.14.186 -> <null>
Sep 2 11:35:25 someserver ntpd[24731]: 50.205.244.36 local addr 162.243.14.186 -> <null>
JournalCTL is another way to view logs, but it is only usable on certain platforms and gives the same info as syslog:
(1:906)# journalctl -u ntp.service
-- Logs begin at Sat 2018-10-27 12:08:45 CDT, end at Fri 2022-09-02 12:26:02 CDT. --
Sep 02 11:09:23 someserver systemd[1]: Starting Network Time Service...
Sep 02 11:09:23 someserver ntpd[24727]: ntpd 4.2.8p10@1.3728-o (1): Starting
Sep 02 11:09:23 someserver ntpd[24727]: Command line: /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 103:105
Sep 02 11:09:23 someserver systemd[1]: Started Network Time Service.
Sep 02 11:09:23 someserver ntpd[24731]: proto: precision = 0.071 usec (-24)
Sep 02 11:09:23 someserver ntpd[24731]: restrict: ignoring line 40, mask '::' unusable.
Sep 02 11:09:23 someserver ntpd[24731]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): good hash signature
Sep 02 11:09:23 someserver ntpd[24731]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): loaded, expire=2022-12-28T00:00:00Z last=2017-01-01T00:00
Sep 02 11:09:23 someserver ntpd[24731]: Listen and drop on 0 v4wildcard 0.0.0.0:123
Sep 02 11:09:23 someserver ntpd[24731]: Listen normally on 1 lo 127.0.0.1:123
Sep 02 11:09:23 someserver ntpd[24731]: Listen normally on 2 eth0 x.x.x.x:123
Sep 02 11:09:23 someserver ntpd[24731]: Listen normally on 3 eth0 10.13.0.7:123
Sep 02 11:09:23 someserver ntpd[24731]: Listening on routing socket on fd #20 for interface updates
Sep 02 11:09:24 someserver ntpd[24731]: Soliciting pool server 216.232.132.102
Sep 02 11:09:25 someserver ntpd[24731]: Soliciting pool server 203.57.51.100
Sep 02 11:09:25 someserver ntpd[24731]: Soliciting pool server 209.51.161.238
Sep 02 11:09:26 someserver ntpd[24731]: Soliciting pool server 142.147.88.111
Sep 02 11:09:26 someserver ntpd[24731]: Soliciting pool server 208.81.1.244
Sep 02 11:09:26 someserver ntpd[24731]: Soliciting pool server 185.35.202.197
Sep 02 11:09:27 someserver ntpd[24731]: Soliciting pool server 69.89.207.199
Sep 02 11:09:27 someserver ntpd[24731]: Soliciting pool server 38.100.216.142
Sep 02 11:09:27 someserver ntpd[24731]: Soliciting pool server 84.16.67.12
Sep 02 11:09:27 someserver ntpd[24731]: Soliciting pool server 216.229.4.66
Sep 02 11:09:28 someserver ntpd[24731]: Soliciting pool server 66.220.9.122
Sep 02 11:09:28 someserver ntpd[24731]: Soliciting pool server 178.33.101.170
Sep 02 11:09:28 someserver ntpd[24731]: Soliciting pool server 91.189.94.4
Sep 02 11:09:29 someserver ntpd[24731]: Soliciting pool server 185.125.190.58
Sep 02 11:09:29 someserver ntpd[24731]: Soliciting pool server 162.159.200.1
Sep 02 11:09:29 someserver ntpd[24731]: Soliciting pool server 185.120.22.12
Sep 02 11:09:30 someserver ntpd[24731]: Soliciting pool server 91.189.91.157
Sep 02 11:09:30 someserver ntpd[24731]: Soliciting pool server 50.205.244.36
Sep 02 11:09:30 someserver ntpd[24731]: Soliciting pool server 2600:3c00::f03c:91ff:fe96:cbca
Sep 02 11:30:50 someserver ntpd[24731]: 216.232.132.102 local addr 162.243.14.186 -> <null>
Sep 02 11:34:08 someserver ntpd[24731]: 216.229.4.66 local addr 162.243.14.186 -> <null>
Sep 02 11:34:12 someserver ntpd[24731]: 185.35.202.197 local addr 162.243.14.186 -> <null>
Sep 02 11:34:14 someserver ntpd[24731]: 208.81.1.244 local addr 162.243.14.186 -> <null>
Sep 02 11:34:18 someserver ntpd[24731]: 178.33.101.170 local addr 162.243.14.186 -> <null>
Sep 02 11:34:22 someserver ntpd[24731]: 203.57.51.100 local addr 162.243.14.186 -> <null>
Sep 02 11:35:21 someserver ntpd[24731]: 142.147.88.111 local addr 162.243.14.186 -> <null>
Sep 02 11:35:25 someserver ntpd[24731]: 50.205.244.36 local addr 162.243.14.186 -> <null>
Sep 02 11:50:02 someserver ntpd[24731]: 66.220.9.122 local addr 162.243.14.186 -> <null>
Public vs Private
public IP are … public, make sure you are accessing servers that respond and are serving time. private IP are likely something internal like a switch or other server
ntpq commands
ntpq has many commands:
ntpq> ?
ntpq commands:
:config drefid mreadlist readvar
addvars exit mreadvar reslist
apeers help mrl rl
associations host mrulist rmvars
authenticate hostnames mrv rv
authinfo ifstats ntpversion saveconfig
cl iostats opeers showvars
clearvars kerninfo passociations sysinfo
clocklist keyid passwd sysstats
clockvar keytype peers timeout
config-from-file lassociations poll timerstats
cooked lopeers pstats version
cv lpassociations quit writelist
debug lpeers raw writevar
delay monstats readlist
You can review this info in the “man ntpq”
Peers
ntpq gives solid info with the “peers” command or -p:
st = stratum, and stratum 16 means “unreliable, don’t use” st 1,2,3 are what you’ll likely see with 1 being rude unless you have permission. when = last time it checked reach = an octal representation of the last 8 checks (which is then represented in binary). Anything besides 377 is an issue. You can see binary of octal using this conversion.
delay/offset will depend almost entirely on your network connection and will vary depending on what network you use to access the servers. jitter is the variation found between max and min each time it checks.
You would not typically want only 1 ntp server, instead you would want 3+ so the daemon can figure out the best time from all inputs.
config files are going to be /etc/ntp.conf or /etc/sysconfig/ntp.conf and you should update your pool to servers close to you, or at least in your own country:
(1:902)# cat /etc/ntp.conf | grep pool
pool 0.ubuntu.pool.ntp.org iburst
pool 1.ubuntu.pool.ntp.org iburst
pool 2.ubuntu.pool.ntp.org iburst
pool 3.ubuntu.pool.ntp.org iburst
pool ntp.ubuntu.com
You can see in the next command that the associations in my ntp.conf (default) are rejected.
inside of ntpq, the “associations” command will show all known servers it can query
rv (details)
You can see details of a particular peer with the rv command
ntpq> rv 55862
associd=55862 status=0x1414,
srcadr=185.125.190.58, srcport=123, dstadr=x.x.x.x, dstport=123,
leap=0, stratum=2, precision=-25, rootdelay=1.175, rootdisp=0.244,
refid=167.28.20.25, reftime=0xe6bcb210.bb4a0c05,
rec=0xe6bcb22f.644d6797, reach=0xff, unreach=0, hmode=3, pmode=4,
hpoll=7, ppoll=7, headway=0, flash=0x0, keyid=0, offset=1.016,
delay=69.124, dispersion=3.261, jitter=5.965, xleave=0.058,
filtdelay= 78.06 69.72 69.30 69.12 69.37 69.21 84.24 112.71,
filtoffset= 4.90 1.20 0.74 1.02 0.89 1.14 9.06 14.02,
filtdisp= 0.00 0.98 1.95 2.91 3.90 4.86 5.82 6.81
Crude Speed Check
If you are trying to decide from a list of peers, you can see from the “associations” command that some seem better than others.
If for some reason you cannot use this info you can do a homebrew bash script test to check a list of servers in a batch and see if one is responding slowly consistently. Just update the “D” with a list of servers you want to test and it will ping them in a loop.
D="8.8.8.8 208.67.222.222 4.2.2.2"; for ip in $D; do ping -c 1 $ip | grep from; done
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=2.71 ms
64 bytes from 208.67.222.222: icmp_seq=1 ttl=60 time=4.30 ms
64 bytes from 4.2.2.2: icmp_seq=1 ttl=60 time=2.53 ms
timedatectl
timedatectl is a bit easier to read, but is systemd based (debian)
(1:888)# timedatectl
Local time: Fri 2022-09-02 11:53:24 CDT
Universal time: Fri 2022-09-02 16:53:24 UTC
RTC time: Fri 2022-09-02 16:53:25
Time zone: America/Chicago (CDT, -0500)
System clock synchronized: yes
systemd-timesyncd.service active: yes
RTC in local TZ: no