ntp
Check Point Firewall NTP Time
NTP Overview
NTP works based on a tier of upstream time servers defined as strata. Strata 1 is the most accurate and uses specialized timing such as satellite sync. Strata 2 asks strata 1 and so on, down to strata 16 which are considered unreliable.
Best practice is to have 4-10 time servers for the most accurate time sync.
Version 4 is the latest version and is backwards compatible with Version 3. V3 and V4 use security that V1 and V2 do not use.
Time is important for cryptographic functions, certificates, log reporting and many other activities.
NTP works over port 123/UDP
Verify and Compare
This is the basic check. If these are the same, no action is needed.
hwclock and /bin/date should both report the actual, accurate time
Use watch and a command like the below to verify multiple cluster members at the same time.
[Expert@fw:0]# /bin/date && hwclock && clish -c "show clock"
Fri Sep 2 16:37:40 EDT 2022
Fri Sep 2 16:29:37 2022 -0.410451 seconds
Fri Sep 2 16:37:41 2022 EDT
ntpstat
[Expert@fw:0]# ntpstat
synchronised to NTP server (1.2.3.4) at stratum 6
time correct to within 1350 ms
polling server every 64 s
To see details
[Expert@fw:0]# ntpstat -pn
remote refid st t when poll reach delay offset jitter
==============================================================================================
1.2.3.4 1.2.3.6 5 u 55 64 1 0.130 81.994 10.212
1.2.3.5 1.2.3.7 5 u 55 64 1 0.130 81.994 10.212
Reach should show “377” (it will show 1, until it has enough checks completed) Jitter should be under 100
Check Syncronization
“unspecified” is an error. Normally it shows the peer it is synchronized to.
[Expert@fw:0]# ntpstat
synchronised to unspecified at stratum 6
time correct to within 19539 ms
polling server every 1024 s
A normal view would look similar to this:
[Expert@fw]# ntpstat
synchronised to NTP server (10.253.7.13) at stratum 6
time correct to within 563 ms
polling server every 512 s
This should say synchronized, but is showing instead an error:
[Expert@fw:0]# clish -c "show ntp current"
primary and secondary servers are not synchronized
A normal display looks like this:
[Expert@fw]# clish -c "show ntp current"
10.253.7.13
Verify Peer Information
See sk92602 for additional info about check point ntp.
Enter ntpq shell:
[Expert@fw:0]# ntpq
Jitter should be closer to 100 than 1600, preferably 0 Remote is the peer being asked st is the stratum (5) Poll is the poll interval (1024) Reach is an octal number representing the last 8 checks (377 is good)
n=`eval printf "%d" 0377`; dc -e "$n 2op"
11111111
ntpq> peers
remote refid st t when poll reach delay offset jitter
==============================================================================
some.host.a 1.4.5.6 5 u 561 1024 377 0.485 505392. 1668.19
some.host.b 1.5.4.2 5 u 973 1024 377 0.531 505164. 1636.34
View Associations
Look up information such as status 902a, reject, reach (if no), etc
ntpq> as
ind assid status conf reach auth condition last_event cnt
===========================================================
1 19764 902a yes yes none reject sys_peer 2
2 19765 901a yes yes none reject sys_peer 1
View Association Details
You can see details of an association ID by using the rv command followed by the id to investigate.
ntpq> rv 19764
associd=19764 status=902a conf, reach, sel_reject, 2 events, sys_peer,
srcadr=some.host.h, srcport=123, dstadr=10.1.3.3,
dstport=123, leap=00, stratum=5, precision=-6, rootdelay=101.913,
rootdisp=442.612, refid=10.1.3.5,
reftime=e6bce606.cdded35f Fri, Sep 2 2022 16:28:54.804,
rec=e6bce514.302f5589 Fri, Sep 2 2022 16:24:52.188, reach=377,
unreach=0, hmode=3, pmode=4, hpoll=10, ppoll=10, headway=0,
flash=400 peer_dist, keyid=0, offset=505392.555, delay=0.485,
dispersion=35.156, jitter=1668.194, xleave=0.017,
filtdelay= 0.52 0.48 0.63 0.51 0.61 1.70 0.47 0.49,
filtoffset= 505850. 505392. 504913. 504309. 504085. 503571. 503105. 502633.,
filtdisp= 15.63 31.77 47.95 63.91 80.10 96.16 112.00 127.60
Look up the flash codes
400 means “peer distance too far”, or it took longer than 1.5s somewhere in the ntp stratum. I tested this one and can see that it’s responding quickly to the one listed, so somewhere upstream there is an issue.
Jitter should be much lower, closer to 100 or less. Ideally 0. Offset should be very low. Other firewalls were at 0 to 62, not 504k A normal display looks closer to this:
[Expert@fw:0]# ntpq
ntpq> peers
remote refid st t when poll reach delay offset jitter
==============================================================================
+some.host.a 1.2.7.112 4 u - 512 377 0.333 -96.279 22.539
*some.host.b 1.2.7.111 5 u 93 512 377 0.192 -46.625 54.705
ntpq> as
ind assid status conf reach auth condition last_event cnt
===========================================================
1 31475 945a yes yes none candidate sys_peer 5
2 31476 962a yes yes none sys.peer sys_peer 2
ntpq> rv 31476
associd=31476 status=962a conf, reach, sel_sys.peer, 2 events, sys_peer,
srcadr=some.host.t, srcport=123, dstadr=10.2.4.3,
dstport=123, leap=00, stratum=5, precision=-6, rootdelay=101.837,
rootdisp=331.848, refid=1.2.7.111,
reftime=e6bcfad2.39ace174 Fri, Sep 2 2022 17:57:38.225,
rec=e6bcfbbf.cd2f8961 Fri, Sep 2 2022 18:01:35.801, reach=377,
unreach=0, hmode=3, pmode=4, hpoll=9, ppoll=9, headway=0, flash=00 ok,
keyid=0, offset=-46.625, delay=0.192, dispersion=30.557, jitter=54.705,
xleave=0.016,
filtdelay= 0.35 0.20 0.19 0.30 0.99 0.22 0.59 0.23,
filtoffset= -123.04 -32.50 -46.62 -81.62 -71.75 -125.02 -125.67 -20.92,
filtdisp= 15.63 23.55 31.33 39.39 47.53 55.27 63.21 67.18
Review Syslogs
Ntpd by default reports to syslog, which will show up in /var/log/messages. Make sure you are looking the right log as it rotates frequently:
[Expert@fw:0]# pwd
/var/log
[Expert@fw:0]# ls -alF messages*
-rw------- 1 admin root 781436 Sep 2 17:13 messages
-rw-rw-r-- 1 admin root 1073280 Sep 1 10:56 messages.1
-rw-r--r-- 1 admin root 1057555 Aug 17 18:12 messages.10
-rw-rw-r-- 1 admin root 1049662 Aug 30 17:23 messages.2
-rw-rw-r-- 1 admin root 1120658 Aug 29 04:31 messages.3
-rw-r--r-- 1 admin root 1056547 Aug 27 10:18 messages.4
-rw-r--r-- 1 admin root 1048903 Aug 26 14:43 messages.5
-rw-r--r-- 1 admin root 1056799 Aug 24 13:01 messages.6
-rw-r--r-- 1 admin root 1105503 Aug 22 21:49 messages.7
-rw-r--r-- 1 admin root 1063438 Aug 21 03:36 messages.8
-rw-r--r-- 1 admin root 1053111 Aug 19 09:24 messages.9
View the log entries with grep:
[Expert@fw:0]# grep ntp messages
[Expert@fw:0]# grep ntp messages.1
[Expert@fw:0]# grep ntp messages.2
Finally something is found in messages.10
[Expert@fw:0]# grep ntp messages.10
Aug 16 08:29:20 2022 fw ntpd[10283]: 0.0.0.0 0613 03 spike_detect +0.132155 s
Aug 16 08:48:48 2022 fw ntpd[10283]: 0.0.0.0 0618 08 no_sys_peer
Aug 16 09:22:03 2022 fw ntpd[10283]: 0.0.0.0 061c 0c clock_step -0.135575 s
Aug 16 09:22:03 2022 fw ntpd[10283]: 0.0.0.0 0615 05 clock_sync
Aug 16 09:22:04 2022 fw ntpd[10283]: 0.0.0.0 c618 08 no_sys_peer
Aug 16 10:34:29 2022 fw ntpd[10283]: 0.0.0.0 0613 03 spike_detect -0.192106 s
Aug 16 10:51:52 2022 fw ntpd[10283]: 0.0.0.0 061c 0c clock_step -0.130591 s
Aug 16 10:51:52 2022 fw ntpd[10283]: 0.0.0.0 0615 05 clock_sync
Aug 16 10:51:53 2022 fw ntpd[10283]: 0.0.0.0 c618 08 no_sys_peer
Aug 16 17:54:12 2022 fw ntpd[10283]: 0.0.0.0 0613 03 spike_detect -0.160660 s
Aug 16 18:29:18 2022 fw ntpd[10283]: 0.0.0.0 061c 0c clock_step -0.160506 s
Verify Process is Running
[Expert@fw:0]# ps -ef | grep ntp | grep -v grep
admin 10283 10262 0 Aug08 ? 00:00:02 /usr/sbin/ntpd -n -g -c /etc/ntp.conf
View Generated Configs
Do not edit this file, but it gives us clues to the ntp configuration:
[Expert@fw:0]# cat /etc/ntp.conf
# This file was AUTOMATICALLY GENERATED
# Generated by /bin/ntp_xlate on Mon Aug 8 14:56:15 2022
#
# DO NOT EDIT
#
restrict default ignore
restrict 127.0.0.1
server 1.2.7.111 version 1 iburst
restrict 1.2.7.111 nomodify notrap nopeer noquery
server 1.2.7.112 version 1 prefer iburst
restrict 1.2.7.112 nomodify notrap nopeer noquery
driftfile /var/lib/ntp/ntp.drift
View GAIA Config
Some of our firewalls are using version 1 and some are using version 4. A packet capture shows that at least one of them is wrong:
[Expert@fw:0]# clish -c "show configuration" | grep ntp
set ntp active on
set ntp server primary 1.2.7.111 version 1
set ntp server secondary 1.2.7.112 version 1
Verify with packet capture
[Expert@fw:0]# tcpdump -nnei eth1 host 1.2.7.111 and port 123
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:19:13.817108 e4:43:4b:35:85:64 > 00:00:0c:07:ac:02, ethertype IPv4 (0x0800), length 90: 10.2.4.3.123 > 1.2.7.111.123: NTPv1, Client, length 48
18:19:13.817497 d8:67:d9:6f:5e:c3 > e4:43:4b:35:85:64, ethertype IPv4 (0x0800), length 90: 1.2.7.111.123 > 10.2.4.3.123: NTPv3, Server, length 48
Note: I have updated firewalls to version 4 (as it is backwards compatible with v3)
How to Update Time
Method 1 (all linux):
/bin/date MMDDHHMMCCYY
Method 2 (clish only):
[Expert@fw:0]# clish -c "set time 16:43:00"
Verify Time is Now Accurate:
In monitoring for about an hour, time drifted almost 30 seconds. (this is of course a problem!)
[Expert@fw:0]# /bin/date && hwclock && clish -c "show clock"
Fri Sep 2 17:39:40 EDT 2022
Fri Sep 2 17:38:37 2022 -0.410451 seconds
Fri Sep 2 17:39:41 2022 EDT
Restart NTPD
If you need to issue a restart to the daemon, it will not affect traffic and can be done as a last resort to see if synchronization begins again:
[Expert@fw:0]# service ntpd restart
Shutting down ntpd: [ OK ]
Starting ntpd: [ OK ]
Note, in my example, the fw took almost an hour to finally synchronize with it’s peer. Here is the final (notice jitter and condition compared to before restart)
[Expert@fw:0]# ntpq -pn
remote refi st t when poll reach delay offset jitter
==============================================================================
+1.2.3.4 1.2.3.23 5 u 54 64 377 0.453 81.406 59.110
*1.2.5.4 1.2.5.41 5 u 50 64 377 0.433 -10.227 35.042
[Expert@fw:0]# ntpq
ntpq> as
ind assid status conf reach auth condition last_event cnt
===========================================================
1 17773 942a yes yes none candidate sys_peer 2
2 17774 964a yes yes none sys.peer sys_peer 4
ntpq> rv 17774
associd=17774 status=964a conf, reach, sel_sys.peer, 4 events, sys_peer,
srcadr=some.host.z, srcport=123, dstadr=10.254.1.5,
dstport=123, leap=00, stratum=5, precision=-6, rootdelay=101.837,
rootdisp=341.904, refid=1.2.5.41,
reftime=e6bcfef7.a46a3d8b Fri, Sep 2 2022 18:15:19.642,
rec=e6bcff1a.98b22ecd Fri, Sep 2 2022 18:15:54.596, reach=377,
unreach=0, hmode=3, pmode=4, hpoll=6, ppoll=6, headway=377, flash=00 ok,
keyid=0, offset=25.854, delay=0.465, dispersion=19.263, jitter=27.590,
xleave=0.017,
filtdelay= 0.51 0.56 0.48 0.61 0.46 0.50 0.49 0.67,
filtoffset= 93.03 36.19 32.23 25.28 25.85 24.16 17.26 1.55,
filtdisp= 15.63 16.66 17.70 18.70 19.69 20.70 21.73 22.77
ntpq> quit
Verify Time Again
[Expert@fw:0]# /bin/date && hwclock && clish -c "show clock"
Fri Sep 2 18:39:40 EDT 2022
Fri Sep 2 18:39:40 2022 -0.410451 seconds
Fri Sep 2 18:39:40 2022 EDT