Hi everyone,
We[1] are running some public NTP servers that are included in the NTP Pool[2]. In recent months we've had some issues with our servers being marked as unavailable by the monitoring host that tracks availability for the pool. The logs from the NTP pool website showed high packet loss.
However we could not see any issues on our side and pings from the looking glass closest to the monitoring host didn't show any packet loss at all.
After quite some debugging we realized that the packet loss only occurs for NTP packets via one of our transit providers, namely Liberty Global (AS6830). We queried their support, our theory being that they're trying to do some sort of DDoS protection for NTP reflection attacks. However they aren't aware of anything like this and also couldn't figure out why this is happening.
So I was wondering, has anyone else encountered this issue or something similar? We worked around the issues by routing the traffic around AS6830 but this still bothers me somehow.
Kind regards and see you soon at Swinog #35, Stefan
Hello Stefan
On 27.04.2019 11:36, Stefan Brand wrote:
After quite some debugging we realized that the packet loss only occurs for NTP packets via one of our transit providers, namely Liberty Global (AS6830).
I had do dig in my e-mail archive as I already ran into this way back in March 2017. I did discuss this with the Pool team (ticket #5636). I my case it is the IP address (62.2.85.186) on a UPC Business Cable Line ("Fiber Power"). The NTP Pool is aware, that when traffic goes from their monitoring system through NTT, that then occasionally (or more) packet loss does happen. From what we discovered back then, it seems that NTT may have some DDoS protection in place. From the Pool team I even got that information (unfortunately the PDF is not available any more): "After reading http://www.nttcomsecurity.com/uploads/files/US_2015_GTIR-ddos-observations_P... I won't be surprised if NTT has some automatic rate limiting on NTP traffic in place."
We queried their support, our theory being that they're trying to do some sort of DDoS protection for NTP reflection attacks. However they aren't aware of anything like this and also couldn't figure out why this is happening.
As the problem persisted I also did open a ticket with UPC in October 2017 (ticket B2BCH0000641230 and B2BCH0000643146). But they did not really understand the problem and did only update/downgrade the firmware on my modem (I knew that this will not help), but they did not see any problems. :-(
As this IP address already had bad scoring for a while, it got dropped out of the pool a few weeks later. In the past I had tried to add it back a few times, but it still fails even for the initial check the pool is doing. I have just tried a few times again without any luck.
But luckily I still have some other servers in the Pool: https://www.pool.ntp.org/user/home4u.ch
So I was wondering, has anyone else encountered this issue or something similar? We worked around the issues by routing the traffic around AS6830 but this still bothers me somehow.
Back then the problem was worse, as the routing was asymmetrical, and the path was only from the ntp pool monitoring through NTT, but not back from UPC. As I see it now, the routing is symmetrical through NTT.
The pool as a tracroute tool available at (change the IP to your own): https://trace.ntppool.org/traceroute/62.2.85.186
As the pool provides some nice things with e.g. the data of the last 50 checks (change to your IP, mine does not have any data): https://www.pool.ntp.org/scores/62.2.85.186/log?limit=50 You see that it does a check around every 20 minutes (the timestamp is in seconds since epoch). I did run a tcpdump at my end, when a packet arrived, then also the pool monitoring system got my answer back and data got recorded. So it was clear to me, that packets got dropped on the way from the Pool monitoring to me.
I did monitor on my end with: tcpdump -tt -i em1 host 207.171.3.17 and port 123 But it may be that the IP has changed now, so you have to ask the Pool Team for current information. If you already got e-mails from Ask with subject "NTP Pool: Problems with your NTP server ...", just write back to it. :)
Eventually it is also helpful to subscribe to the Pool mailing list (or search in their archive): http://lists.ntp.org/listinfo/pool
Hope this information helps and you are in a better position to tell UPC that their upstream NTT is doing something not very nice.
Best regards, Fabian
Hello
On 27.04.19 19:25, Fabian Wenk wrote:
As the problem persisted I also did open a ticket with UPC in October 2017 (ticket B2BCH0000641230 and B2BCH0000643146). But they did not
I just realized, that the ticket B2BCH0000641230 does not have anything to do with the packet loss from the NTP Pool monitoring. In my case it is just related to NTP when a certain type of cable modem is in use. That modem does not properly work with huge amounts of multiple UDP sessions and then totally loses the whole IP connectivity.
Best regards, Fabian
-
Fabian Wenk -
Stefan Brand