Hello Benoit
On 24.05.2013 12:03, Benoit Panizzon wrote:
It looks like our customers Netgear routers (known ones: WNR3500Lv2, WNDR4500) are asking our DNS Server for the A record of: time-g.netgear.com or time- a.netgear.com
For me this looks like entries for timeservers (NTP). This two destination share the same IP address (so it is not a very good fail safe solution ;) :
fabian@flashback:~ $ host time-g.netgear.com time-g.netgear.com is an alias for time-a.netgear.com. time-a.netgear.com has address 209.249.181.22 fabian@flashback:~ $ host time-a.netgear.com time-a.netgear.com has address 209.249.181.22 fabian@flashback:~ $
And the PTR also looks interesting (sorry for line wrapping):
fabian@flashback:~ $ host 209.249.181.22 22.181.249.209.in-addr.arpa is an alias for 22.0-127.181.249.209.in-addr.arpa. 22.0-127.181.249.209.in-addr.arpa domain name pointer time-a.on-networks.com. 22.0-127.181.249.209.in-addr.arpa domain name pointer time-a.netgear.com. fabian@flashback:~ $
This IP address does answer to ntp requests (sorry again for line wrapping):
fabian@flashback:~ $ ntpdate -q 209.249.181.22 server 209.249.181.22, stratum 1, offset 0.004557, delay 0.19078 24 May 12:41:50 ntpdate[55957]: adjust time server 209.249.181.22 offset 0.004557 sec fabian@flashback:~ $
Instead of an A record reply, they get a CNAME as answer with additional information the A record of that CNAME. That is what netgear has published on their DNS Servers.
It could be, that Netgear did change something in their DNS configuration (eg. moving time-g from A record to CNAME), which the used ntpd or sntp on this routers do not understand and so do re-request the DNS entry again because it could not sync the time.
Those routers are not happy with that reply and just start sending several hundred requests per second for A time-g.netgear.com resulting in considerable load and traffic on our DNS caches. Some customers have already transfered 35GB of DNS traffic, only since today midnight.
Are the high requests numbers only for time-g.netgear.com and not for time-a.netgear.com? If yes, this could prove the above idea of ntpd/sntp on this devices not properly working with a CNAME entry.
Do you have configuration access to such routers? If yes, check the entries for NTP and probably change some of them e.g. to ch.pool.ntp.org and/or 1.ch.pool.ntp.org.
I have contacted netgear technical support. The issue is yet unknown to them. They got my pcap files to analyze :-)
It could eventually be a good idea to also point them to this DNS entries, eventually the time-g server died and the sysadmin added the CNAME without knowing the impact this could have.
bye Fabian