Heyo!
Any others who are being affected?
It looks like our customers Netgear routers (known ones: WNR3500Lv2, WNDR4500) are asking our DNS Server for the A record of: time-g.netgear.com or time- a.netgear.com
Instead of an A record reply, they get a CNAME as answer with additional information the A record of that CNAME. That is what netgear has published on their DNS Servers.
Those routers are not happy with that reply and just start sending several hundred requests per second for A time-g.netgear.com resulting in considerable load and traffic on our DNS caches. Some customers have already transfered 35GB of DNS traffic, only since today midnight.
I have contacted netgear technical support. The issue is yet unknown to them. They got my pcap files to analyze :-)
Any others observing that behaviour of netgear products? Any know remedies?
Mit freundlichen Grüssen
Benoit Panizzon
I have the same issue since some weeks.
The problem is that the customer does not understand the problem. So if Netgear has solved the problem in a new firmware the customer should update it, but does he knows how to do this???
What can you do to limit this stupid traffic: - rate limit the queries per customer (not really a good idea) - rate limit this special kind of queries. (that's the best way at the moment)
I haven't had the time to look into the packets to limit this queries. If they are all similiar you can set up a drop filter in the iptables like you should already have with the isc.org ANY requests. -> Problem not really solved but you should be happy with this :-)
the rule should be:
$IPTABLES -I INPUT -p udp --dport 53 -m string --from 47 --algo bm --hex-string '|0000FF0001|' -m recent --set --name dnsanyquery $IPTABLES -I INPUT -p udp --dport 53 -m string --from 47 --algo bm --hex-string '|0000FF0001|' -m recent --name dnsanyquery --rcheck --seconds 600 --hitcount 3 -j DROP
but what's the hex string for this kind of query. anybody got it?
Freundliche Grüsse
sasag Kabelkommunikation AG Michael Richter Professional Bachelor ODEC in Engineering mrichter@sasag.ch 052 633 01 71
________________________________________ Von: swinog-bounces@lists.swinog.ch [swinog-bounces@lists.swinog.ch]" im Auftrag von "Benoit Panizzon [benoit.panizzon@imp.ch] Gesendet: Freitag, 24. Mai 2013 12:03 An: swinog@swinog.ch Betreff: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
Heyo!
Any others who are being affected?
It looks like our customers Netgear routers (known ones: WNR3500Lv2, WNDR4500) are asking our DNS Server for the A record of: time-g.netgear.com or time- a.netgear.com
Instead of an A record reply, they get a CNAME as answer with additional information the A record of that CNAME. That is what netgear has published on their DNS Servers.
Those routers are not happy with that reply and just start sending several hundred requests per second for A time-g.netgear.com resulting in considerable load and traffic on our DNS caches. Some customers have already transfered 35GB of DNS traffic, only since today midnight.
I have contacted netgear technical support. The issue is yet unknown to them. They got my pcap files to analyze :-)
Any others observing that behaviour of netgear products? Any know remedies?
Mit freundlichen Grüssen
Benoit Panizzon -- I m p r o W a r e A G - ______________________________________________________
Zurlindenstrasse 29 Tel +41 61 826 93 07 CH-4133 Pratteln Fax +41 61 826 93 02 Schweiz Web http://www.imp.ch ______________________________________________________
_______________________________________________ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
On 2013-05-24 12:52 , Michael Richter wrote: [..]
What can you do to limit this stupid traffic: - rate limit the queries per customer (not really a good idea) - rate limit this special kind of queries. (that's the best way at the moment)
I haven't had the time to look into the packets to limit this queries. If they are all similiar you can set up a drop filter in the iptables like you should already have with the isc.org ANY requests. -> Problem not really solved but you should be happy with this :-)
[..]
but what's the hex string for this kind of query. anybody got it?
You want to deploy RRL.
iptables is not the right location for doing this kind of stuff as you will have false positives.
Please see http://www.redbarn.org/dns/ratelimits
Greets, Jeroen
You want to deploy RRL.
iptables is not the right location for doing this kind of stuff as you will have false positives.
Please see http://www.redbarn.org/dns/ratelimits
I agree that iptables might not be the perfect solution for that, however, as we have also been confronted with that problem some months ago with a lot of affected devices (each with >1000pps of those queries) we have limited those queries for some time as it is easy to deploy quickly. At that point of time time-g.netgear.com had no entry at all so the clients did not stop with the folding. Today it looks a little different, as there is at least a cname for that entry. We have used the u32 module for matching, we check name=time-g.netgear.com and type=A within the query. The matching line looks like:
iptables -A INPUT -p udp --dport 53 -m u32 --u32 "0x0>>0x16&0x3c@0x14=0x674696d&&0x0>>0x16&0x3c@0x18=0x652d6707&&0x0>>0x16&0x3c@0x1c=0x6e657467&&0x0>>0x16&0x3c@0x20=0x65617203&&0x0>>0x16&0x3c@0x24=0x636f6d00&&0x0>>0x16&0x3c@0x28&0xffff0000=0x10000" -j YOUR_CHAIN_OR_WHATEVER You can then use the limit module for example.
Just as a thought, maybe it would change something to send the clients a ntp server in the dhcp response, as it is obviously looking for an ntp server. Has someone maybe already tried that?
Cheers, Gregor
Thanks Gregor!! that was exactly what I was looking for.
have a nice weekend
---------------------------
You want to deploy RRL.
iptables is not the right location for doing this kind of stuff as you will have false positives.
Please see http://www.redbarn.org/dns/ratelimits
I agree that iptables might not be the perfect solution for that, however, as we have also been confronted with that problem some months ago with a lot of affected devices (each with >1000pps of those queries) we have limited those queries for some time as it is easy to deploy quickly. At that point of time time-g.netgear.com had no entry at all so the clients did not stop with the folding. Today it looks a little different, as there is at least a cname for that entry. We have used the u32 module for matching, we check name=time-g.netgear.com and type=A within the query. The matching line looks like:
iptables -A INPUT -p udp --dport 53 -m u32 --u32 "0x0>>0x16&0x3c@0x14=0x674696d&&0x0>>0x16&0x3c@0x18=0x652d6707&&0x0>>0x16&0x3c@0x1c=0x6e657467&&0x0>>0x16&0x3c@0x20=0x65617203&&0x0>>0x16&0x3c@0x24=0x636f6d00&&0x0>>0x16&0x3c@0x28&0xffff0000=0x10000" -j YOUR_CHAIN_OR_WHATEVER You can then use the limit module for example.
Just as a thought, maybe it would change something to send the clients a ntp server in the dhcp response, as it is obviously looking for an ntp server. Has someone maybe already tried that?
Cheers, Gregor
_______________________________________________ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Hi Jeroen
You want to deploy RRL.
Please see http://www.redbarn.org/dns/ratelimits
Excellent, thank you. Didn't know that bind feature.
Mit freundlichen Grüssen
Benoit Panizzon
Hey All
If it is really hurting you big time you may choose to run a very mean hack: temporarily setup a netgear.com-Zone on your dns-servers and point these records to a useful NTP server. Adding an A-record for their website would probably a good idea as well. ;)
Yes, it is an EXTREMELY UGLY HACK. But as stated above: it might be easier to cut yourself a hand off than loosing the whole arm...
but what's the hex string for this kind of query. anybody got it?
Had there somebody fun with Stefans presentation of yesterday...? ;)
Hey all
We had the same problem, at last a few weeks ago. We reported it to Netgear in Dec 12 for the first Time -> no result We tried to overwrite these records with another -> just for testing. The routers were still asking k-times a second.
I think it's not a DNS-problem, cause it doesn't matter what's the answer on a request is, the router is still asking. Only a reboot of device stops the 'attack'.
Best Regards
Beat
-----Ursprüngliche Nachricht----- Von: swinog-bounces@lists.swinog.ch [mailto:swinog-bounces@lists.swinog.ch] Im Auftrag von Roman Hochuli Gesendet: Freitag, 24. Mai 2013 14:33 An: swinog@swinog.ch Betreff: Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
Hey All
If it is really hurting you big time you may choose to run a very mean hack: temporarily setup a netgear.com-Zone on your dns-servers and point these records to a useful NTP server. Adding an A-record for their website would probably a good idea as well. ;)
Yes, it is an EXTREMELY UGLY HACK. But as stated above: it might be easier to cut yourself a hand off than loosing the whole arm...
but what's the hex string for this kind of query. anybody got it?
Had there somebody fun with Stefans presentation of yesterday...? ;)
-- Best regards, Roman Hochuli Operations Manager
nexellent ag Saegereistrasse 33 CH-8152 Glattbrugg
Phone: +41 44 872 20 00 Fax: +41 44 872 20 01 URL: www.nexellent.ch X-NCC-RegID: ch.nexellent
Imagination is the one weapon in the war against reality. -- Jules de Gaultier
_______________________________________________ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Hi folks
We have experienced this issue a lot with the WNDR4500 model in the last months. This is definitely a bug, which can be fixed with a newer FW release. Whenever we see constant traffic of approx. 10 Mbps and high CPU on our name servers, it's a WNDR4500.
I'm not completely sure, but it seems like the DNS query flooding is triggered by a temporary link down on the router's WAN port.
No advices for hacking from my side. Just hunt down the customer who's causing your problem :)
Cheers, Reto
-----Original Message----- From: swinog-bounces@lists.swinog.ch [mailto:swinog- bounces@lists.swinog.ch] On Behalf Of Beat Bodenmann Sent: Friday, May 24, 2013 2:59 PM To: swinog@swinog.ch Subject: Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
Hey all
We had the same problem, at last a few weeks ago. We reported it to Netgear in Dec 12 for the first Time -> no result We tried to overwrite these records with another -> just for testing. The routers were still asking k-times a second.
I think it's not a DNS-problem, cause it doesn't matter what's the answer on a request is, the router is still asking. Only a reboot of device stops the 'attack'.
Best Regards
Beat
-----Ursprüngliche Nachricht----- Von: swinog-bounces@lists.swinog.ch [mailto:swinog- bounces@lists.swinog.ch] Im Auftrag von Roman Hochuli Gesendet: Freitag, 24. Mai 2013 14:33 An: swinog@swinog.ch Betreff: Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
Hey All
If it is really hurting you big time you may choose to run a very mean hack: temporarily setup a netgear.com-Zone on your dns-servers and point these records to a useful NTP server. Adding an A-record for their website would probably a good idea as well. ;)
Yes, it is an EXTREMELY UGLY HACK. But as stated above: it might be easier to cut yourself a hand off than loosing the whole arm...
but what's the hex string for this kind of query. anybody got it?
Had there somebody fun with Stefans presentation of yesterday...? ;)
-- Best regards, Roman Hochuli Operations Manager
nexellent ag Saegereistrasse 33 CH-8152 Glattbrugg
Phone: +41 44 872 20 00 Fax: +41 44 872 20 01 URL: www.nexellent.ch X-NCC-RegID: ch.nexellent
Imagination is the one weapon in the war against reality. -- Jules de Gaultier
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
If it is really hurting you big time you may choose to run a very mean hack: temporarily setup a netgear.com-Zone on your dns-servers and point these records to a useful NTP server. Adding an A-record for their website would probably a good idea as well. ;)
Yes, it is an EXTREMELY UGLY HACK. But as stated above: it might be easier to cut yourself a hand off than loosing the whole arm...
Just set up netgear.com on our cache DNSes. Thankfully that is not a signed zone :-)
I see the client's request for time-g.netgear.com is now being replied with 157.161.1.4 (our NTP server), but those clients still are not happy and keep sending up to hundreds of request/s. So I fear also sending them an ntp-server dhcp option would not solve the issue.
netgear tech support has confirmed there is a problem in one of the models and released a fixed firmware. Great! How do I tell the customers?
Mit freundlichen Grüssen
Benoit Panizzon
netgear tech support has confirmed there is a problem in one of the
models and released a fixed firmware. Great! How do I tell the customers? Mit redirect the http traffic for those customer to an webpage which explain and offer a download link on that page maybe combine with a dns ratelimit for a while.
my 5 cents ...
Hello Benoit
Yes, it is an EXTREMELY UGLY HACK.
Just set up netgear.com on our cache DNSes. I see the client's request for time-g.netgear.com is now being replied with 157.161.1.4 (our NTP server), but those clients still are not happy and keep sending up to hundreds of request/s.
Well, bad luck with this hack then. :-(
Apparently this was already pointed out by Beat Bodenmann shortly after my mail.
On 2013-05-24 14:33 , Roman Hochuli wrote:
Hey All
If it is really hurting you big time you may choose to run a very mean hack: temporarily setup a netgear.com-Zone on your dns-servers and point these records to a useful NTP server. Adding an A-record for their website would probably a good idea as well. ;)
Yes, it is an EXTREMELY UGLY HACK. But as stated above: it might be easier to cut yourself a hand off than loosing the whole arm...
If you are doing that, which is a workable solution for you and your customers do it only for time-g.netgear.com by defining a zone for that and using '@' to get the record defined, that way you don't cause colateral damage to the many other records that might exist in netgear.com.
Of course the best avenue is to contact netgear and really hammer on them to fix it; though it seems to be a device issue, not a DNS problem.
but what's the hex string for this kind of query. anybody got it?
Had there somebody fun with Stefans presentation of yesterday...? ;)
Tranalyzer only analyzes as far as I recall and the slides do not indicate differently...
Greets, Jeroen
Hello Jeroen
If you are doing that, do it only for time-g.netgear.com by defining a zone for that and using '@' to get the record defined, that way you don't cause colateral damage to the many other records that might exist in netgear.com
Thanks for pointing out. Your solution is much a nicer than my approach. Looks like Scalpell vs. Hammer. :)
Tranalyzer only analyzes as far as I recall and the slides do not indicate differently...
You are right. I was more referring to his presentation-style at SwiNOG #26 which referred a lot to "what's that hex?" ;)
IMHO, this is also one of the things that unbound is superior to BIND.
You can simply configure "local-data" in the general configuration file in one line: local-data: "time-g.netgear.com 9600 IN A 209.249.181.22"
Ref: http://www.unbound.net/documentation/unbound.conf.html
Roque
On Sat, May 25, 2013 at 1:40 PM, Roman Hochuli roman.hochuli@nexellent.chwrote:
Hello Jeroen
If you are doing that, do it only for time-g.netgear.com by defining a zone for that and using '@' to get the record defined, that way you don't cause colateral damage to the many other records that might exist in netgear.com
Thanks for pointing out. Your solution is much a nicer than my approach. Looks like Scalpell vs. Hammer. :)
Tranalyzer only analyzes as far as I recall and the slides do not indicate differently...
You are right. I was more referring to his presentation-style at SwiNOG #26 which referred a lot to "what's that hex?" ;)
-- Best regards, Roman Hochuli Operations Manager
nexellent ag Saegereistrasse 33 CH-8152 Glattbrugg
Phone: +41 44 872 20 00 Fax: +41 44 872 20 01 URL: www.nexellent.ch X-NCC-RegID: ch.nexellent
Imagination is the one weapon in the war against reality. -- Jules de Gaultier
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Netgear has said that this is an known issue and is resolved with a software upgrade.
Best Akshay
On Sat, May 25, 2013 at 9:03 AM, Roque Gagliano rgaglian@gmail.com wrote:
IMHO, this is also one of the things that unbound is superior to BIND.
You can simply configure "local-data" in the general configuration file in one line: local-data: "time-g.netgear.com 9600 IN A 209.249.181.22"
Ref: http://www.unbound.net/documentation/unbound.conf.html
Roque
On Sat, May 25, 2013 at 1:40 PM, Roman Hochuli <roman.hochuli@nexellent.ch
wrote:
Hello Jeroen
If you are doing that, do it only for time-g.netgear.com by defining a zone for that and using '@' to get the record defined, that way you don't cause colateral damage to the many other records that might exist in netgear.com
Thanks for pointing out. Your solution is much a nicer than my approach. Looks like Scalpell vs. Hammer. :)
Tranalyzer only analyzes as far as I recall and the slides do not indicate differently...
You are right. I was more referring to his presentation-style at SwiNOG #26 which referred a lot to "what's that hex?" ;)
-- Best regards, Roman Hochuli Operations Manager
nexellent ag Saegereistrasse 33 CH-8152 Glattbrugg
Phone: +41 44 872 20 00 Fax: +41 44 872 20 01 URL: www.nexellent.ch X-NCC-RegID: ch.nexellent
Imagination is the one weapon in the war against reality. -- Jules de Gaultier
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
--
At least I did something Don Draper - Mad Men
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Hello Benoit
On 24.05.2013 12:03, Benoit Panizzon wrote:
It looks like our customers Netgear routers (known ones: WNR3500Lv2, WNDR4500) are asking our DNS Server for the A record of: time-g.netgear.com or time- a.netgear.com
For me this looks like entries for timeservers (NTP). This two destination share the same IP address (so it is not a very good fail safe solution ;) :
fabian@flashback:~ $ host time-g.netgear.com time-g.netgear.com is an alias for time-a.netgear.com. time-a.netgear.com has address 209.249.181.22 fabian@flashback:~ $ host time-a.netgear.com time-a.netgear.com has address 209.249.181.22 fabian@flashback:~ $
And the PTR also looks interesting (sorry for line wrapping):
fabian@flashback:~ $ host 209.249.181.22 22.181.249.209.in-addr.arpa is an alias for 22.0-127.181.249.209.in-addr.arpa. 22.0-127.181.249.209.in-addr.arpa domain name pointer time-a.on-networks.com. 22.0-127.181.249.209.in-addr.arpa domain name pointer time-a.netgear.com. fabian@flashback:~ $
This IP address does answer to ntp requests (sorry again for line wrapping):
fabian@flashback:~ $ ntpdate -q 209.249.181.22 server 209.249.181.22, stratum 1, offset 0.004557, delay 0.19078 24 May 12:41:50 ntpdate[55957]: adjust time server 209.249.181.22 offset 0.004557 sec fabian@flashback:~ $
Instead of an A record reply, they get a CNAME as answer with additional information the A record of that CNAME. That is what netgear has published on their DNS Servers.
It could be, that Netgear did change something in their DNS configuration (eg. moving time-g from A record to CNAME), which the used ntpd or sntp on this routers do not understand and so do re-request the DNS entry again because it could not sync the time.
Those routers are not happy with that reply and just start sending several hundred requests per second for A time-g.netgear.com resulting in considerable load and traffic on our DNS caches. Some customers have already transfered 35GB of DNS traffic, only since today midnight.
Are the high requests numbers only for time-g.netgear.com and not for time-a.netgear.com? If yes, this could prove the above idea of ntpd/sntp on this devices not properly working with a CNAME entry.
Do you have configuration access to such routers? If yes, check the entries for NTP and probably change some of them e.g. to ch.pool.ntp.org and/or 1.ch.pool.ntp.org.
I have contacted netgear technical support. The issue is yet unknown to them. They got my pcap files to analyze :-)
It could eventually be a good idea to also point them to this DNS entries, eventually the time-g server died and the sysadmin added the CNAME without knowing the impact this could have.
bye Fabian
Look here:
Same two device types I have found.
Mit freundlichen Grüssen
Benoit Panizzon
Benoit Panizzon <benoit.panizzon <at> imp.ch> writes:
Heyo!
Any others who are being affected?
It looks like our customers Netgear routers (known ones: WNR3500Lv2,
WNDR4500)
are asking our DNS Server for the A record of: time-g.netgear.com or
time-
a.netgear.com
Instead of an A record reply, they get a CNAME as answer with additional information the A record of that CNAME. That is what netgear has
published on
their DNS Servers.
Those routers are not happy with that reply and just start sending
several
hundred requests per second for A time-g.netgear.com resulting in
considerable
load and traffic on our DNS caches. Some customers have already
transfered
35GB of DNS traffic, only since today midnight.
I have contacted netgear technical support. The issue is yet unknown to
them.
They got my pcap files to analyze
Any others observing that behaviour of netgear products? Any know
remedies?
Mit freundlichen Grüssen
Benoit Panizzon
Netgear in the Netherlands/Belgium first denied a problem and needed more complains before escalating it L3 suppport. Seems later L3 support reported this issue should have been resolved from firmware version 1.0.1.6. This is for the WNDR4500 model. Somehow i don't think the problem has been resolved. Did anybody received a solution yet? Maybe beta firmware?
For the WNR3500Lv2 model is new firmware available. Short copy paste from the release notes (16 may 2013):
WNR3500Lv2 Firmware Version 1.2.0.18 •Fixed DNS Flood issue, the router would be triggered to send lots of DNS queries to the DNS server under some conditions.
-
A Patel -
Beat Bodenmann -
Benoit Panizzon -
Fabian Wenk -
Gregor Jeker -
Jeroen Massar -
Leis -
Michael Richter -
Reto Büeler -
Roger Schmid -
Roman Hochuli -
Roque Gagliano