Since we see >1Tbps DDOS attacs in the wild, I suppose out-of-the-box DDOS mitigation suppliers have lost this race. There is no operator in Switzerland which can handle 1Tbps DDOS attacks.
When we saw DDOS against digitec.ch and others earlier this year, I was a bit surprised that none of the so called "experts" proposed regional BGP propagation as a remedy.
Given that e-commerce such as digitec.ch is assumingly making 99.9% of the revenue within Switzerland, their prefix doesn't need to reachable from all over the world. If the prefix of a Swiss e-commerce would be reachable from Swiss broadband providers only, the DDOS is mitigated, as the vast majority of the botnet is lacking a route to the targeted victim IP address.
To achieve this I think we need a collaborative community effort setting up a common procedure and define a BGP communitiy with the effect "do not announce beyond Switzerland".
An e-commerce should be able to hit the button injecting this defined BGP community when under attack (or permanently, of course).
I suppose to make this idea a success we need to have all major operators in Switzerland on board (3303, 6730, 6830) and I suppose the smaller operators will follow in their own interest to avoid blackholes.
Anyone? I think it's good if a somewhat "neutral body" with decent BGP knowledge could take the lead for such a working group, maybe SWITCH or SwissIX?
-- Fredy Kuenzler
--------------------- Fiber7. No Limits. https://www.fiber7.ch ---------------------
Init7 (Switzerland) Ltd. AS13030 St.-Georgen-Strasse 70 CH-8400 Winterthur Skype: flyingpotato Phone: +41 44 315 4400 Fax: +41 44 315 4401 Twitter: @init7 / @kuenzler http://www.init7.net/
Hi,
On Sat, Oct 01, 2016 at 04:51:36PM +0200, Fredy Kuenzler wrote:
To achieve this I think we need a collaborative community effort setting up a common procedure and define a BGP communitiy with the effect "do not announce beyond Switzerland".
I think this is an awesome idea.
The situation is similar here in DE - nobody could stand an 1 Tbit DDoS attack, and a large number of content offerings are targeted only to german speaking customers, so if DE/A/CH work, 99% of the customers are still able to reach the site.
I'm not really sure how this would work in your example - what if you have two customers in a given BGP announcement, one of them *does* want to be reached world-wide (like, corporate VPNs) and the other one is attacked? Split the aggregate, or bit the bullet and have all of them with limited reach, for the time being?
(We currently work this "the other way round" by using the "out of country" and "out of continent" blackhole communities offered by NTT - so the customer under attack would be announced as a "faraway RTBH" route - but this isn't good enough yet either, as not all transits offer this...)
Gert Doering -- NetMaster
On 01.10.2016 17:35, Gert Doering wrote:
I think this is an awesome idea.
The situation is similar here in DE - nobody could stand an 1 Tbit DDoS attack, and a large number of content offerings are targeted only to german speaking customers, so if DE/A/CH work, 99% of the customers are still able to reach the site.
Maybe we should widen the approach and define a collaborative BGP community "do announce only in country X", when X is some ISO-3166 country number? A prefix then can contain multiple communities, i.E. to cover the whole DACH region.
https://de.wikipedia.org/wiki/ISO-3166-1-Kodierliste
I'm not really sure how this would work in your example - what if you have two customers in a given BGP announcement, one of them *does* want to be reached world-wide (like, corporate VPNs) and the other one is attacked? Split the aggregate, or bit the bullet and have all of them with limited reach, for the time being?
I suppose the e-commerces using such a mechanism would be able to afford their own /24 and a decent block of IPv6 space (in other words: buy legacy PI or become LIR). Another option is new business for managed hosting "DDOS bullet proof Switzerland Hosting", where the hoster dedicates a /24 or bigger for permanent limited propagation.
On 2016-10-01 16:51, Fredy Kuenzler wrote: [..]
To achieve this I think we need a collaborative community effort setting up a common procedure and define a BGP communitiy with the effect "do not announce beyond Switzerland".
Great initiative! If you need extra hands, don't hesitate to yell...
Did you btw see: http://www.trustednetworksinitiative.nl/ https://www.nl-ix.net/solutions/security-solutions/trusted-routing https://ams-ix.net/technical/trusted-networks-initiative
We should have a Swiss equivalent: - trusted and direct contacts - require BCP38 where possible - proper statistics/monitoring - proper & standardized "You are DDoS'ing" notifications providing Flow info as "proof". - proper & standardized "We put customer in walled garden"
The problem with the latter: VoIP... thus the walled garden needs to not block that due to "emergency services". Thus a throttle and a call to the customer might be needed to inform them...
As for the BGP thing... I thought folks had a deal like that per default for all their prefixes :)
It is after all the reason why quite a few IRC servers live(d) in PI /24....: - always the prefix to local peers - when 'normal' also announce to transit providers
When DDoS comes: - stop announcing to transits - check monitoring/stats tools which local peers are sending crap traffic and kick them hard
Now, the more important part is actually that: - You have good relationship with your transit - You have amazing relationship with your local peers: so that you can call them and notify them of the problem - Have proper instrumentation
Of course, when you have that, you might also want to peek at: - RPF / BCP38 kinda stuff and 'force' or 'require' that from your peers thus avoiding any spoofed traffic from them.
Not that BCP38 actually solves anything for these DDoS's as there are just thousands of botted devices involved...
Proper flows everywhere, proper notification and shutdowns at the source are the only way to go there.
And that will involve people calling helpdesks because: - their botted host is sending too much traffic making "The Internet Slow" and them complaining - they are disconnected, as you caught them participating.
Which might not fly with management in many places as helpdesk == money.
Hence, maybe to cover that at least, having a admin.ch rule, BAKOM maybe, that allows an ISP to "restrict access", eg wall-garden an endpoint that is causing DDOS attack would be a good thing.
Though, does not have to go that high actually, having a general consensus between ISPs that this is the case and putting it in the end-user agreement could be good enough to cover their ass a bit.
Greets, Jeroen
Agree, this has to be done.
See also https://fe.nix.cz/en/ in Czech republic some ISP realized this idea about 3 year ago.
And see also some IP-Transit-Providers already have regional restricted route propagation in their BGP community. https://www.gtt.net/services/internet-services/ip-transit/bgp-communities/
So in case of DDOS it need only to add this community to the propagated network instead of black holing one address.
Best regards
Milan
TRENKA INFORMATIK AG _______________________ Seefeldstrasse 108 8008 Zürich Tel: +41 44 383 63 07 mailto:mt@trenka.ch
-----Ursprüngliche Nachricht----- Von: swinog-bounces@lists.swinog.ch [mailto:swinog-bounces@lists.swinog.ch] Im Auftrag von Jeroen Massar Gesendet: Samstag, 1. Oktober 2016 18:04 An: Fredy Kuenzler kuenzler@init7.net; swinog@swinog.ch Betreff: Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!
On 2016-10-01 16:51, Fredy Kuenzler wrote: [..]
To achieve this I think we need a collaborative community effort setting up a common procedure and define a BGP communitiy with the effect "do not announce beyond Switzerland".
Great initiative! If you need extra hands, don't hesitate to yell...
Did you btw see: http://www.trustednetworksinitiative.nl/ https://www.nl-ix.net/solutions/security-solutions/trusted-routing https://ams-ix.net/technical/trusted-networks-initiative
We should have a Swiss equivalent: - trusted and direct contacts - require BCP38 where possible - proper statistics/monitoring - proper & standardized "You are DDoS'ing" notifications providing Flow info as "proof". - proper & standardized "We put customer in walled garden"
The problem with the latter: VoIP... thus the walled garden needs to not block that due to "emergency services". Thus a throttle and a call to the customer might be needed to inform them...
As for the BGP thing... I thought folks had a deal like that per default for all their prefixes :)
It is after all the reason why quite a few IRC servers live(d) in PI /24....: - always the prefix to local peers - when 'normal' also announce to transit providers
When DDoS comes: - stop announcing to transits - check monitoring/stats tools which local peers are sending crap traffic and kick them hard
Now, the more important part is actually that: - You have good relationship with your transit - You have amazing relationship with your local peers: so that you can call them and notify them of the problem - Have proper instrumentation
Of course, when you have that, you might also want to peek at: - RPF / BCP38 kinda stuff and 'force' or 'require' that from your peers thus avoiding any spoofed traffic from them.
Not that BCP38 actually solves anything for these DDoS's as there are just thousands of botted devices involved...
Proper flows everywhere, proper notification and shutdowns at the source are the only way to go there.
And that will involve people calling helpdesks because: - their botted host is sending too much traffic making "The Internet Slow" and them complaining - they are disconnected, as you caught them participating.
Which might not fly with management in many places as helpdesk == money.
Hence, maybe to cover that at least, having a admin.ch rule, BAKOM maybe, that allows an ISP to "restrict access", eg wall-garden an endpoint that is causing DDOS attack would be a good thing.
Though, does not have to go that high actually, having a general consensus between ISPs that this is the case and putting it in the end-user agreement could be good enough to cover their ass a bit.
Greets, Jeroen
_______________________________________________ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Speaking for PCH, which operates the largest DNS CDN, what you’re discussing is similar to our practice, in effect. We have server clusters in 145 IXPs, including Switzerland. 90% of those advertise services only through peering, so, only to our peers at each specific exchange and their customers, but not to global transit. 10% of our locations also advertise services through global transit. In our experience, although the vast majority of our legitimate traffic is handled through peering, DDoS attacks rarely have any significant effect on the peering-only locations, while they have disproprortionately large effect on the sites with global transit.
As well, the inter-provider coordination and assistance, which we already do quite a bit of through NSP-Sec and INOC-DBA, is invaluable in mitigating the effects of DDoS attacks.
So, what’s being proposed seems eminently sensible to me, and PCH would happily participate, whether within a Swiss scope, or globally.
-Bill
On 2016-10-01 16:51, Fredy Kuenzler wrote: [..]
To achieve this I think we need a collaborative community effort setting up a common procedure and define a BGP communitiy with the effect "do not announce beyond Switzerland".
Great initiative! If you need extra hands, don't hesitate to yell...
Did you btw see: http://www.trustednetworksinitiative.nl/ https://www.nl-ix.net/solutions/security-solutions/trusted-routing https://ams-ix.net/technical/trusted-networks-initiative
We should have a Swiss equivalent:
- trusted and direct contacts
- require BCP38 where possible
- proper statistics/monitoring
- proper & standardized "You are DDoS'ing" notifications providing Flow info as "proof".
- proper & standardized "We put customer in walled garden"
If I recall correctly 3303 has some communities which can be used for such a matter...
gruss
-steven
Am 01.10.2016 um 16:51 schrieb Fredy Kuenzler kuenzler@init7.net:
Since we see >1Tbps DDOS attacs in the wild, I suppose out-of-the-box DDOS mitigation suppliers have lost this race. There is no operator in Switzerland which can handle 1Tbps DDOS attacks.
When we saw DDOS against digitec.ch and others earlier this year, I was a bit surprised that none of the so called "experts" proposed regional BGP propagation as a remedy.
Given that e-commerce such as digitec.ch is assumingly making 99.9% of the revenue within Switzerland, their prefix doesn't need to reachable from all over the world. If the prefix of a Swiss e-commerce would be reachable from Swiss broadband providers only, the DDOS is mitigated, as the vast majority of the botnet is lacking a route to the targeted victim IP address.
To achieve this I think we need a collaborative community effort setting up a common procedure and define a BGP communitiy with the effect "do not announce beyond Switzerland".
An e-commerce should be able to hit the button injecting this defined BGP community when under attack (or permanently, of course).
I suppose to make this idea a success we need to have all major operators in Switzerland on board (3303, 6730, 6830) and I suppose the smaller operators will follow in their own interest to avoid blackholes.
Anyone? I think it's good if a somewhat "neutral body" with decent BGP knowledge could take the lead for such a working group, maybe SWITCH or SwissIX?
-- Fredy Kuenzler
Fiber7. No Limits. https://www.fiber7.ch
Init7 (Switzerland) Ltd. AS13030 St.-Georgen-Strasse 70 CH-8400 Winterthur Skype: flyingpotato Phone: +41 44 315 4400 Fax: +41 44 315 4401 Twitter: @init7 / @kuenzler http://www.init7.net/
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Hi
I'm a employee of a good known E-Commerce site here in switzerland and I would like to share some thoughts from my side if that's okay for all. I hope I understood well enough what you plan. Otherwise just ignore what I just wrote :)
Given that e-commerce such as digitec.ch is assumingly making 99.9% of the revenue within Switzerland, their prefix doesn't need to reachable from all over the world.
That's correct, the *customer* doesn't need to the reach the website from outsite switzerland normaly. But there're many 3rd-Party Provider for Newsletter, Monitoring etc. and distributors that need to be able to resolve digitec.ch outside of switzerland for example. (because there server are not located in switzerland) Mostly it's dispensable if they can't reach the website or a ftp server for some minutes, but if they can't access the page for days the E-Commerce Site will have issue with orders, product availability, newsletter shipping etc. Also some 3rd- Party Scripts may use a dns lookup and would fail then. There's also a possibilty that the employee reach the internet via a proxy outside of switzerland (due to a enterprise policy) so they wouldn't be able to access there site and couldn't work at all.
Of course if the site isn't available at all it's not a good experience for the customer and they may order there article on a other onlineshop, but if the website is online and doesn't work properly that's also not a optimal solution either.
Addiontally to the fact that more and more E-Commerce Websites use DDoS- Protection services like akamai or cloudflare, only about half hosting there website on server in switzerland.
-- Patrick Albrecht Powered by FastMail
On 2016-10-01 20:24, Patrick Albrecht wrote:
Hi
I'm a employee of a good known E-Commerce site here in switzerland and I would like to share some thoughts from my side if that's okay for all. I hope I understood well enough what you plan. Otherwise just ignore what I just wrote :)
Given that e-commerce such as digitec.ch is assumingly making 99.9% of the revenue within Switzerland, their prefix doesn't need to reachable from all over the world.
That's correct, the /customer/ doesn't need to the reach the website from outsite switzerland normaly. But there're many 3rd-Party Provider for Newsletter, Monitoring etc. and distributors that need to be able to resolve digitec.ch outside of switzerland for example.
"resolve" implies DNS.
Peering is about BGP.
(because there server are not located in switzerland) Mostly it's dispensable if they can't reach the website or a ftp server for some minutes, but if they can't access the page for days the E-Commerce Site will have issue with orders, product availability, newsletter shipping etc. Also some 3rd-Party Scripts may use a dns lookup and would fail then.
You need to see that 'limited announce of prefix' would only happen in the case of a DDoS, this, so that local sites / direct peers can reach it, but it is 'dead' over transit, thus cutting off most DDoS traffic that comes from strange countries (not .ch).
As for those external companies, if you are worried about them failing: peer directly with them, setup a VPN or: move your stuff more local where you have control.
Also, do realize that providing Swiss customer data to a foreign entity might break various privacy regulations.... do ask your legal team and of course inform your customers.
There's also a possibilty that the employee reach the internet via a proxy outside of switzerland (due to a enterprise policy) so they wouldn't be able to access there site and couldn't work at all.
That is a weird "Enterprise policy". Doing business that way opens you up to all kind of fun international laws concerning your business.
Also note that you can of course always announce to trusted peers that are not in Switzerland...
The major point is "trusted peers". Ones that will clean up their attacking hosts the moment you notify them.
Of course if the site isn't available at all it's not a good experience for the customer and they may order there article on a other onlineshop, but if the website is online and doesn't work properly that's also not a optimal solution either.
Better test it out today what happens when your site gets DDoSsed to bits, as the script kiddies have access to the same botnet know that Krebs got sent after him... (see other mail).
Addiontally to the fact that more and more E-Commerce Websites use DDoS-Protection services like akamai or cloudflare, only about half hosting there website on server in switzerland.
You might want to reconsider your hosting location ;)
Also, if you are paying those kind of companies: prepare to dig deep in your pockets for DDoS protection... we are going to have a fun X-mas this year...
Greets, Jeroen
Hi Fredy, Everyone,
I realised my previous reply was sent encrypted, sorry about the noise. Here is the clear content :
Taking back on a wider point of view again, I think temporary and localised/more specific BGP announcements isn't a so bad idea, but I'm the awful example and I can reach most the contents in .ch thru the peering location around my network.
That's also valid for the content I host, or almost because I'm not yet at the point where I can tweak my announcements to all the bigger players in .ch, for instance to that very cooperative cable operator, because they still won't peer with these tiny networks. So I'm already out of that pack of users.
And further more, I also heard that the transit is becoming cheaper than peering, with some LIR/ISP getting it from, let's say HE and Cogent, even if they are based in .ch. I see this becoming a blocking point, if we don't remind the local LIR/ISP that for user experience, we should try to keep traffic local, if possible.
Maybe we should ask the big hosting location in .ch to get free x-connects for the peering ports ?
Will
On 01 Oct 2016, at 22:15, Jeroen Massar jeroen@massar.ch wrote:
On 2016-10-01 20:24, Patrick Albrecht wrote:
Hi
I'm a employee of a good known E-Commerce site here in switzerland and I would like to share some thoughts from my side if that's okay for all. I hope I understood well enough what you plan. Otherwise just ignore what I just wrote :)
Given that e-commerce such as digitec.ch is assumingly making 99.9% of the revenue within Switzerland, their prefix doesn't need to reachable from all over the world.
That's correct, the /customer/ doesn't need to the reach the website from outsite switzerland normaly. But there're many 3rd-Party Provider for Newsletter, Monitoring etc. and distributors that need to be able to resolve digitec.ch outside of switzerland for example.
"resolve" implies DNS.
Peering is about BGP.
(because there server are not located in switzerland) Mostly it's dispensable if they can't reach the website or a ftp server for some minutes, but if they can't access the page for days the E-Commerce Site will have issue with orders, product availability, newsletter shipping etc. Also some 3rd-Party Scripts may use a dns lookup and would fail then.
You need to see that 'limited announce of prefix' would only happen in the case of a DDoS, this, so that local sites / direct peers can reach it, but it is 'dead' over transit, thus cutting off most DDoS traffic that comes from strange countries (not .ch).
As for those external companies, if you are worried about them failing: peer directly with them, setup a VPN or: move your stuff more local where you have control.
Also, do realize that providing Swiss customer data to a foreign entity might break various privacy regulations.... do ask your legal team and of course inform your customers.
There's also a possibilty that the employee reach the internet via a proxy outside of switzerland (due to a enterprise policy) so they wouldn't be able to access there site and couldn't work at all.
That is a weird "Enterprise policy". Doing business that way opens you up to all kind of fun international laws concerning your business.
Also note that you can of course always announce to trusted peers that are not in Switzerland...
The major point is "trusted peers". Ones that will clean up their attacking hosts the moment you notify them.
Of course if the site isn't available at all it's not a good experience for the customer and they may order there article on a other onlineshop, but if the website is online and doesn't work properly that's also not a optimal solution either.
Better test it out today what happens when your site gets DDoSsed to bits, as the script kiddies have access to the same botnet know that Krebs got sent after him... (see other mail).
Addiontally to the fact that more and more E-Commerce Websites use DDoS-Protection services like akamai or cloudflare, only about half hosting there website on server in switzerland.
You might want to reconsider your hosting location ;)
Also, if you are paying those kind of companies: prepare to dig deep in your pockets for DDoS protection... we are going to have a fun X-mas this year...
Greets, Jeroen
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Dear team,
Since we see >1Tbps DDOS attacs in the wild, I suppose out-of-the-box DDOS mitigation suppliers have lost this race. There is no operator in Switzerland which can handle 1Tbps DDOS attacks.
When we saw DDOS against digitec.ch and others earlier this year, I was a bit surprised that none of the so called "experts" proposed regional BGP propagation as a remedy.
May I offer up UTRS as a model or perhaps part of your solution?
https://www.team-cymru.org/UTRS/
UTRS is a system that helps mitigate large infrastructure attacks by leveraging an existing network of cooperating BGP speakers such as ISPs, hosting providers and educational institutions that automatically distributes verified BGP-based filter rules from victim to cooperating networks.
Victims can now effectively alleviate attacks quickly and across the world at lightning speed. Additionally, by using UTRS, operators will also be stopping the attack traffic at the source, saving many would-be attack packets from their own network, as well as preventing them from taking up unnecessary network resources at every other network in between.
Be well, Rob. - -- Rabbi Rob Thomas Team Cymru "It is easy to believe in freedom of speech for those with whom we agree." - Leo McKern
On 2016-10-02 05:27, Rabbi Rob Thomas wrote:
Dear team,
Since we see >1Tbps DDOS attacs in the wild, I suppose out-of-the-box DDOS mitigation suppliers have lost this race. There is no operator in Switzerland which can handle 1Tbps DDOS attacks.
When we saw DDOS against digitec.ch and others earlier this year, I was a bit surprised that none of the so called "experts" proposed regional BGP propagation as a remedy.
May I offer up UTRS as a model or perhaps part of your solution?
Good one.
Lets see if the SwiNOG #30 PC gives a slot for this discussion, then we'll include it in the slidedeck and conversation.
Unless somebody from TC is hopping over, I'll use the details on: https://www.cymru.com/jtk/misc/utrs.html as input. Pointing out that TC is of course the one who runs UTRS :)
Greets, Jeroen
On 02 Oct 2016, at 09:53, Jeroen Massar jeroen@massar.ch wrote:
On 2016-10-02 05:27, Rabbi Rob Thomas wrote:
Dear team,
Since we see >1Tbps DDOS attacs in the wild, I suppose out-of-the-box DDOS mitigation suppliers have lost this race. There is no operator in Switzerland which can handle 1Tbps DDOS attacks.
When we saw DDOS against digitec.ch and others earlier this year, I was a bit surprised that none of the so called "experts" proposed regional BGP propagation as a remedy.
May I offer up UTRS as a model or perhaps part of your solution?
Good one.
Lets see if the SwiNOG #30 PC gives a slot for this discussion, then we'll include it in the slidedeck and conversation.
Unless somebody from TC is hopping over, I'll use the details on: https://www.cymru.com/jtk/misc/utrs.html as input. Pointing out that TC is of course the one who runs UTRS :)
I actually already submitted a talk about that topic (not exactly on 1Tbps DDoS), and UTRS is already there. But with this thread I see me extending my content with our new information ;)
Greets, Jeroen
Cheers,
Will
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Dear team,
May I offer up UTRS as a model or perhaps part of your solution?
Good one.
Lets see if the SwiNOG #30 PC gives a slot for this discussion, then we'll include it in the slidedeck and conversation.
Unless somebody from TC is hopping over, I'll use the details on: https://www.cymru.com/jtk/misc/utrs.html as input. Pointing out that TC is of course the one who runs UTRS :)
I actually already submitted a talk about that topic (not exactly on 1Tbps DDoS), and UTRS is already there. But with this thread I see me extending my content with our new information ;)
We're happy to help with the presentation, so please don't hesitate to ask. I don't believe we have someone slotted to attend this Swinog.
Be well! Rob. - -- Rabbi Rob Thomas Team Cymru "It is easy to believe in freedom of speech for those with whom we agree." - Leo McKern
-
Bill Woodcock -
Fredy Kuenzler -
Gert Doering -
Jeroen Massar -
Milan Trenka -
Patrick Albrecht -
Rabbi Rob Thomas -
Steven Glogger -
Will van Gulik