Friday night we observed several Brocade MLXe linecards rebooting (several locations, i.e. Amsterdam, Frankfurt, Geneva), which caused network instability due to flapping iBGP etc.
As of now we know that iWay and nine.ch suffered from similar issues in the same time window, and we believe that they use also Brocade MLXe.
Tonight around 4 a.m. symptoms occurred again, on a lesser degree. nine.ch reports it too.
For reference the tickets: http://www.init7.net/de/status/?ticket=10348 https://status.nine.ch/en/messages/544 https://www.iway.ch/iway/status/
Did anyone else observed similar symptoms?
Since these routers operate in various locations and ASNs but are the same make/model I suppose it's a software security issue. Can malicious packets force a linecard reboot?
Regards,
-- Fredy Kuenzler Init7 (Switzerland) Ltd. St.-Georgen-Strasse 70 CH-8400 Winterthur Switzerland
Please contact Brocade TAC which is able to provide a patch It is NOT been seen with customers using releases below 5.9.00
Regards, Chris
-----Original Message----- From: swinog-bounces@lists.swinog.ch [mailto:swinog-bounces@lists.swinog.ch] On Behalf Of Fredy Kuenzler Sent: Sunday, September 18, 2016 08:11 To: swinog@swinog.ch Subject: [swinog] Brocade MLXe: Random Linecard Reboot - anyone else?
Friday night we observed several Brocade MLXe linecards rebooting (several locations, i.e. Amsterdam, Frankfurt, Geneva), which caused network instability due to flapping iBGP etc.
As of now we know that iWay and nine.ch suffered from similar issues in the same time window, and we believe that they use also Brocade MLXe.
Tonight around 4 a.m. symptoms occurred again, on a lesser degree. nine.ch reports it too.
For reference the tickets: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.init7.net_de_status_... https://urldefense.proofpoint.com/v2/url?u=https-3A__status.nine.ch_en_messa... https://urldefense.proofpoint.com/v2/url?u=https-3A__www.iway.ch_iway_status...
Did anyone else observed similar symptoms?
Since these routers operate in various locations and ASNs but are the same make/model I suppose it's a software security issue. Can malicious packets force a linecard reboot?
Regards,
-- Fredy Kuenzler Init7 (Switzerland) Ltd. St.-Georgen-Strasse 70 CH-8400 Winterthur Switzerland
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.init7.net_&d=DQI...
_______________________________________________ swinog mailing list swinog@lists.swinog.ch https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.swinog.ch_cgi-2Dbi...
Hello Fredy,
We were suffering the same issues during the same timeframes. From what I heard from TAC your guess that malicious packets are forcing the linecards to reload seems to be correct, not confirmed tough. I am eager to get further details.
Still waiting for information regarding the patch...
Cheers, Julian
Am 18.09.2016 um 08:11 schrieb Fredy Kuenzler kuenzler@init7.net:
Friday night we observed several Brocade MLXe linecards rebooting (several locations, i.e. Amsterdam, Frankfurt, Geneva), which caused network instability due to flapping iBGP etc.
As of now we know that iWay and nine.ch suffered from similar issues in the same time window, and we believe that they use also Brocade MLXe.
Tonight around 4 a.m. symptoms occurred again, on a lesser degree. nine.ch reports it too.
For reference the tickets: http://www.init7.net/de/status/?ticket=10348 https://status.nine.ch/en/messages/544 https://www.iway.ch/iway/status/
Did anyone else observed similar symptoms?
Since these routers operate in various locations and ASNs but are the same make/model I suppose it's a software security issue. Can malicious packets force a linecard reboot?
Coincidence?
https://twitter.com/schneierblog/status/775783898366160896
;-)
I have been suspecting this as well, I plan to actually get this firmware, get a spare MLX from someone and put it online, pcap the hell out of it and see what happens until it reboots.
Either this is a very strange bug or something bigger.
----- Ursprüngliche Mail ----- Von: "Rainer Duffner" rainer@ultra-secure.de [...]
Since these routers operate in various locations and ASNs but are the same make/model I suppose it's a software security issue. Can malicious packets force a linecard reboot?
Coincidence?
https://twitter.com/schneierblog/status/775783898366160896
;-)
_______________________________________________ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
-
Christoph Kaelin -
Fredy Kuenzler -
Julian Rutz -
Rainer Duffner -
Silvan M. Gebhardt