On 20210225, at 16:52, Jean-Pierre Schwickerath swinog@hilotec.net wrote:
Hi Jeroen
that "sinkhole" is just a misconfigured/internet-ignorant "load balancer": those things do not care about ICMP...
you are thus reaching the dest, it is just misconfigured: the Internet is just HTTP for many, they do not care about this TCP, ICMP or IP thing... be happy there is some kind of IPv6...
I wouldn't have noticed the issue if the loadbalancer / webserver had actually returned a webpage on port TCP/443. But it doesn't. So I tried from a different network to see if the issue is reproducible and that when I noticed the path taken by the traceroute packets.
Check with a tcpdump, don't forget to include ICMP.
Could also be an MTU issue or something on your side killing it.
Of course the behavior of the "load balancer" says quite a few things... sbb.ch is like that too...
Btw, when complaining about something, it is wise to include IP addresses, especially for the source...
The first hop of the traceroute is actually a good indicator for the source.
192.168.205.240 ?
DNS is ambiguous and reverses do not always match forwards. Including the actual IPs can thus be very useful...
That is, if you actually want it resolved.
You might want contact swisscom directly (good luck with that) or at least your ISPs that provide the connectivity, they might have better chance at contacting them. (taking transit over swissix is a fun one; but yea it is not that swisscom likes to peer, what else would a monopoly do)
Greets, Jeroen
Hi Jeroen
There were two issues, one has been resolved, which was the routing issue I was seeing from one of the networks where I tested from.
What remains is that I can't open a successful TCP connection to www.swisscom.ch on Port 443 when coming from a non-swisscom network.
Check with a tcpdump, don't forget to include ICMP.
Could also be an MTU issue or something on your side killing it.
Of course the behavior of the "load balancer" says quite a few things... sbb.ch is like that too...
I tried different ISPs to which I have access and I can't establish a IPv6 connection to port TCP/443 to neither 2a02:a90:c400:4001::2 nor 2a02:a90:c400:5001::2 which seems the two loadbalancers responsible for www.swisscom.ch except from a Swisscom Business DSL connection.
Init7 -> fail
UPC Business -> fail
Tineo/Netrics -> fail
NTS -> fail
TCP dump looks like that
tcpdump -v -ni eth1 host 2a02:a90:c400:4001::2 tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes 08:19:41.525301 IP6 (hlim 64, next-header TCP (6) payload length: 40) 2a00:c38:102::194.39622 > 2a02:a90:c400:4001::2.443: Flags [S], cksum 0x7192 (incorrect -> 0xdf6f), seq 788917482, win 28800, options [mss 1440,sackOK,TS val 2090291639 ecr 0,nop,wscale 7], length 0 08:19:42.522914 IP6 (hlim 64, next-header TCP (6) payload length: 40) 2a00:c38:102::194.39622 > 2a02:a90:c400:4001::2.443: Flags [S], cksum 0x7192 (incorrect -> 0xde75), seq 788917482, win 28800, options [mss 1440,sackOK,TS val 2090291889 ecr 0,nop,wscale 7], length 0 08:19:44.526884 IP6 (hlim 64, next-header TCP (6) payload length: 40) 2a00:c38:102::194.39622 > 2a02:a90:c400:4001::2.443: Flags [S], cksum 0x7192 (incorrect -> 0xdc80), seq 788917482, win 28800, options [mss 1440,sackOK,TS val 2090292390 ecr 0,nop,wscale 7], length 0 08:19:48.534889 IP6 (hlim 64, next-header TCP (6) payload length: 40) 2a00:c38:102::194.39622 > 2a02:a90:c400:4001::2.443: Flags [S], cksum 0x7192 (incorrect -> 0xd896), seq 788917482, win 28800, options [mss 1440,sackOK,TS val 2090293392 ecr 0,nop,wscale 7], length 0 08:19:53.528648 IP6 (hlim 51, next-header TCP (6) payload length: 20) 2a02:a90:c400:4001::2.443 > 2a00:c38:102::194.39622: Flags [R.], cksum 0x85fa (correct), seq 0, ack 788917483, win 0, length 0
Sometimes, if I try hard enough I get a TCP connection and something like a TLS1.3 session but no successul TLS negociation (neither with wget nor with openssl s_client): then it looks like that:
08:27:26.984895 IP6 (flowlabel 0x76030, hlim 64, next-header TCP (6) payload length: 20) 2a02:a90:c400:4001::2.443 > 2001:1a88:2200:505::6.53984: Flags [R.], cksum 0xcb4a (correct), seq 448778729, ack 2002587856, win 234, length 0 08:27:32.727483 IP6 (flowlabel 0x90338, hlim 64, next-header TCP (6) payload length: 40) 2001:1a88:2200:505::6.53988 > 2a02:a90:c400:4001::2.443: Flags [S], cksum 0x9a58 (incorrect -> 0x3075), seq 3336725868, win 64800, options [mss 1440,sackOK,TS val 1531978773 ecr 0,nop,wscale 7], length 0 08:27:32.727864 IP6 (flowlabel 0x53256, hlim 64, next-header TCP (6) payload length: 32) 2a02:a90:c400:4001::2.443 > 2001:1a88:2200:505::6.53988: Flags [S.], cksum 0x6587 (correct), seq 83163391, ack 3336725869, win 28800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0 08:27:32.727910 IP6 (flowlabel 0x90338, hlim 64, next-header TCP (6) payload length: 20) 2001:1a88:2200:505::6.53988 > 2a02:a90:c400:4001::2.443: Flags [.], cksum 0x9a44 (incorrect -> 0x14cb), seq 1, ack 1, win 507, length 0 08:27:32.728373 IP6 (flowlabel 0x90338, hlim 64, next-header TCP (6) payload length: 337) 2001:1a88:2200:505::6.53988 > 2a02:a90:c400:4001::2.443: Flags [P.], cksum 0x9b81 (incorrect -> 0x4bc2), seq 1:318, ack 1, win 507, length 317 08:27:32.728556 IP6 (flowlabel 0x53256, hlim 64, next-header TCP (6) payload length: 20) 2a02:a90:c400:4001::2.443 > 2001:1a88:2200:505::6.53988: Flags [.], cksum 0x149f (correct), seq 1, ack 318, win 234, length 0 08:27:44.732866 IP6 (flowlabel 0x53256, hlim 64, next-header TCP (6) payload length: 20) 2a02:a90:c400:4001::2.443 > 2001:1a88:2200:505::6.53988: Flags [R.], cksum 0x149b (correct), seq 1, ack 318, win 234, length 0
I'm by no means a tcpdump expert:
Those incorrect checksums: are my systems generating incorrect checksums or is it the swisscom side? It seems weird that different systems with different OS at different customers would all start making wrong tcp checksums.
Best Regards
Jean-Pierre
I don't seem to have problem from AS6772:
$ openssl s_client -connect [2a02:a90:c400:5001::2]:https [ssh handshake stuff] GET / HTTP/1.0
HTTP/1.0 301 Moved Permanently Location: https://www.swisscom.ch/de/privatkunden.html Connection: close Content-Length: 252 [...]
Isn't that the case with tcp_offload enabled in the NIC that tcpdump will see incorrect checksums?
I'm by no means a tcpdump expert:
Those incorrect checksums: are my systems generating incorrect checksums or is it the swisscom side? It seems weird that different systems with different OS at different customers would all start making wrong tcp checksums.
Best Regards
Jean-Pierre
On 26.02.21 12:29, Silvan M. Gebhardt wrote:
Isn't that the case with tcp_offload enabled in the NIC that tcpdump will see incorrect checksums?
I'm by no means a tcpdump expert:
Those incorrect checksums: are my systems generating incorrect checksums or is it the swisscom side? It seems weird that different systems with different OS at different customers would all start making wrong tcp checksums.
Checksum errors are rather common to originate in virtualization platforms. It is one of the things to check for when deploying new infrastructure. Even some bigger resellers hand out VMs with these problems: I occasionally have to add a "ethtool -K $IFACE rx off tx off" command to the boot process.
Cheers Claudio
Hi,
On Fri, Feb 26, 2021 at 05:01:38PM +0100, Claudio Luck wrote:
Checksum errors are rather common to originate in virtualization platforms. It is one of the things to check for when deploying new infrastructure. Even some bigger resellers hand out VMs with these problems: I occasionally have to add a "ethtool -K $IFACE rx off tx off" command to the boot process.
These are not true checksum "errors".
It's just that the kernel knows it does not need to bother, because hardware will take care of it, so spends your CPU cycles for more useful work.
*tcpdump* does not know. All tcpdump can see is "I see a packet handed towards the NIC, and the checksum does not match" - which is reported.
Tcpdump does not see how the packet will end up on the wire.
Gert Doering -- NetMaster
I tried different ISPs to which I have access and I can't establish a IPv6 connection to port TCP/443 to neither 2a02:a90:c400:4001::2 nor 2a02:a90:c400:5001::2 which seems the two loadbalancers responsible for www.swisscom.ch except from a Swisscom Business DSL connection.
Init7 -> fail
Strange, from AS13030 (Fiber7 residential), both IPs work fine.
And I can't get a TLS 1.3 handshake to succeed, the load balancers seem to support only TLS 1.1 and 1.2.
Those incorrect checksums: are my systems generating incorrect checksums or is it the swisscom side? It seems weird that different systems with different OS at different customers would all start making wrong tcp checksums.
The incorrect checksums are only shown for packets sent by your client, so most likely not something Swisscom can do anything about. I wouldn't bother too much about them though, this can be caused by checksum offloading and is usually corrected after tcpdump has seen the packets. But it can't hurt to try turning such features on/off to see if it makes a difference.
The first dump shows the client attempting a TCP handshake (multiple SYN messages), but there is no response from the server. Eventually, the server sends a RST, so it must have gotten some of your SYNs.
The second dump shows a successful handshake (SYN, SYN+ACK, ACK) and some data being PuSHed by the client, which is ACKed by the server. Then the server sends a RST. This could indicate incorrect or incomplete packets being sent from your client. A failed TLS 1.3 handshake should look more like SYN, SYN+ACK, ACK, PSH+ACK, PSH+ACK, FIN+ACK.
Perhaps something is wrong with your test client? Wrong MTU? OS issue? Did you try a different machine?
Hi Gregor
Thank you for your insights.
On 02.03.21 20:51, Gregor Riepl wrote:
I tried different ISPs to which I have access and I can't establish a IPv6 connection to port TCP/443 to neither 2a02:a90:c400:4001::2 nor 2a02:a90:c400:5001::2 which seems the two loadbalancers responsible for www.swisscom.ch except from a Swisscom Business DSL connection.
Init7 -> fail Strange, from AS13030 (Fiber7 residential), both IPs work fine.
I just tried it again from a customer with a Fiber7 access and I still can't open www.swisscom.ch neither in a browser edge / firefox on a windows machine, nor from wget on a linux server.
Perhaps something is wrong with your test client? Wrong MTU? OS issue? Did you try a different machine?
I tried from various windows and linux machines from our office (via netrics), from customers (fiber7) and from our datacenter (nts) and it's all the same - I use different routers, different OS versions.
My datacenter ISP (nts) suspects that it's a reverse path issue and advised me to use atlas probes to confirm.
I made two mesurement but the results show more successes that failure:
https://atlas.ripe.net/measurements/29275604/#probes
https://atlas.ripe.net/measurements/29275617/#probes
So maybe it's really only me and I have to live with the fact that I can't login to the swisscom website when v6 connectivity is enabled.
Strange thing is that I even found a swisscom sme fiber customer where the connection to https://www.swisscom.ch times out and one on a ip-plus fiber access. Maybe that's my best bet to open a ticket with their support.
It would be so much easier to debug if their webserver would answer to pings on v6.
I'll keep the list updated if I have any relevant news.
Best Regards and thanks again to all of you who spend some time on this issue already
Jean-Pierre
I have.... your answer :)
$ telnet -6 www.swisscom.ch 80 Trying 2a02:a90:c400:5001::2... telnet: Unable to connect to remote host: Connection refused
same for 443. It takes a while for it to respond with a RST... which is a really cool one:
0000 52 54 00 b2 ca 34 cc 4e 24 45 ef 00 86 dd 60 00 RT...4.N$E....`. 0010 00 00 00 3d 06 2f 2a 02 0a 90 c4 00 40 01 00 00 ...=./*.....@... 0020 00 00 00 00 00 02 20 01 16 20 20 b0 00 00 00 00 ...... .. ..... 0030 00 00 00 00 00 55 01 bb ee 1c 00 00 00 00 c4 a0 .....U.......... 0040 ea f9 50 14 00 00 ff 13 00 00 42 49 47 2d 49 50 ..P.......BIG-IP 0050 3a 20 5b 30 78 32 62 33 38 30 62 35 3a 32 37 31 : [0x2b380b5:271 0060 5d 20 68 61 6e 64 73 68 61 6b 65 20 74 69 6d 65 ] handshake time 0070 6f 75 74 out
I guess that tells it all :)
The fun thing is, that is just a telnet (should have used netcat to ignore telnet attributes).
Greets, Jeroen
--
On 20210309, at 10:49, Jean-Pierre Schwickerath swinog@hilotec.net wrote:
Hi Gregor
Thank you for your insights.
On 02.03.21 20:51, Gregor Riepl wrote:
I tried different ISPs to which I have access and I can't establish a IPv6 connection to port TCP/443 to neither 2a02:a90:c400:4001::2 nor 2a02:a90:c400:5001::2 which seems the two loadbalancers responsible for www.swisscom.ch except from a Swisscom Business DSL connection.
Init7 -> fail Strange, from AS13030 (Fiber7 residential), both IPs work fine.
I just tried it again from a customer with a Fiber7 access and I still can't open www.swisscom.ch neither in a browser edge / firefox on a windows machine, nor from wget on a linux server.
Perhaps something is wrong with your test client? Wrong MTU? OS issue? Did you try a different machine?
I tried from various windows and linux machines from our office (via netrics), from customers (fiber7) and from our datacenter (nts) and it's all the same - I use different routers, different OS versions.
My datacenter ISP (nts) suspects that it's a reverse path issue and advised me to use atlas probes to confirm.
I made two mesurement but the results show more successes that failure:
https://atlas.ripe.net/measurements/29275604/#probes
https://atlas.ripe.net/measurements/29275617/#probes
So maybe it's really only me and I have to live with the fact that I can't login to the swisscom website when v6 connectivity is enabled.
Strange thing is that I even found a swisscom sme fiber customer where the connection to https://www.swisscom.ch times out and one on a ip-plus fiber access. Maybe that's my best bet to open a ticket with their support.
It would be so much easier to debug if their webserver would answer to pings on v6.
I'll keep the list updated if I have any relevant news.
Best Regards and thanks again to all of you who spend some time on this issue already
Jean-Pierre
-- HILOTEC Engineering + Consulting AG - Langnau im Emmental IT für KMUs: Netzwerke, Server, PCs, Linux, Telefonanlagen, VOIP, Hosting, Datenbanken, Entwicklung, WLAN, Cloud, Firewalls Tel: +41 34 408 01 00 - https://www.hilotec.com/
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Dear all
On 09.03.21 11:16, Jeroen Massar wrote:
I have.... your answer :)
$ telnet -6 www.swisscom.ch 80 Trying 2a02:a90:c400:5001::2... telnet: Unable to connect to remote host: Connection refused
I have an update on this issue. I was so blunt and opened a ticket for an IP-Plus customer, claiming that they could not access www.swisscom.ch over IPv6.
It took some time and at least three second-level supporters until someone within IP-Plus found someone from the webserver team who knew who was managing that load-balancer and now IPv6 seems to work again. I have yet to receive an official confirmation that the issue is permanently solved. Maybe they'll share the root cause of the issue but I doubt it.
Best Regards and thank again to all who gave insights into those technical details within tcpdump.
Jean-Pierre
-
Benoît Panizzon -
Claudio Luck -
Gert Doering -
Gregor Riepl -
Jean-Pierre Schwickerath -
Jeroen Massar -
Silvan M. Gebhardt