OpenDNS London - Peering messed up?

Comments

8 comments

  • Avatar
    rotblitz

    Did you also open a support ticket?

    And the following command outputs may help too:

    nslookup -type=txt which.opendns.com. 208.67.222.222
    nslookup -type=txt which.opendns.com. 208.67.220.220
    tracert 208.67.222.222
    tracert 208.67.220.220

    This is for Windows.  If you have Linux or Mac OSX, use traceroute instead.

  • Avatar
    ivan_mcbeal

    Yes, seems to be widespread. for me, Frankfurt yesterday Washington today

    C:\>nslookup -type=txt which.opendns.com. 208.67.222.222
    Server:  resolver1.opendns.com
    Address:  208.67.222.222

    Non-authoritative answer:
    which.opendns.com       text =

            "3.ash"

    C:\>nslookup -type=txt which.opendns.com. 208.67.220.220
    Server:  resolver2.opendns.com
    Address:  208.67.220.220

    Non-authoritative answer:
    which.opendns.com       text =

            "9.ash"

    C:\>tracert 208.67.222.222

    Tracing route to resolver1.opendns.com [208.67.222.222]
    over a maximum of 30 hops:

      1    <1 ms    <1 ms     1 ms  dsldevice.lan [192.168.1.254]
      2    10 ms     9 ms    48 ms  lo0-central10.pcl-ag01.plus.net [195.166.128.182]
      3     8 ms     9 ms     8 ms  link-b-central10.pcl-gw02.plus.net [212.159.2.162]
      4     8 ms     8 ms     8 ms  xe-10-0-0.pcl-cr02.plus.net [212.159.0.194]
      5     8 ms     9 ms     8 ms  ae2.pcl-cr01.plus.net [195.166.129.6]
      6     8 ms    10 ms     8 ms  xe-11-2-0.edge3.London2.Level3.net [212.187.201.213]
      7    89 ms    88 ms    88 ms  vl-3201-ve-128.ebr2.London2.Level3.net [4.69.202.177]
      8    88 ms    88 ms    88 ms  ae-43-43.ebr1.Paris1.Level3.net [4.69.159.90]
      9    90 ms    89 ms    89 ms  ae-61-61.csw1.Paris1.Level3.net [4.69.161.78]
     10    88 ms    88 ms    88 ms  ae-62-62.ebr2.Paris1.Level3.net [4.69.161.93]
     11    89 ms    88 ms    89 ms  ae-44-44.ebr2.Washington1.Level3.net [4.69.137.62]
     12    88 ms    88 ms    88 ms  ae-82-82.csw3.Washington1.Level3.net [4.69.134.154]
     13    89 ms    89 ms    88 ms  ae-3-80.edge1.Washington4.Level3.net [4.69.149.143]
     14   135 ms    88 ms    88 ms  OPEN-DNS-IN.edge1.Washington4.Level3.net [4.53.112.158]
     15    88 ms    89 ms    88 ms  resolver1.opendns.com [208.67.222.222]

    Trace complete.

    C:\>tracert 208.67.220.220

    Tracing route to resolver2.opendns.com [208.67.220.220]
    over a maximum of 30 hops:

      1    <1 ms     1 ms     1 ms  dsldevice.lan [192.168.1.254]
      2    13 ms    10 ms     8 ms  lo0-central10.pcl-ag01.plus.net [195.166.128.182]
      3     8 ms     8 ms     8 ms  link-b-central10.pcl-gw02.plus.net [212.159.2.162]
      4     8 ms     8 ms     8 ms  xe-10-0-0.pcl-cr02.plus.net [212.159.0.194]
      5    10 ms     8 ms     8 ms  ae1.ptw-cr02.plus.net [195.166.129.2]
      6     9 ms    10 ms     8 ms  ae2.ptw-cr01.plus.net [195.166.129.4]
      7    25 ms    15 ms     9 ms  te-3-4.car5.London1.Level3.net [217.163.45.181]

      8    84 ms    83 ms    94 ms  ae-52-52.csw2.London1.Level3.net [4.69.139.120]

      9    84 ms    84 ms    84 ms  ae-57-222.ebr2.London1.Level3.net [4.69.153.133]

     10    83 ms    83 ms    83 ms  ae-44-44.ebr1.NewYork1.Level3.net [4.69.137.78]

     11    83 ms    83 ms    83 ms  ae-47-47.ebr1.NewYork2.Level3.net [4.69.201.46]

     12    84 ms    83 ms    83 ms  ae-40-40.ebr2.Washington1.Level3.net [4.69.201.93]
     13    83 ms    83 ms    83 ms  ae-72-72.csw2.Washington1.Level3.net [4.69.134.150]
     14    83 ms    83 ms    85 ms  ae-2-70.edge1.Washington4.Level3.net [4.69.149.79]
     15    83 ms   124 ms    83 ms  OPEN-DNS-IN.edge1.Washington4.Level3.net [4.53.112.158]
     16    83 ms    83 ms    83 ms  resolver2.opendns.com [208.67.220.220]

    Trace complete.

  • Avatar
    rotblitz

    Yes, funny routing.  As nothing is reported at http://system.opendns.com/, what you can do is to talk to your ISP and to open a support ticket with OpenDNS, as already said.  Both may have limited support during weekends.

  • Avatar
    -andyh-

    I've got open tickets with OpenDNS and Level3, although I doubt they will be picked up until Monday. 

    This is quite a big problem because every Level3 European core (30+) is sending OpenDNS traffic (208.67.222.0/24) to Washington. There is little ISPs can do, other than raise it with Level3 and OpenDNS.

     

    The BGP Tables are totally messed up:

     

    Traceroute results from Frankfurt, Germany
    to 208.67.222.222(resolver1.opendns.com)
    1 vlan80.csw3.Frankfurt1.Level3.net (4.69.154.190) 88 msec
    vlan70.csw2.Frankfurt1.Level3.net (4.69.154.126) 92 msec
    vlan90.csw4.Frankfurt1.Level3.net (4.69.154.254) 92 msec
    2 ae-61-61.ebr1.Frankfurt1.Level3.net (4.69.140.1) 100 msec
    ae-81-81.ebr1.Frankfurt1.Level3.net (4.69.140.9) 88 msec
    ae-71-71.ebr1.Frankfurt1.Level3.net (4.69.140.5) 88 msec
    3 ae-47-47.ebr2.Paris1.Level3.net (4.69.143.142) 88 msec 88 msec 88 msec
    4 ae-41-41.ebr2.Washington1.Level3.net (4.69.137.50) 88 msec
    ae-43-43.ebr2.Washington1.Level3.net (4.69.137.58) 88 msec
    ae-44-44.ebr2.Washington1.Level3.net (4.69.137.62) 92 msec
    5 ae-82-82.csw3.Washington1.Level3.net (4.69.134.154) 92 msec
    ae-92-92.csw4.Washington1.Level3.net (4.69.134.158) 88 msec 92 msec
    6 ae-3-80.edge1.Washington4.Level3.net (4.69.149.143) 88 msec 116 msec
    ae-2-70.edge1.Washington4.Level3.net (4.69.149.79) 88 msec
    7 OPEN-DNS-IN.edge1.Washington4.Level3.net (4.53.112.158) 52 msec 212 msec 224 msec
    8 resolver1.opendns.com (208.67.222.222) [AS36692 {OPENDNS}] 208 msec 208 msec 216 msec

    Traceroute results from AMSIX NAP Amsterdam
    to 208.67.222.222(resolver1.opendns.com)
    1 ae-52-52.csw2.Amsterdam1.Level3.net (4.69.139.185) 216 msec 212 msec 216 msec
    2 ae-57-222.ebr2.Amsterdam1.Level3.net (4.69.153.205) 208 msec
    ae-59-224.ebr2.Amsterdam1.Level3.net (4.69.153.213) 204 msec 212 msec
    3 ae-46-46.ebr2.London1.Level3.net (4.69.143.74) 364 msec 144 msec
    ae-45-45.ebr2.London1.Level3.net (4.69.143.70) 228 msec
    4 ae-44-44.ebr1.NewYork1.Level3.net (4.69.137.78) 112 msec
    ae-43-43.ebr1.NewYork1.Level3.net (4.69.137.74) 168 msec
    ae-44-44.ebr1.NewYork1.Level3.net (4.69.137.78) 84 msec
    5 ae-47-47.ebr1.NewYork2.Level3.net (4.69.201.46) 116 msec
    ae-57-57.ebr2.Washington12.Level3.net (4.69.201.66) 148 msec
    ae-55-55.ebr2.Washington12.Level3.net (4.69.148.50) 288 msec
    6 ae-40-40.ebr2.Washington1.Level3.net (4.69.201.93) 208 msec
    ae-46-46.ebr2.Washington1.Level3.net (4.69.202.53) 204 msec
    ae-45-45.ebr2.Washington1.Level3.net (4.69.143.221) 160 msec
    7 ae-62-62.csw1.Washington1.Level3.net (4.69.134.146) 80 msec
    ae-72-72.csw2.Washington1.Level3.net (4.69.134.150) 80 msec
    ae-82-82.csw3.Washington1.Level3.net (4.69.134.154) 84 msec
    8 ae-3-80.edge1.Washington4.Level3.net (4.69.149.143) 84 msec
    ae-4-90.edge1.Washington4.Level3.net (4.69.149.207) 80 msec
    ae-1-60.edge1.Washington4.Level3.net (4.69.149.15) 84 msec
    9 OPEN-DNS-IN.edge1.Washington4.Level3.net (4.53.112.158) 84 msec 84 msec 80 msec
    10 resolver1.opendns.com (208.67.222.222) [AS36692 {OPENDNS}] 84 msec 84 msec 80 msec

  • Avatar
    rotblitz

    "There is little ISPs can do, other than raise it with Level3 and OpenDNS."

    That's it.  As network carriers do not accept issue reports from end users, just from their customers, i.e. ISPs and OpenDNS.  I made painful experiences with this in the past.

  • Avatar
    -andyh-

    Hopefully OpenDNS will give an answer on this - I have tried with Level3 before without success (as you also found).

    Routing is back to Frankfurt again (for the whole of Europe).

    Unless OpenDNS deliberately want all European Level3 traffic to be directed to Frankfurt, I don't understand why everything is sent there when you have local datacentres in some countries. 

  • Avatar
    -andyh-

    Just an update on this one. OpenDNS were very good at providing an explanation as to what happened:

    We were having packet loss issues with Level3 specifically over the weekend, and at various times had to bring down our connection with them at our London, Frankfurt, and Amsterdam datacenters. Those datacenters do have additional providers, but it appears that Level3 was preferring to route you internally.

    Frankfurt in particular was only turned off for a brief amount of time on Sunday, but as the Level3 connections to both London and Amsterdam were also off at that point, Level3 apparently choose to route you to our datacenter in Ashburn, Virginia. Unfortunately, we cannot control how Level3 chooses to route its traffic on it's own network.

    This periodic routing to Frankfurt is the same issue, as we occasionally do have packet loss issues with them and must disable the connection until they are resolved.

     

    So was a Level3 issue after all.

  • Avatar
    rotblitz

    Thanks!

Post is closed for comments.