Why Google Went Offline Today and a Bit about How the Internet Works

Interesting Read from CloudFlare.

Today, Google's services experienced a limited outage for about 27 minutes over some portions of the Internet. The reason this happened dives into the deep, dark corners of networking. I'm a network engineer at CloudFlare and I played a small part in helping ensure Google came back online. Here's a bit about what happened.

At around 6:24pm PST / 02:24 UTC, CloudFlare employees noticed that Google's services were offline. We use Google Apps for things like email so when we can't reach their servers the office notices quickly. I'm on the Network Engineering team so I jumped online to figure out if the problem was local to us or global.

Troubleshooting

I quickly realised that we were unable to resolve all of Googles services — or even reach 8.8.8.8, Googles public DNS server — so I started troubleshooting DNS.
$ dig +trace google.com

Here's the response I got when I tried to reach any of Google.com's name servers:
google.com. 172800 IN NS ns2.google.com. google.com. 172800 IN NS ns1.google.com. google.com. 172800 IN NS ns3.google.com. google.com. 172800 IN NS ns4.google.com. ;; Received 164 bytes from 192.12.94.30#53(e.gtld-servers.net) in 152 ms ;; connection timed out; no servers could be reached

The fact that no servers could be reached means something was wrong. Specifically, it meant that from our office network we were unable to reach any of Googles DNS servers.

I started to look at the network layer, see if that's where the problems lay.
PING 216.239.32.10 (216.239.32.10): 56 data bytes Request timeout for icmp_seq 0 92 bytes from 1-1-15.edge2-eqx-sin.moratelindo.co.id (202.43.176.217): Time to live exceeded

That was curious. Normally, we shouldn't be seeing an Indonesian ISP (Moratel) in the path to Google. I jumped on one of CloudFlare's routers to check what was going on. Meanwhile, others reports from around the globe on Twitter suggested we weren't the only ones seeing the problem.

Internet Routing

To understand what went wrong you need to understand a bit about how networking on the Internet works. The Internet is a collection of networks, known as "Autonomous Systems" (AS). Each network has a unique number to identify it known as AS number. CloudFlare's AS number is 13335, Google's is 15169. The networks are connected together by what is known as Border Gateway Protocol (BGP). BGP is literally the glue of the Internet — announcing what IP addresses belong to each network and establishing the routes from one AS to another. An Internet "route" is exactly what it sounds like: a path from the IP address on AS to another.

BGP is largely a trust-based system. Networks trust each other to say which IP addresses and other networks are behind them. When you send a packet or make a request across the network, your ISP connects to its upstream providers or peers and finds the shortest path from your ISP to the destination network.

Unfortunately, if a network starts to send out an announcement of a particular IP address or network behind it, when in fact it is not, if that network is trusted by its upstreams and peers then packets can end up misrouted. That is what was happening here.

I looked at the BGP Routes for a Google IP Address. The route traversed Moratel (23947), an Indonesian ISP. Given that I'm looking at the routing from California and Google is operating Data Centre's not far from our office, packets should never be routed via Indonesia. The most likely cause was that Moratel was announcing a network that wasn't behind them.

The BGP Route I saw at the time was:
tom@edge01.sfo01> show route 216.239.34.10 inet.0: 422168 destinations, 422168 routes (422154 active, 0 holddown, 14 hidden) + = Active Route, - = Last Active, * = Both 216.239.34.0/24 *[BGP/170] 00:15:47, MED 18, localpref 100 AS path: 4436 3491 23947 15169 I > to 69.22.153.1 via ge-1/0/9.0

Looking at other routes, for example to Google's Public DNS, it was also stuck routing down the same (incorrect) path:
tom@edge01.sfo01> show route 8.8.8.8 inet.0: 422196 destinations, 422196 routes (422182 active, 0 holddown, 14 hidden) + = Active Route, - = Last Active, * = Both 8.8.8.0/24 *[BGP/170] 00:27:02, MED 18, localpref 100 AS path: 4436 3491 23947 15169 I > to 69.22.153.1 via ge-1/0/9.0

Route Leakage



(Image Credit: The Simpsons)

Situations like this are referred to in the industry as "route leakage", as the route has "leaked" past normal paths. This isn't an unprecedented event. Google previously suffered a similar outage when Pakistan was allegedly trying to censor a video on YouTube and the National ISP of Pakistan null routed the service's IP addresses. Unfortunately, they leaked the null route externally. Pakistan Telecom's upstream provider, PCCW, trusted what Pakistan Telecom's was sending them and the routes spread across the Internet. The effect was YouTube was knocked offline for around 2 hours.

The case today was similar. Someone at Moratel likely "fat fingered" an Internet route. PCCW, who was Moratel's upstream provider, trusted the routes Moratel was sending to them. And, quickly, the bad routes spread. It is unlikely this was malicious, but rather a misconfiguaration or an error evidencing some of the failings in the BGP Trust model.

The Fix

The solution was to get Moratel to stop announcing the routes they shouldn't be. A large part of being a network engineer, especially working at a large network like CloudFlare's, is having relationships with other network engineers around the world. When I figured out the problem, I contacted a colleague at Moratel to let him know what was going on. He was able to fix the problem at around 2:50 UTC / 6:50pm PST. Around 3 minutes later, routing returned to normal and Google's services came back online.

Looking at peering maps, I'd estimate the outage impacted around 3–5% of the Internet's population. The heaviest impact will have been felt in Hong Kong, where PCCW is the incumbent provider. If you were in the area and unable to reach Google's services around that time, now you know why.

Building a Better Internet

This all is a reminder about how the Internet is a system built on trust. Today's incident shows that, even if you're as big as Google, factors outside of your direct control can impact the ability of your customers to get to your site so it's important to have a network engineering team that is watching routes and managing your connectivity around the clock. CloudFlare works every day to ensure our customers get the optimal possible routes. We look out for all the websites on our network to ensure that their traffic is always delivered as fast as possible. Just another day in our ongoing efforts to #savetheweb.

Microsoft Is Turning Kinect Into a Narc

This really creeps me out. From Gizmodo.

Kinect is tons of fun. Have you ever played Dance Central 3? Great game. But according to a newly discovered patent, the Xbox add-on is also maybe spying on you, which is totally not cool, man.

This very big brother-y piece of intellectual property—Content Distribution Regulation by Viewing User—uses Kinect's camera to count the number of people in the room and in some cases, identify who they are. This "consumer detector" will charge you licensing fees based on how many bodies are present, and could even stop playback to collect on you if it detects more humans than you've paid for.

Sorry if you have a baby face, because the tech could also check on ages and cut off mature content if the system doesn't think you're old enough. God, Kinect. Such a tattletale! [USPTO viaGeekwire via BetaBeat]

SNMP Basics

Here is a great writeup on SNMP basics that I have blatantly stolen from Paessler.com.

SNMP, MIBs and OIDs - an Overview

SNMP is one of the most commonly used technologies when it comes to network monitoring. Bandwidth Monitoring programs like PRTG Network Monitor use it. But how does SNMP work? What are MIBs and OIDs? Read this short introduction into the world of SNMP!

SNMP Basics

SNMP stands for Simple Network Management Protocol and consists of three key components: managed devices, agents, and network-management systems (NMSs). A managed device is a node that has an SNMP agent and resides on a managed network. These devices can be routers and access server, switches and bridges, hubs, computer hosts, or printers. An agent is a software module residing within a device. This agent translates information into a compatible format with SNMP. An NMS runs monitoring applications. They provide the bulk of processing and memory resources required for network management.

MIB, OID, etc.

MIB stands for Management Information Base and is a collection of information organized hierarchically. These are accessed using a protocol such as SNMP. There are two types of MIBs: scalar and tabular. Scalar objects define a single object instance whereas tabular objects define multiple related object instances grouped in MIB tables.
OIDs or Object Identifiers uniquely identify manged objects in a MIB hierarchy. This can be depicted as a tree, the levels of which are assigned by different organizations. Top level MIB object IDs (OIDs) belong to different standard organizations. Vendors define private branches including managed objects for their own products.
SNMP version 1, which is the SNMP standard supported by PRTG Network Monitor, was the initial development of the SNMP protocol. A description can be found in Request for Comments (RFC) 1157 and it functions within the specification of the Structure of Management Information (SMI). It operates over User Datagram Protocol (UDP), Internet Protocol (IP), OSI Connectionless Network Services (CLNS), AppleTalk Datagram Delivery Prtocol (DDP), and Novell Internet Packet Exchange (IPX). SNMP v1 is considered the de facto network management protocol in the Internet community.
SNMP works on the basis that network management systems send out a request and the managed devices return a response. This is implemented using one of four operations: Get, GetNext, Set, and Trap. SNMP messages consist of a header and a PDU (protocol data units). The headers consist of the SNMP version number and the community name. The community name is used as a form of security in SNMP. The PDU depends on the type of message that is being sent. The Get, GetNext, and Set, as well as the response PDU, consist of PDU type, Request ID, Error status, Error index and Object/variable fields. The Trap consist of Enterprise, Agent, Agent address, Generic trap type, Specific trap code, Timestamp and Object/Value fields.
MIBs are a collection of definitions which define the properties of the managed object within the device to be managed (such as a router, switch, etc.) Each managed device keeps a database of values for each of the definitions written in the MIB. As such, it is not actually database but implementation dependant. Each vendor of SNMP equipment has an exclusive section of the MIB tree structure under their control.
In order for all of this to be properly organized, all of the manageable features of all products (from each vendor) are arranged in this tree. Each 'branch' of this tree has a number and a name, and the complete path from the top of the tree down to the point of interest forms the name of that point. This is the OID. Nodes near the top of the tree are extremely general I nature. For example, to get to the Internet, one has to reach to the fourth tier. As one moves further down, the names get more and more specific, until one gets to the bottom, where each node represents a particular feature on a specific device (or agent).

Mac Terminal Tricks

I use the Terminal application on my Mac all day long.  I have numerous different systems that I connect to for different tasks.  Generally, I have one terminal open and I can just ssh to various systems from that window.  Lately however, I have been customizing things a bit.  One thing I have liked is creating a default look (profile) for various systems so I know specifically what box I am on.  Some systems have nearly identical test / dev systems and the last thing I want to do is enter a command on a production system when it was meant for test.  Once inside the terminal application I can simply do a Shift-Command-N to bring up the command window and type in the profile for that system.  Since I can enter the initial command for each profile, this makes connecting to various systems with a specialized look very simple and easy.  So I can have my test system with green text and my production systems all glowing red.

*update*  It appears that you only get the custom colors and not the actual command.  I have posted in the Support Community. I'll update here if I get any response.

Procedures, Documentation, and Checklists, Oh My!

I cannot stress the importance of procedures, documentation and checklists enough when working in information technology. Not just using them, but KEEPING THEM UP TO DATE!

Just yesterday I ran into a situation where I thought something was being done one way, when someone else in my group did it a different way. We were both following procedures laid out in a checklist that was approved by our managers but something was left off the list. I thought it was always being done one way while he did it a different way. Apparently my way wasn't being done at all and since his method was different, he never thought there was any issue. When I was questioned about it by my manager, it was discovered that something was missing from our checklist.

Having all the checklists and procedures in the world is rendered completely useless if they aren't updated and checked regularly and verified against old old work to ensure uniformity and accuracy.

Next week: "Tools and Recommendations for Tracking Procedures"