Google’s 10 rules for designing data centers — Tech News and Analysis

Good information on Google Datacenters.

Google’s 10 rules for designing data centers — Tech News and Analysis:

Why Google Went Offline Today and a Bit about How the Internet Works

Interesting Read from CloudFlare.

Today, Google's services experienced a limited outage for about 27 minutes over some portions of the Internet. The reason this happened dives into the deep, dark corners of networking. I'm a network engineer at CloudFlare and I played a small part in helping ensure Google came back online. Here's a bit about what happened.

At around 6:24pm PST / 02:24 UTC, CloudFlare employees noticed that Google's services were offline. We use Google Apps for things like email so when we can't reach their servers the office notices quickly. I'm on the Network Engineering team so I jumped online to figure out if the problem was local to us or global.

Troubleshooting

I quickly realised that we were unable to resolve all of Googles services — or even reach 8.8.8.8, Googles public DNS server — so I started troubleshooting DNS.
$ dig +trace google.com

Here's the response I got when I tried to reach any of Google.com's name servers:
google.com. 172800 IN NS ns2.google.com. google.com. 172800 IN NS ns1.google.com. google.com. 172800 IN NS ns3.google.com. google.com. 172800 IN NS ns4.google.com. ;; Received 164 bytes from 192.12.94.30#53(e.gtld-servers.net) in 152 ms ;; connection timed out; no servers could be reached

The fact that no servers could be reached means something was wrong. Specifically, it meant that from our office network we were unable to reach any of Googles DNS servers.

I started to look at the network layer, see if that's where the problems lay.
PING 216.239.32.10 (216.239.32.10): 56 data bytes Request timeout for icmp_seq 0 92 bytes from 1-1-15.edge2-eqx-sin.moratelindo.co.id (202.43.176.217): Time to live exceeded

That was curious. Normally, we shouldn't be seeing an Indonesian ISP (Moratel) in the path to Google. I jumped on one of CloudFlare's routers to check what was going on. Meanwhile, others reports from around the globe on Twitter suggested we weren't the only ones seeing the problem.

Internet Routing

To understand what went wrong you need to understand a bit about how networking on the Internet works. The Internet is a collection of networks, known as "Autonomous Systems" (AS). Each network has a unique number to identify it known as AS number. CloudFlare's AS number is 13335, Google's is 15169. The networks are connected together by what is known as Border Gateway Protocol (BGP). BGP is literally the glue of the Internet — announcing what IP addresses belong to each network and establishing the routes from one AS to another. An Internet "route" is exactly what it sounds like: a path from the IP address on AS to another.

BGP is largely a trust-based system. Networks trust each other to say which IP addresses and other networks are behind them. When you send a packet or make a request across the network, your ISP connects to its upstream providers or peers and finds the shortest path from your ISP to the destination network.

Unfortunately, if a network starts to send out an announcement of a particular IP address or network behind it, when in fact it is not, if that network is trusted by its upstreams and peers then packets can end up misrouted. That is what was happening here.

I looked at the BGP Routes for a Google IP Address. The route traversed Moratel (23947), an Indonesian ISP. Given that I'm looking at the routing from California and Google is operating Data Centre's not far from our office, packets should never be routed via Indonesia. The most likely cause was that Moratel was announcing a network that wasn't behind them.

The BGP Route I saw at the time was:
tom@edge01.sfo01> show route 216.239.34.10 inet.0: 422168 destinations, 422168 routes (422154 active, 0 holddown, 14 hidden) + = Active Route, - = Last Active, * = Both 216.239.34.0/24 *[BGP/170] 00:15:47, MED 18, localpref 100 AS path: 4436 3491 23947 15169 I > to 69.22.153.1 via ge-1/0/9.0

Looking at other routes, for example to Google's Public DNS, it was also stuck routing down the same (incorrect) path:
tom@edge01.sfo01> show route 8.8.8.8 inet.0: 422196 destinations, 422196 routes (422182 active, 0 holddown, 14 hidden) + = Active Route, - = Last Active, * = Both 8.8.8.0/24 *[BGP/170] 00:27:02, MED 18, localpref 100 AS path: 4436 3491 23947 15169 I > to 69.22.153.1 via ge-1/0/9.0

Route Leakage



(Image Credit: The Simpsons)

Situations like this are referred to in the industry as "route leakage", as the route has "leaked" past normal paths. This isn't an unprecedented event. Google previously suffered a similar outage when Pakistan was allegedly trying to censor a video on YouTube and the National ISP of Pakistan null routed the service's IP addresses. Unfortunately, they leaked the null route externally. Pakistan Telecom's upstream provider, PCCW, trusted what Pakistan Telecom's was sending them and the routes spread across the Internet. The effect was YouTube was knocked offline for around 2 hours.

The case today was similar. Someone at Moratel likely "fat fingered" an Internet route. PCCW, who was Moratel's upstream provider, trusted the routes Moratel was sending to them. And, quickly, the bad routes spread. It is unlikely this was malicious, but rather a misconfiguaration or an error evidencing some of the failings in the BGP Trust model.

The Fix

The solution was to get Moratel to stop announcing the routes they shouldn't be. A large part of being a network engineer, especially working at a large network like CloudFlare's, is having relationships with other network engineers around the world. When I figured out the problem, I contacted a colleague at Moratel to let him know what was going on. He was able to fix the problem at around 2:50 UTC / 6:50pm PST. Around 3 minutes later, routing returned to normal and Google's services came back online.

Looking at peering maps, I'd estimate the outage impacted around 3–5% of the Internet's population. The heaviest impact will have been felt in Hong Kong, where PCCW is the incumbent provider. If you were in the area and unable to reach Google's services around that time, now you know why.

Building a Better Internet

This all is a reminder about how the Internet is a system built on trust. Today's incident shows that, even if you're as big as Google, factors outside of your direct control can impact the ability of your customers to get to your site so it's important to have a network engineering team that is watching routes and managing your connectivity around the clock. CloudFlare works every day to ensure our customers get the optimal possible routes. We look out for all the websites on our network to ensure that their traffic is always delivered as fast as possible. Just another day in our ongoing efforts to #savetheweb.

Google Chrome Canary Now Available for Mac [Downloads]

Google Chrome Canary Now Available for Mac [Downloads]: "OS X: Google's released the 'Canary' version of their popular Chrome browser for OS X, giving Mac users earlier access to advanced features. More »"

Gmail Now Lets You Make Longer Labels (Without Wasting Screen Space) [In Brief]

Another Gmail bonus. Not that I have super long labels.

Gmail Now Lets You Make Longer Labels (Without Wasting Screen Space) [In Brief]: "

Up until today, you could only make new Gmail labels that were 40 characters long—which could sometimes prove difficult. They've now extended that limit to 225 characters, giving you a lot more leeway in how you organize your messages. They've also truncated the actual labels themselves in your inbox, so they don't take up too much space if they're long or buried in a long trail of nested folders (mousing over them, however, will show you the full label name or path). It's just a few small changes, but certainly welcome ones for label addicts. [Official Gmail Blog] More »"

How to Port Your Number to Google Voice Without Paying an Arm and a Leg [Video]

I've always wondered if this was possible...
How to Port Your Number to Google Voice Without Paying an Arm and a Leg [Video]: "

Google Voice is a great service, but changing your phone number can seem like pulling teeth. If you've been thinking about porting your existing number to Voice, here's what you need to know to make it go as smoothly as possible. More »"

Google Fiber coming to Kansas

Looks like Google has decided on Kansas City, KS for the first fiber installation. We can only hope this will reach as far as Lawrence in the near future.

Google Fiber on YouTube

How Your Username May Betray You

How Your Username May Betray You: "An anonymous reader writes "By creating a distinctive username—and reusing it on multiple websites—you may be giving online marketers and scammers a simple way to track you. Four researchers from the French National Institute of Computer Science (INRIA) studied over 10 million usernames—collected from public Google profiles, eBay accounts, and several other sources. They found that about half of the usernames used on one site could be linked to another online profile, potentially allowing marketers and scammers to build a more complex picture the users."

Google Family Safety Center

I'll be adding this to the list or home computing resources. What a great tool.

Google Family Safety Center

Survey from Google Docs

I have created my first survey from Google Docs. Please take a look and help out with my research!

Jiggle the Cable Survey
(Results)

Facebook Mail?

Given the amount of times I've had to "re-secure" my account settings, I think I'll pass on this one. At least I think/hope the data Google makes available is anonymous. When developers have access to the data (Farmville) you can be sure it's not.

Why You'll Give Up Gmail for Facebook Mail [Facebook]:

"Get ready for a new invasion wave from Facebook: Mail. According to Techcrunch's sources, a full webmail client integrated with The One and Only Social Network will debut next Monday. This is why it may become your favorite webmail service. More »"

SuperOneClick Roots Virtually Every Android Phone Out There [Downloads]

SuperOneClick Roots Virtually Every Android Phone Out There [Downloads]: "Windows only: We recently featured Universal Androot, an app that rooted many Android phones, but had a few limitations (and Google's already killing it with a patch). Windows program SuperOneClick is the easiest, most universal rooting app we've seen yet. More »"

Safe Browsing Alerts for Network Administrators

Safe Browsing Alerts for Network Administrators: "Posted by Nav Jagpal and Ke Wang, Security Team

Google has been working hard to protect its users from malicious web pages, and also to help webmasters keep their websites clean. When we find malicious content on websites, we attempt to notify their webmasters via email about the bad URLs. There is even a Webmaster Tools feature that helps webmasters identify specific malicious content that has been surreptitiously added to their sites, so that they can clean up their site and help prevent it from being compromised in the future.

Today, we’re happy to announce Google Safe Browsing Alerts for Network Administrators -- an experimental tool which allows Autonomous System (AS) owners to receive early notifications for malicious content found on their networks. A single network or ISP can host hundreds or thousands of different websites. Although network administrators may not be responsible for running the websites themselves, they have an interest in the quality of the content being hosted on their networks. We’re hoping that with this additional level of information, administrators can help make the Internet safer by working with webmasters to remove malicious content and fix security vulnerabilities.

To get started, visit safebrowsingalerts.googlelabs.com.

Google Family Safety Center

Announcing our new Family Safety Center: "posted by Kate Hammond, Marketing Manager

Helping your children use the Internet safely is similar to teaching them to navigate the offline world. There are parts of the real world that you wouldn’t let your children explore unsupervised—and that goes for the online world as well. But while most of us remember being taught to cross the road and not talk to strangers, we probably weren’t taught how much personal information we should share online or how to handle cyberbullies.

Therefore, it’s no surprise when parents and teachers tell us they want to learn more about how to help their kids use the Internet safely and responsibly.

Today, we’re launching our new Family Safety Center; a one-stop shop about staying safe online. We’ve included advice from leading child safety organizations around the world, tips and ideas from parents here at Google, as well as information on how to use the safety tools and controls built into Google products.

For day-to-day practical tips we asked some of our parents at Google to share their own ideas. Tactics they use range from limiting screen time and preventing computers in kids’ bedrooms to ad hoc checks on their browser history and social networking profiles. Everyone has different ideas and there’s no right or wrong answer, but hopefully some of these will resonate and inspire you. See more videos and let us know your own thoughts on our YouTube channel.



To answer some of the toughest questions most important to parents, such as accessing inappropriate content and meeting strangers online, we went straight to the people that know best; the organizations that advocate and promote child safety and digital literacy. Organizations that we’ve partnered with around the world include the U.S. Federal Trade Commission’s OnGuard Online initiative, the Canadian Centre for Child Protection, the Australian Communications and Media Authority, U.K.’s ChildNet, and New Zealand’s NetSafe.

The new Safety Center also provides information on the safety tools built into Google products. SafeSearch and YouTube Safety Mode can help you control what content your children stumble across. Sharing controls in YouTube, Picasa, Blogger and others ensure your videos, photos and blogs are shared only with the right people. And in response to popular requests, we’ve added a section on managing geolocation features on mobiles.

With kids growing up in an age where digital know-how is essential, it’s increasingly important to ensure that they’re developing healthy, safe and responsible online habits. And we’re thinking every day about how we can help parents and teachers to do just that.

Millions of Android users hit by malicious data theft app

Surprised?

Millions of Android users hit by malicious data theft app: "An app distributed by Google's Android Market has collected private data from millions of users and forwarded it to servers China, validating Apple's uniquely strong stance on mobile security in the iPhone App Store."

Lawrence to become first city in Kansas to get smart electricity meters

Never thought I'd see the day. Maybe I can get some of that Google PowerMeter action now.

---

Lawrence to become first city in Kansas to get smart electricity meters: "Thanks to a $19 million grant from the U.S. Department of Energy, Lawrence will be the first city in Kansas hooked into the smart grid."

Google Toolbar Tracks Your Browsing, Even When Off

Are we supposed to be surprised? Isn't this what it's designed to do?

---

Google Toolbar Tracks Your Browsing, Even When Off: "garg0yle writes 'Google's Toolbar is supposed to allow the user to disable it. However, it was discovered by a researcher that it was still sending information even when disabled. A patch is now available, and Google claims this was just a bug, not a feature.'

GMail Enables HTTPS by Default

I've always wondered why this wasn't the case to begin with...

---

To disable this feature, go to Gmail's settings page, select 'Don't always use https' and click on 'Save changes'. If you can't use Gmail offline when this feature is enabled, try this workaround.

Gmail's HTTPS Access Is Enabled by Default: "Unlike other popular webmail services, Gmail allows you to read your messages using a secure connection by visiting https://mail.google.com. In 2008, Gmail added an option that redirected you to the https version and now this option is enabled by default.

'Using https helps protect data from being snooped by third parties, such as in public wifi hotspots. We initially left the choice of using it up to you because there's a downside: https can make your mail slower since encrypted data doesn't travel across the web as quickly as unencrypted data. Over the last few months, we've been researching the security/latency tradeoff and decided that turning https on for everyone was the right thing to do,' explains Gmail's blog.

Even if this feature is restricted to Gmail, there's an interesting side-effect: if you open Google Calendar, Google Docs, Google Sites and Google Reader by clicking on Gmail's navigational links, you'll use the https versions of those services.

Google Autocomplete

Recently I started seeing a lot of these and it intrigued me. So I tried my own and discovered how easy (and addicting) it can be. I quickly discovered there is a blog dedicated to Google Autocompelete (AutocompleteMe.com).

Google Dashboard - For all your Google Settings

I never really thought it was hard to manage this stuff before. Perhaps I will find something new. Still this is a great idea for heavy Google users.

---

Transparency, choice and control - now complete with a Dashboard!: "Posted by Alma Whitten, Software Engineer, Yariv Adan, Product Manager, and Marissa Mayer, VP of Search Products and User Experience

(Cross-posted from the Official Google Blog.)

Today, we are excited to announce the launch of Google Dashboard. Have you ever wondered what data is stored with your Google Account? The Google Dashboard offers a simple view into the data associated with your account — easily and concisely in one location.

Over the past 11 years, Google has focused on building innovative products for our users. Today, with hundreds of millions of people using those products around the world, we are very aware of the trust that you have placed in us, and our responsibility to protect your privacy and data. In the past, we've taken numerous steps in this area, investing in educating our users with our Privacy Center, making it easier to move data in and out of Google with our Data Liberation Front, and allowing you to control the ads you see with interest-based advertising. Transparency, choice and control have become a key part of Google's philosophy, and today, we're happy to announce that we're doing even more.

In an effort to provide you with greater transparency and control over their own data, we've built the Google Dashboard. Designed to be simple and useful, the Dashboard summarizes data for each product that you use (when signed in to your account) and provides you direct links to control your personal settings. Today, the Dashboard covers more than 20 products and services, including Gmail, Calendar, Docs, Web History, Orkut, YouTube, Picasa, Talk, Reader, Alerts, Latitude and many more. The scale and level of detail of the Dashboard is unprecedented, and we're delighted to be the first Internet company to offer this — and we hope it will become the standard. Watch this quick video to learn more and then try it out for yourself at www.google.com/dashboard.

Gmail Security Tips

Here are some great tips for Gmail account security from the GMail Blog. I think the last one is really important. There have been numerous cases in the media recently about free email accounts being compromised using "security questions". At the very least, make sure you have a reliable backup email account listed for password recovery.

---

Gmail account security tips: "Posted by Sarah Price, Online Operations Strategist

As part of National Cyber Security Awareness Month, we recently posted about how to pick a smart password. Having a strong password goes a long way in helping to protect your data, but there are a number of additional steps you can take to help you keep your Gmail account secure:

1. Remember to sign out. Especially when using a public computer, be careful to sign out of your Google account when you're finished. Just click the 'Sign out' link at the top right corner of your inbox. If you're using a public or shared computer and want to be extra thorough, you can also clear the browser's cache, cookies and history. Then, completely close the browser. On your personal computer, you can also lock your computer with a password-protected screensaver if you need to step away momentarily. Learn the best ways to lock your screen in Windows or in Mac OS X. Forgot to sign out? Open up a new Gmail session on another computer and use Gmail's remote sign out feature to close any sessions that might still be open elsewhere.

2. Be careful about sending certain sensitive information via email. Once you send an email, you're no longer in control of the information it contains. The recipients, if they so choose, could forward the email or post its contents in a public place. Even if you know and trust the people you're emailing, that information may become exposed if their accounts become compromised or they get a virus on their machines. As a rule of thumb, should you need to provide a credit card number or financial account number to respond to a message, provide it over the phone or in person — not over email. And never share your password with anyone. Google does not email you to ask you for your password, your social security number, or other personal information — so don't send it!

3. Enable 'Always use HTTPS.' Any time you visit a webpage, your computer needs to send and receive information across the Internet. HTTPS is used to encrypt data as it is transmitted between computers on the Internet, so look for the 'https' in the URL bar of your browser to indicate that the connection between your computer and Gmail's servers is encrypted. We use HTTPS on the Gmail login page, and you can choose to protect your entire Gmail session with HTTPS as well. HTTPS can make your mail slower, so we let you make the choice for yourself. Open Settings and choose 'Always use HTTPS' on the General tab if you want to turn it on.

4. Be wary of unexpected attachments.To help protect you from viruses and malware, Gmail automatically scans every attachment when it's delivered to you, and again each time you open a message. Attachments you send are also scanned. That said, no system is foolproof, so if you happen to get an email from a friend with an attachment you didn't expect, don't be afraid to ask the sender what it is before you decide whether to open it.

5. Make sure your account recovery information is up-to-date. Your account recovery information helps you regain access to your account if you ever forget your password, or if someone gains access to your account without your permission. We currently offer several paths to account recovery. Every Gmail user must select a security question and answer — be sure to choose a combination that is easy for you to remember, but hard for others to guess or come across by investigating. Don't choose a question like "What is my favorite color?" as others may easily guess the answer. We also encourage you to provide a secondary email address and/or a mobile phone number, so we can send you a link to reset your password if you lose access to your account.