[ad_1]
Facebook and its sister properties Instagram and WhatsApp suffer from ongoing global blackouts. We don’t yet know why this happened, but the how is clear: Earlier this morning, something inside Facebook prompted the company to revoke key digital records that tell computers and other connected devices on the Internet how to find these destinations online.
Kentik’s take on the Facebook, Instagram and WhatsApp breakdown.
Doug Madory is Director of Internet Analytics at Kentik, a San Francisco-based network monitoring company. Madory said that at around 11:39 a.m. ET today (3:39 p.m. UTC), someone at Facebook caused an update to the company’s Border Gateway Protocol (BGP) records. BGP is a mechanism by which the world’s Internet service providers share information about the providers responsible for routing Internet traffic to which specific groups of Internet addresses.
Put simply, this morning Facebook took away the map telling computers around the world how to find its various properties online. As a result, when typing Facebook.com into a web browser, the browser has no idea where to find Facebook.com and therefore returns an error page.
In addition to blocking billions of users, the Facebook outage also prevented its employees from communicating with each other using their internal Facebook tools. This is because Facebook’s emails and tools are all managed in-house and through the same domains that are now blocked.
“Not only are Facebook’s services and applications inaccessible to the public, but its internal tools and communication platforms, including Workplace, are also available,†the New York Times technical reporter said. Ryan mac tweeted. “No one can work. Several people I spoke to told me that it was the equivalent of a “snow day†at the company. “
Breakdowns occur just a few hours after 60 minutes of CBS broadcast a long-awaited interview with Francoise Haugen, the Facebook whistleblower who recently disclosed a number of internal Facebook investigations showing the company knew its products were causing massive damage and was prioritizing profits rather than taking bolder steps to reduce abuse on its platform, including disinformation and hate speech.
We don’t know how or why the outages persist on Facebook and its other properties, but the changes must have come from within the company, as Facebook manages these records internally. Whether the changes were made maliciously or by accident is a puzzle at this point.
Madory said it could be that someone on Facebook screwed up.
“Over the last year or so, we’ve seen a lot of these big blackouts where some sort of global network configuration update has gone wrong,†Madory said. “We obviously can’t rule out that someone might hack them, but they could have done that to themselves as well.”
Update, 4:37 p.m. ET: Sheera Frenkel with the New York Times tweeted that Facebook employees told him they were having trouble accessing Facebook buildings because their employee badges no longer worked. This could be one of the reasons this outage has persisted for so long: Facebook engineers may have difficulty physically accessing the computer servers needed to upload new BGP records to the global internet.
Update, 6:16 p.m. ET: A trusted source who spoke to someone about the recovery effort on Facebook was told the outage was caused by a routine BGP update gone awry. The source explained that the flawed update prevented Facebook employees – the majority of whom work remotely – from rolling back the changes. Meanwhile, those who had physical access to Facebook buildings could not access Facebook’s internal tools as these were all linked to the company’s blocked domains.
Update, 7:46 p.m. ET: Facebook says its domains are slowly coming back online for most users. In a tweet, the company thanked users for their patience, but still provided no explanation for the outage.
Update, 8:05 p.m. ET: This fascinating discussion thread on Hacker News takes a look at some of the not-so-obvious side effects of today’s outages: Many organizations have seen network disruption and sluggishness thanks to billions of devices constantly asking for current contact details. from Facebook.com, Instagram.com and WhatsApp .com. Bill woodcock, Executive Director of the Packet Clearing House, noted his organization saw an overall 40% increase in temperamental DNS traffic throughout the outage.
Update, 8:32 p.m. ET: Cloudflare posted a detailed and somewhat technical article on the BGP changes that caused today’s outage. Still no word from Facebook on what happened.
Update, 11:32 p.m. ET: Facebook posted a blog post saying the outage was the result of a faulty configuration change:
“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted that communication,†Facebook said. Santosh janardhan wrote. “This disruption in network traffic has had a cascading effect on the way our data centers communicate, shutting down our services. “
“We want to make it clear at this point that we believe the root cause of this failure was a faulty configuration change,†Janardhan continued. “We also have no evidence that user data has been compromised as a result of this downtime.”
Several different domain registrars have now put the Facebook.com domain up for sale. This has happened through automated systems that look for registered domains that appear to be expired, abandoned, or recently released. There was never any reason to believe that Facebook.com would be sold as a result, but it’s fun to consider how many billions of dollars it could make in the open market.
This is a developing story and will likely be updated throughout the day.
[ad_2]
