According to database pioneer Michael Stonebraker, Facebook is operating a huge, complex MySQL implementation equivalent to “a fate worse than death,” and the only way out is “bite the bullet and rewrite everything.”
Not that it’s necessarily Facebook’s fault, though. Stonebraker says the social network’s predicament is all too common among web startups that start small and grow to epic proportions.
During an interview this week, Stonebraker explained to me that Facebook has split its MySQL database into 4,000 shards in order to handle the site’s massive data volume, and is running 9,000 instances of memcached in order to keep up with the number of transactions the database must serve. I’m checking with Facebook to verify the accuracy of those numbers, but Facebook’s history with MySQL is no mystery.
The oft-quoted statistic from 2008 is that the site had 1,800 servers dedicated to MySQL and 805 servers dedicated to memcached, although multiple MySQL shards and memcached instances can run on a single server. Facebook even maintains a MySQL at Facebook page dedicated to updating readers on the progress of its extensive work to make the database scale along with the site.
The widely accepted problem with MySQL is that it wasn’t built for webscale applications or those that must handle excessive transaction volumes. Stonebraker said the problem with MySQL and other SQL databases is that they consume too many resources for overhead tasks (e.g., maintaining ACID compliance and handling multithreading) and relatively few on actually finding and serving data. This might be fine for a small application with a small data set, but it quickly becomes too much to handle as data and transaction volumes grow.
This is a problem for a company like Facebook because it has so much user data, and because every user clicking “Like,” updating his status, joining a new group or otherwise interacting with the site constitutes a transaction its MySQL database has to process. Every second a user has to wait while a Facebook service calls the database is time that user might spend wondering if it’s worth the wait.
Not just a Facebook problem
In Stonebraker’s opinion, “old SQL (as he calls it) is good for nothing” and needs to be “sent to the home for retired software.” After all, he explained, SQL was created decades ago before the web, mobile devices and sensors forever changed how and how often databases are accessed.
But products such as MySQL are also open-source and free, and SQL skills aren’t hard to come by. This means, Stonebraker says, that when web startups decide they need to build a product in a hurry, MySQL is natural choice. But then they hit that hockey-stick-like growth rate like Facebook did, and they don’t really have the time to re-engineer the service from the database up. Instead, he said, they end up applying Band-Aid fixes that solve problems as they occur, but that never really fix the underlying problem of an inadequate data-management strategy.
There have been various attempts to overcome SQL’s performance and scalability problems, including the buzzworthy NoSQL movement that burst onto the scene a couple of years ago. However, it was quickly discovered that while NoSQL might be faster and scale better, it did so at the expense of ACID consistency. As I explained in a post earlier this year about Citrusleaf, a NoSQL provider claiming to maintain ACID properties:
ACID is an acronym for “Atomicity, Consistency, Isolation, Durability” — a relatively complicated way of saying transactions are performed reliably and accurately, which can be very important in situations like e-commerce, where every transaction relies on the accuracy of the data set.
Stonebraker thinks sacrificing ACID is a “terrible idea,” and, he noted, NoSQL databases end up only being marginally faster because they require writing certain consistency and other functions into the application’s business logic.
Stonebraker added, though, that NoSQL is a fine option for storing and serving unstructured or semi-structured data such as documents, which aren’t really suitable for relational databases. Facebook, for example, created Cassandra for certain tasks and also uses the Hadoop-based HBase heavily, but it’s still a MySQL shop for much of its core needs.
Is ‘NewSQL’ the cure?
But Stonebraker — an entrepreneur as much as a computer scientist — has an answer for the shortcoming of both “old SQL” and NoSQL. It’s called NewSQL (a term coined by 451 Group analyst Matthew Aslett) or scalable SQL, as I’ve referred to it in the past. Pushed by companies such as Xeround, Clustrix, NimbusDB, GenieDB and Stonebraker’s own VoltDB, NewSQL products maintain ACID properties while eliminating most of the other functions that slow legacy SQL performance. VoltDB, an online-transaction processing (OLTP) database, utilizes a number of methods to improve speed, including by running entirely in-memory instead of on disk.
It would be easy to accuse Stonebraker of tooting his own horn, but NewSQL vendors have been garnering lots of attention, investment and customers over the past year. There’s no guarantee they’re the solution for Facebook’s MySQL woes — the complexity of Facebook’s architecture and the company’s penchant for open source being among the reasons — but perhaps NewSQL will help the next generation of web startups avoid falling into the pitfalls of their predecessors. Until, that is, it, too, becomes a relic of the Web 3.0 era.
Police investigating allegations of phone hacking by the News of the World have called for patience as they contact almost 4,000 people whose names appear in documents seized in 2005.
The Met’s Deputy Assistant Commissioner Sue Akers said she understood people were upset but police needed time.
She said officers were scouring 11,000 pages of notes as well as dealing with more who feared they were targets.
News International said allegations against its paper had been horrifying.
Among them are claims the paper hacked into the mobile phone of murdered schoolgirl Milly Dowler, the families of 7/7 bombing victims and bereaved military families.
According to a Daily Telegraph report, the phone numbers of relatives of service personnel killed in Iraq and Afghanistan were found in the files of private investigator Glenn Mulcaire, who had been working for the News of the World.
Police have not approached relatives of the soldiers but some families say a newspaper has contacted them suggesting they were victims of phone hacking.
Dep Asst Commissioner Sue Akers, who is in charge of the police investigation known as Operation Weeting, said officers had already spoken to many people and would contact others “as quickly as possible”.
“I have huge sympathy for those who may have been the victims of phone hacking or intrusion into their private lives,” she said.
“It must be incredibly distressing to see details of the information held, or speculation about what may be held, about them in the media. This is forcing them to relive devastating experiences.”
“I stand by my commitment that Operation Weeting will contact all those who have some personal contact details found in the documents seized in 2005 and my officers are working hard to ensure it is fulfilled as soon as possible,” she added.
“This is taking a significant amount of time and resources.”
After the allegations were made over bereaved military families, the Royal British Legion announced it was cutting ties with the News of the World as its campaigning partner, saying it was “shocked to the core”.
The charity campaigned with the newspaper on Military Covenant issues and was set to mount another joint initiative to save the chief coroner’s office from abolition.
Meanwhile, the Independent Police Complaints Commission has been asked to supervise the Met Police’s internal investigation into payments by journalists to police for information.
Met Commissioner Sir Paul Stephenson said a small number of officers were alleged to have taken illegal payments. “If true, I will be determined to root them out, find them and put them in front of the criminal court,” he said.
In other developments:
- Prime Minister David Cameron is consulting MPs about the nature of a public inquiry into the phone-hacking claims, amid support by the deputy prime minister and the Labour leader for a judge-led hearing, with powers to call evidence and examine witnesses under oath
- Shares in BSkyB fall on fears that the News of the World phone-hacking scandal could hinder parent company News Corp’s bid for the broadcaster
- Culture Secretary Jeremy Hunt is set to delay his decision on whether to allow News Corp’s bid for BSkyB after receiving 100,000 submissions on the issue
- Michael Mansfield QC, who represented Mohamed Al Fayed at the Princess Diana inquest, has been told his phone may have been hacked
- The Crown Office say Strathclyde Police have been asked to look at evidence given by witnesses during the Tommy Sheridan perjury trial, in light of recent allegations
- Sainsbury’s supermarket, O2, Flybe, Specsavers and Dixons follow the example of other companies including Ford and Npower by suspending advertising in the NoW
- The government is to urgently review its advertising contracts with the News of the World
- Peter Ridsdale, chairman of Plymouth Argyle, tells BBC Radio Devon, his e-mails were hacked into and he is seeking damages from the News of the World following an article published by the paper when he was chairman of Cardiff City
News International is co-operating with a police inquiry into hacking at the News of the World and is conducting its own investigation into the claims.
“If these allegations are true we are absolutely appalled and horrified,” the company said in a statement, adding that its “record as a friend of the armed services and of our servicemen and servicewomen, is impeccable”.
eBay has acquired mobile payments company Zong for $240 million in cash. The transaction is expected to close in the third quarter of 2011.Zong has been one of the pioneers in the mobile payments space, adding a compelling new way for consumers to pay for items online. Simply put, it lets you pay for things, particularly virtual goods online, via direct billing to your mobile phone. Consumers simply enter their mobile phone numbers in the payments process.
When a user wants to purchase an item, he can enters his cell phone number on a site, the site sends a text message to the phone, the user confirms the transaction with a short reply, and all the charges show up on his phone bill. Zong powers this entire transaction. The company has partnered with over 250 carriers worldwide to offer the technology to mobile phone users.
eBay says that Zong will add ‘complementary technology and talent’ to its PayPal division, giving consumers more ways to pay for virtual goods and products online. Scott Thompson, president of PayPal, said this in a release: “Commerce is changing. With mobile phones, we walk around with a mall in our pockets. PayPal helps to make money work better for customers in this new commerce reality – no matter how they want to pay or what device they’re using…We believe that Zong will strengthen this value by helping us reach the more than 4 billion people who have mobile phones, giving them more choice and security when they pay.”
Zong, which was founded in 2008 by entrepreneur David Marcus, has raised a total of$27.5 million in funding. Matrix Partners’ Dana Stadler, who was the former CTO of PayPal, is on Zong’s board of directors.
Last year, the company was spun off from its European parent, Echovox, and Marcus moved to the U.S. to run the fast growing company, which landed a pretty big deal last year with Facebook to become a an early mobile payment provider for Facebook Credits. Other partners include IMVU, Sulake (makers of Habbo Hotel), Big Fish Games, Sony Online Entertainment, Zynga, Playdom (owned by Disney), and Bigpoint. The company also launched Zong+, an extension of the mobile payment startup which lets users bill microtransactions to credit, debit and prepaid cards (instead of their phones).
Marcus wrote in a post on the company’s site: I am so excited by the unique combination of PayPal’s 8 million merchants, brand power, risk management expertise, and financial stability, with Zong’s Carrier DNA, its largest direct carrier payments network, product innovation, and best-in-class carrier billing technology. This industry first is going to allow us to scale what we’ve built over the course of the past 3 years (and then some) in a massive way!
Zong faces competition from Boku, which was also rumored to be the target of an acquisition as well.
eBay and PayPal have been on a bit of an acquisition spree, so it’s not surprising that the e-commerce giant made another big buy. PayPal is facing competition from fast growing startups like Square and even Google, and the payments company needs to add compelling technologies to help draw merchants, consumers and local businesses. As we’ve seen in the past few months, PayPal acquired local payments and advertising company Where, and shelled out cash for mobile payments company Fig Card as well.
Zong provides a seamless payments product that could help improve conversions for online merchants and digital goods (PayPal processed $3.4 billion in transactions for digital goods in 2010), and in the end, frictionless online payments is PayPal’s bread and butter. In fact, PayPal just upped estimates of the amount of mobile payments transactions using the technology this year; doubling the estimate to $3 billion in mobile total payments volume (TPV) in 2011.
PayPal’s CTO Patrick Dupuis tells us the acquisition is an expansion of vision fo enabling commerce anytime anywhere, adding another payments source for PayPal’s 9 million merchants. Marcus tells us that PayPal’s ability to scale and accelerate with Zong’s built made it an appealing new home.
Another draw for PayPal—Zong has pretty massive international reach, offering mobile payments in in 21 languages and 45 countries. The service’s technology is especially appealing in countries where mobile phone usage is high (as opposed to internet connectivity). Zong should also help PayPal expand its footprint in developing countries.
In terms of integration, PayPal and Zong arent’s saying much as to how Zong will be branded in the future. Marcus says he will be staying on at PayPal.
Unsurprisingly, Dupuis says that Marcus and his team will have a critical role in PayPal’s future transformation in the payments business.
Google has revealed that it is working on a Google+ experience for businesses and is asking brands not to create Google+ profiles just yet.
In a post and accompanying YouTube video on Google+, Product Manager Christian Oestlien says that the Google+ team is working on creating a unique experience for businesses that includes deep analytics and the ability to connect to products like AdWords. “How users communicate with each other is different from how they communicate with brands,” Oestlien argues.
As a result, Google is asking businesses to put their Google+ ambitions on hold.
“The business experience we are creating should far exceed the consumer profile in terms of its usefulness to businesses,” Oestlien says in his post. “We just ask for your patience while we build it. In the meantime, we are discouraging businesses from using regular profiles to connect with Google+ users. Our policy team will actively work with profile owners to shut down non-user profiles.”
While the new Google+ experience for businesses won’t be ready until “later this year,” the company intends to launch a “small experiment with a few marketing partners” to test the brand-oriented accounts over the next few months. It even has opened up a Google Spreadsheet where “non-user entities” can apply for the program. It’s unclear when Google will shut down non-user profiles or how the process will work.
We’re not surprised that Google is building an optimized Google+ experience for businesses, but we are surprised that Google wasn’t more prepared for the wave of brands that have been joining its social network. The same thing happened with Google Buzz and has happened on Facebook, Twitter, Foursquare and countless other social networks. Brands like to go where their customers are.
I recall a time a few years ago when Twitter was scoffed at. It was the blogosphere’s punching-bag. It was the stupid little service that no one in their right mind would ever use. It was for people who wanted to share the mundane bits of their lives — that no one else wanted to read. It was for egomaniacs. Or losers. It would never catch on.
And then it did.
I was thinking about this today as I stood in the East Room of the White House (#humblebrag). Why was I there? To see Twitter co-founder Jack Dorsey moderate a Q&A session with President Obama. Yes, the President of the United States was answering questions that were coming entirely from Twitter, the formerly stupid service that was a joke, remember?
The most fascinating thing about the event today was how well it worked. A week ago, the White House sent out a press release letting people know about the event and asking that questions be submitted via Twitter using the hashtag “#askobama”. Twitter then teamed with Mass Relevance, used their own algorithms, and assigned curators with political expertise to select the questions that would be asked.
In total, there were some 169,395 tweets with the #askobama hashtag, we’re told.
Much like the debates we see during election cycles, neither Dorsey nor the President knew what questions had been selected ahead of time. What ensued was an in-depth and fairly candid conversation about a range of topics that people actually care about. According to the stats compiled by Mass Relevance, 26 percent of the Tweet questions were about jobs, 19 percent were about the budget, 16 percent were about taxes, 10 percent were about education, and 6 percent were about housing. All of the questions were serious in nature.
And questions were also coming in during the event as well. A few times, Dorsey would select one as fresh as 10 minutes old to ask the President. Often, these would be about an answer the President gave earlier during the event. The President would then further clarify his stance. Realtime.
And a few Tweet questions were selected from well-known political entities as well, such as Speaker of the House John Boehner (a Republican) and New York Times columnist Nick Kristof. Some might have a problem with this, since these people can ask questions of the President basically any time — but the transparency of it happening over Twitter was refreshing.
When the President addressed Boehner’s question on debt and job growth, he was quick to note that his rival’s position is “slightly skewed” in his view, but he still answered it — and then made fun of Boehner’s poor typing skills (which apparently wasn’t his fault).
Of course, townhalls are just about as old as modern politics itself. And technology disrupting politics is nothing new either. Many are still certain that Kennedy won the 1960 election simply because he looked better on TV than Nixon, for example. But somehow Twitter and political discourse just feel right together.
I recall having this conversation during the 2008 political primaries. Again, people were largely divided on the issue at the time — as many were still divided on Twitter, a fact not helped by a myriad of downtime issues that year. But with scaling problems far behind them, events like the one today appear to make the Twitter/politics symbiotic relationship more clear than ever. I suspect that during the 2012 election cycle, things are going to get truly insane (hopefully in a good way).
When you compare today’s event to the similar townhall-style event that Facebook did with the President (at Facebook’s headquarters) earlier this year, I don’t think there is any question that this one was better. To me, the Facebook event was more akin to an old-school MTV townhall, or even a “Rock the Vote” event. Facebook is today’s equivalent of MTV. It’s where politicians go to get “cred” in front of a massive audience. But the events themselves are often of little substance.
Twitter, by comparison, is still a fraction of the size of Facebook. The President doesn’t have to use such an event to push some big agenda or to try too hard to appeal to a certain demographic (though there is still some of that, of course), he can just take and answer serious questions seriously. If you had said four years ago that a Twitter event with the President would come across as dignified, everyone would have laughed. Hell, just using the words “Twitter” and “President” in the same sentence would have brought about uncontrollable laughter. But it worked seamlessly today.
Still, some thought today’s event seemed more of a “meaningless marketing stunt” (of course, the author of that post, Umair Haque, is the same guy who held one of the worst Q&A’s the modern tech world has seen at SXSW two years ago with — wait for it — then-Twitter CEO Evan Williams). To me, today seemed like the logical evolution of the townhall format. Forget the questions from the audience which often range from mediocre to poor and involve the politicians needlessly pandering to the crowd — “thank you SO much for your question, and I too grew up on a farm”. Get a question — the best one from anywhere — answer it, move on to the next one.
When I read a few weeks ago that the President would finally be sending his own Tweets from his account, I too was a bit dubious. Why now? Oh, because there is an re-election campaign kicking off, of course. But to be honest, I was more worried that the President being too accessible via a service like Twitter might in some ways demean the postion — not a popular thing to say, perhaps, but it’s something I suspect many people think. I wouldn’t want a President that Tweets all day.
And again, that’s why I think today’s event was the perfect compromise. The President will Tweet from time to time (undoubtedly often to help his campaign), but he should hold back most responses until he can give them in a more dignified setting — like at the White House during a townhall event. Yes, even one that was technically a “Tweetup” (god that word sounds so lame still). And while questions can be brief, answers on important topics often require far more than 140 characters. This format works.
Twitter is a channel through which everyone can be heard. Yes, you still have to sign up for an account, but that’s free — the barrier to entry for participating in these Q&A sessions has never been lower. That’s a great thing. That’s truly powerful. And the White House is right to respect it.
After today’s event, I went back through some of my old posts about Twitter, trying to recall just how little respect the service got even just a few years ago. Here’s one great example: some wondered if Twitter was going to fail because no one knew about it at a random wedding. Right. (And bonus points to me for that part at the end where I wonder what would happen if the iPhone gained a built-in Twitter button…)
Today I stood in the East Room of the White House with a President. To his left hung the Lansdowne portrait — the White House’s old possession — painted in 1797 (and rescued from the 1814 fire) depicting George Washington. To his right hung the companion portrait of Martha Washington painted in 1878. But front and center was a big screen television showing the Tweets directed at the President.
Twitter, the little service that couldn’t, now has the President’s ear.