APRIL 18, 2014
Heartbleed Highlights a Contradiction in the Web
By NICOLE PERLROTH
SAN FRANCISCO -- The Heartbleed bug that made news last week drew attention to one of the least understood elements of the Internet: Much of the invisible backbone of websites from Google to Amazon to the Federal Bureau of Investigation was built by volunteer programmers in what is known as the open-source community.
Heartbleed originated in this community, in which these volunteers, connected over the Internet, work together to build free software, to maintain and improve it and to look for bugs. Ideally, they check one another's work in a peer review system similar to that found in science, or at least on the nonprofit Wikipedia, where motivated volunteers regularly add new information and fix others' mistakes.
This process, advocates say, ensures trustworthy computer code.
But since the Heartbleed flaw got through, causing fears -- as yet unproved -- of widespread damage, members of that world are questioning whether the system is working the way it should.
"This bug was introduced two years ago, and yet nobody took the time to notice it," said Steven M. Bellovin, a computer science professor at Columbia University. "Everybody's job is not anybody's job."
Once Heartbleed was revealed, nearly two weeks ago, companies raced to put patches in place to fix it. But security researchers say more than one million web servers could still be vulnerable to attack. Mandiant, a cyberattack response firm, said on Friday that it had found evidence that attackers used Heartbleed to breach a major corporation's computer system, although it was still assessing whether damage was done.
What makes Heartbleed so dangerous, security experts say, is the so-called OpenSSL code it compromised. That code is just one of many maintained by the open-source community. But it plays a critical role in making our computers and mobile devices safe to use.
OpenSSL code was developed by the OpenSSL Project, which has its roots in efforts in the 1990s to make the Internet safe from eavesdropping. "SSL" refers to "secure sockets layer," a kind of encryption. Those who use this code do not have to pay for it as long as they credit the OpenSSL Project.
Over time, OpenSSL code has been picked up by companies like Amazon, Facebook, Netflix and Yahoo and used to secure the websites of government agencies like the F.B.I. and Canada's tax agency. It is baked into Pentagon weapons systems, devices like Android smartphones, Cisco desktop phones and home Wi-Fi routers.
Companies and government agencies could have used proprietary schemes to secure their systems, but OpenSSL gave them a free and, at least in theory, more secure option.
Unlike proprietary software, which is built and maintained by only a few employees, open-source code like OpenSSL can be vetted by programmers the world over, advocates say.
"Given enough eyeballs, all bugs are shallow" is how Eric S. Raymond, one of the elders of the open-source movement, put it in his 1997 book, "The Cathedral & the Bazaar," a kind of manifesto for open-source philosophy.
In the case of Heartbleed, though, "there weren't any eyeballs," Mr. Raymond said in an interview this week.
Although any programmer may work on OpenSSL code, only a few regularly do, said Ben Laurie, a Google engineer based in Britain who donates time to OpenSSL on nights and weekends. This is a problem, he said, adding that the companies and government agencies that use OpenSSL code have benefited from it but give back little in return.
"OpenSSL is completely unfunded," Mr. Laurie said. "It's used by companies who make a lot of money, but almost none of the companies who use it contribute anything at all."
According to the project's website, OpenSSL has one full-time developer -- Dr. Stephen N. Henson, a British programmer -- and three so-called core volunteer programmers, including Mr. Laurie, in Europe.
Logged records on the OpenSSL site show that Dr. Henson vetted the code containing the Heartbleed bug after it was mistakenly included in a graduate student's code update on New Year's Eve 2011, and the bug was inadvertently included in an OpenSSL software release three months later.
Neither Dr. Henson nor the other two volunteers responded to requests for an interview.
But open-source coders hardly blame Dr. Henson, considering that the OpenSSL project has operated on a shoestring annual budget of $2,000 in donations -- most from individuals -- which is just enough for volunteers to cover their electric bills.
Five years ago, Steve Marquess, then a technology consultant for the Defense Department, was struck by the contradiction that OpenSSL was "ubiquitous," yet no one working on the code was making any money. When he met Dr. Henson, Mr. Marquess said, Dr. Henson was working on OpenSSL code full time and "starving."
So Mr. Marquess started the OpenSSL Software Foundation to help programmers like Dr. Henson make money by consulting for government agencies and companies that were using the code. It also takes in some minimal donations, he said.
Over the last five years, the foundation has never made more than $1 million in commercial contracting revenue a year. This does not go very far in paying for the programmers' work, Mr. Marquess said.
Most corporate OpenSSL users do not contribute money to the group, Mr. Marquess said. Google and Cisco say they contribute by encouraging their own engineers to look for bugs in the code while they are on the clock. The OpenSSL website shows that a Cisco engineer and several Google engineers have discovered bugs and created fixes over the years.
A Google engineer, Neel Mehta, discovered the Heartbleed bug earlier this month, and two other Google engineers came up with the fix.
Likewise, Microsoft and Facebook created the Internet Bug Bounty initiative, which pays engineers who responsibly disclose bugs in widely used systems like OpenSSL. The group paid Mr. Mehta $15,000 for his discovery -- a windfall he donated to the Freedom of the Press Foundation.
But open-source advocates say organizations that rely on the code should do more to help. "Open source is not magic fairy dust" that happens automatically, said Tim O'Reilly, an early advocate of open source and the founder of O'Reilly Media. "It happens because people work at it."
At the least, security experts say, companies and governments should pay for regular code audits, particularly when the security of their own products depends on the trustworthiness of the code.
"They should be taking more responsibility for everything they ship in their product," said Edward W. Felten, a professor of computer science at Princeton University.
Ten years ago, Mr. Laurie, then a freelancer, performed an audit of OpenSSL for the Defense Advanced Research Projects Agency, known as Darpa. It took an entire year. Today, Mr. Laurie said, volunteers simply do not have the time to run that kind of audit.
The problem, Mr. Raymond and other open-source advocates say, boils down to mismatched incentives. Mr. Raymond said firms don't maintain OpenSSL code because they don't profit directly from it, even though it is integrated into their products, and governments don't feel political pain when the code has problems.
With OpenSSL, by contrast, "for those that do work on this, there's no financial support, no salaries, no health insurance," Mr. Raymond said. "They either have to live like monks or work nights and weekends. That is a recipe for serious trouble down the road."
He and other elders of the open-source movement say they want to create a nonprofit group to solicit donations from governments and companies and on Kickstarter that will be used to pay for audits of OpenSSL and other crucial open-source projects.
There was some good news this week. Mr. Marquess said that after Heartbleed helped expose the OpenSSL project's meager resources, the group received $17,000 in donations, almost entirely from individuals outside the United States. The highest individual donation was $300; the lowest was 2 cents.
But there was a hitch, he said: "Unfortunately, the 2 cents were donated through PayPal, and PayPal took both."