[Prerequisite: You should first read Casey's introduction to HKP and Hockeypuck on his blog here.]
Anyone who has ever used Ubuntu, Debian, Launchpad, or apt-get has implicitly trusted a sophisticated public key distribution protocol called "HKP" or, HTTP Keyserver Protocol. Originally designed for encrypting and signing email, asymmetric key pairs are used to sign, encrypt, decrypt and check signatures of thousands of packages on almost any Linux system.
Many (most?) public key servers today, such as keyserver.ubuntu.com, use an open source package called SKS (synchronizing key server) to distribute public keys.
Within Gazzang's zTrustee product, we rely on HKP to exchange public keys between client's and server. In our first implementation, we simply used SKS as installed from the Ubuntu repositories. SKS worked well in some environments, but it didn't scale well to larger environments, where hundreds of thousands of clients running on cloud servers were exchanging public keys in an automated fashion.
Moreover, we envisioned a system where user and host public SSH keys and server public SSL certificates might be exchanged in the same fashion, using the same protocol. We considered trying to extend SKS to improve the scalability and feature set.
In the end, we decided a new HKP implementation, leveraging a modern, high performance NoSQL key-value store -- MongoDB -- and written in modern language -- The Go Programming Language -- would enable us to build a more efficient, type-safe, memory-safe, concurrent, garbage-collected, fast implementation of HKP. We could also extend the feature set with a nice user interface and natively support other public keys.
With the general ideas fleshed out, my esteemed colleague, Casey Marshall, got to work on Hockeypuck -- his implementation of HKP in Golang and MongoDB -- freely available under the GPL. All credit for the development of Hockeypuck up to this point goes entirely to Casey :-) That said, he's really quite interested in outside contributions and help at this point, so if you're proficient in Golang and looking to contribute to an awesome security project, here's your bogey!
We at Gazzang are hosting a reference Hockeypuck server at:
But you don't have to use our Hockeypuck server ... we're absolutely delighted that Hockeypuck has been accepted into Ubuntu's 13.04 (raring) distribution in Universe. It's as easy as:
$ sudo apt-get install hockeypuck
This server has successfully imported the world's current public key ring -- that's 4GB of OpenPGP public key information! Casey's still working on the synchronization, which is based on SKS's "recon protocol". Again, if you're into hard core polynomial math, can read and understand OCaml, and are interested in re-working that algorithm in Golang, get in touch with us :-)
We're really, really interested in your feedback at this point! You can file bugs against the project and packages here. We're also looking for your feature requests... How would you like to use a public key server? Would you find it useful to import your SSH server or host public keys from a key server? Would you find it useful to see "badges" by keys, indicating that key's level or trust? Or perhaps that a key has been "verified"? What about linking public keys to OpenID or OAuth logins? Or what about [insert your idea here!]...
Comments? Bring 'em on!
It will be a lovely week next week! Valentines Day is next Thursday, February 14th, of course. Make sure you have chocolate and beautiful flowers for your sweetheart.
Near and dear to my heart, I'm personally excited that Gazzang will be introduced as one of the newest card-carrying members of the Linux Foundation! I've been an individual member of the Foundation for years, and have attended nearly a dozen LF events. We're extremely, extremely proud to add Gazzang to LF's very impressive list of active corporate members. What excellent company! I feel that we at Gazzang are differentiating ourselves from our competitors with comprehensive offerings around big data security, enterprise class encryption, and innovative key management -- all built exclusively in and on top of Linux.
And in celebration of all this love, Gazzang's fabulous marketing department has created a special Valentine's Day card for Linux, on behalf of all the enterprises and consumers far and wide that are just head over heels in love with the Penguin :-) Enjoy!
If you attended my talk about Entropy at the OpenStack Summit in San Diego earlier this month, or you read my post and slides here on my blog, you noticed that I had a few suggestions as to how we might improve entropy in Linux cloud instances and virtual machines.
There's one very easy approach that you can handle entirely on your end, when launching an instance, if you use Ubuntu's cloud-init utility, which consumes the user-data field from the metadata service.
Cloud-init supports a directive called write_files. Here, you can specify a path, ownerships, permissions, encoding, and content of a given file, which cloud-init will write a boot time. Leveraging, this, you can simply "create" (actually, just append to) the psuedo random device that the Linux kernel provides at /dev/urandom, with is owned by root:root and permissioned rw-rw-rw-. The stanza should look like this:
write_files: - encoding: b64 content: $SEED owner: root:root path: /dev/urandom perms: '0666'
Now, you'll need to generate this using a script on your end, and populate the $SEED variable. To do that, simply use this on your host system where you launch your cloud instance:
SEED="$(head -c 512 /dev/urandom | base64 -w 0)"
This command will read 512 bytes from your locale system's /dev/urandom and base64 encode it without wrapping lines. You could, alternatively, read from your local system's /dev/random if you have enough time and entropy.
Using the recipe above, you can ensure that your instance has at least some bit (actually, 4096 bits) of randomness that was collected outside of your cloud provider's environment.
I gave two presentations today at the OpenStack Design Summit in sunny San Diego, CA, as we prepare for the Grizzly development cycle.
In this presentation, I spent about 40 minutes discussing several research papers over the last 6 years showing the problems with entropy and randomness in cloud computing. Namely:
There's two pieces of the entropy problem in OpenStack and cloud computing that I'm interested in helping improve:
To the first point (better seeds), I suggested a series of technologies that could significantly improve the situation in OpenStack in the near term:
As for improving the ongoing entropy gathering...
Here are the slides from the first of my two presentations on behalf of Gazzang today at the OpenStack Summit in San Diego, CA.
In this presentation, we started out examining the landscape of security within cloud computing, what's changed, and why encryption is essential to the integrity of the industry. We talked about the types of sensitive data that need protection, and looked at some astounding statistics about data breaches in the past 3 years that could have been easily thwarted with comprehensive encryption.
We then discussed the several layers where encryption is essential:
Within the network layer, SSH seems to be well understood and practiced, though we did talk a little about some tools that can help with SSH public key management, namely ssh-copy-id, ssh-import-id, and DNS records for SSHFP. We also talked about TLS and SSL, the fact that many applications and services support SSL, but that it's rarely configured or enabled (even in OpenStack!). PKI and key management tend to be the hard part here...
At the block storage layer, we discussed dmcrypt, and how it can be used to efficient protect entire block devices. We discussed several places within OpenStack where dmcrypt could and should be used. We also discussed some of the shortcomings or limitations of dmcrypt (single key for the entire volume, hard to incrementally backup, all-or-nothing encryption).
I then introduced overlayroot to the OpenStack crowd, as convenient way of all local changes and data within an OpenStack guest.
At the filesystem layer, we discussed per-file encryption with eCryptfs, as well as Gazzang's commercially supported distribution of eCryptfs called zNcrypt. We compared and contrasted eCryptfs with dmcrypt, and discussed the most appropriate places to use each within OpenStack.
Finally we touched on the piece of technology required to bring all of this together -- key management. To actually secure any encrypted data, you need to safely and securely store the keys necessary to access the data somewhere other than on the same systems having the encrypted data. We talked a little about OpenStack Keystone, and how it does and doesn't solve this problem. We also introduced Gazzang zTrustee, which is our commercially supported key management solution that Gazzang offers as a turnkey solution in this space.Enjoy!
So... What's the actual threat model? Why are you insisting that people encrypt their data in the cloud? Where's the risk? When might unencrypted data get compromised? Who is accessing that data?
Wall of Shame' exposes 21M medical record breaches New rules under the Health Information Technology for Economic and Clinical Health Act, By Lucas Mearian, August 7, 2012 06:00 AM ET
Here are a few absolutely astounding numbers from that article, which were pulled from the US Department of Health and Human Services Health Information Privacy website.
Since the data is publicly available, I was able to import the numbers into a spreadsheet and verify ComputerWorld's article. I can confirm that the Mr. Mearian's numbers are quite accurate and quite scary. Since September 2009:
The top 6 breaches all affected more than 1 million individuals:
Such breaches are very costly, too.
Running a few more reports on the public CSV data,
The most disappointing part, to me, is that 72% of those breaches stem from theft, lost devices, and improper disposal -- a total of 15.6 million individual's health records. This means that the vast majority of these compromises were easily preventable, through the use of comprehensive data encryption. And I'd argue that many of the remaining 28% of the breaches attributed to hacking, unauthorized access, and other disclosures could also be thwarted, slowed, or deterred by coupling encryption with advanced key management, access controls, and regular auditing.
So here are a few tips, that I've been espousing for several years now:
There are a wide range of tools available, from free/open source, to paid commercial offerings. On the free/open source side, I'm a proponent, author, and maintainer of both eCryptfs and overlayroot (which uses dmcrypt). These can help protect your home directory and your private data in cloud instances.
Gazzang, sells an enterprise-class encryption product called zNcrypt, and I've architected Gazzang's cloud-compatible key management system, zTrustee. I have no doubt that the combination of these two technologies -- comprehensive data encryption and a robust key management solution -- could have prevented the compromise of millions of these records.
Key management in cloud computing presents a brand new, unique, and distinct set of challenges which are in many cases disparate from the traditional set of key management problems system administrators have been dealing with for decades in physical data centers. In fact, this very topic, in conjunction with data security and privacy, is the subject of two presentations I’m giving in the next 30 days at:
How are you managing your most sensitive information stored in the Cloud? Are you encrypting that data? Where are you storing your cryptographic keys and certificates? And who has access to them? If you have a stake in your organization's security, these questions may be keeping you up at night.
Cloud storage and Big Data present significant opportunities for enterprises, but those opportunities bring several huge challenges. In this session, we’ll explore:
I hope you’ll join me for one or both of these talks!
It's hard to believe that it's been almost 7 years since many of you helped conceive, design, implement, develop extend, test, document eCryptfs back in the IBM Linux Technology Center's Security team, with many more contributions following from Canonical, Red Hat, Intel, etc.
Over those 7 years, we have helped many of the rest of you reading this secure your private data. Today, there are actually millions of eCryptfs users -- through Ubuntu (and other Linux distros) encrypted home directories, Google's ChromeOS, Synology and Seagate encrypted NAS devices, some Android phones, Gazzang's cloud and big data encryption products, among others.
With the 100th release of the user space eCryptfs utilities (ecryptfs-utils), we'd like to take the opportunity to invite any contributors or users of eCryptfs for a celebration!
Gazzang is providing drinks and snacks on its rooftop (and air-conditioned lobby) in downtown Austin, at 8th and Congress, with great views of the Texas state capitol and all of downtown. Please join us at 804 Congress Avenue, Suite 400, from 5pm-8pm on Thursday, August 2, 2012. This is the day before the Texas Linux Fest in San Antonio, so perhaps we might see a few out of town guests join us... I happen to know at least one guest of honor who will be in from out of town for this event ;-)
Hopefully you'll join some of your colleagues around the Linux, open source, security, encryption, key management and data privacy space for an informal get together here in central Texas!
Gazzang has built an interesting business around Transparent Data Encryption, building on top of eCryptfs, adding some mandatory access controls and policy management in a product we call zNcrypt.
With the addition of zTrustee to the Gazzang product portfolio, we have entered an even more interesting, ambitious, new space in modern Cloud computing -- Opaque Object Storage.
eCryptfs and zNcrypt provide "transparent" data encryption, in that when the encrypted filesystem is mounted, allowed users, applications and processes (possessing the appropriate keys) are allowed to seamlessly read and write data as regular files and directories to the mounted storage filesystem. No user, application, or process needs to know anything about encryption -- they simply read and write data "transparently" from and to files and directories. Input/output operations are trapped in the Linux filesystem layer, and eCryptfs handles encrypting and decrypting files as necessary. Assuming you have safeguarded your keys appropriately, an offline attacker with physical or remote access to the disk would not have access to mounted filesystem and instead only see the cryptographically protected data.
zTrustee was designed from the ground up to store and serve the keys necessary to make eCryptfs and zNcrypt work properly. But we implemented it in a manner in which we can store and serve keys, certificates, files, directories, and data of any type -- similar to some object storage systems. However, we added our considerable security expertise to our implementation, and use encryption yet again to our customer's advantage. Each of these objects stored in zTrustee are actually encrypted and signed with the public GPG keys of the client storing and/or retrieving the data. This means that even an administrative user with full root access on the zTrustee server will not have introspection into the contents of the data blobs stored as deposits on the zTrustee server. For this reason, we're calling these deposits "Opaque Objects", and noting that our zTrustee server provides "Opaque Object Storage".
Moreover, the fine-grained security policies that govern the release of these deposits further differentiate zTrustee from various other object storage products. Beyond the individual encryption of each zTrustee deposit (object), the policy by which an object is released can: