[Prerequisite: You should first read Casey's introduction to HKP and Hockeypuck on his blog here.]

Anyone who has ever used Ubuntu, Debian, Launchpad, or apt-get has implicitly trusted a sophisticated public key distribution protocol called "HKP" or, HTTP Keyserver Protocol. Originally designed for encrypting and signing email, asymmetric key pairs are used to sign, encrypt, decrypt and check signatures of thousands of packages on almost any Linux system.

Many (most?) public key servers today, such as keyserver.ubuntu.com, use an open source package called SKS (synchronizing key server) to distribute public keys.

Within Gazzang's zTrustee product, we rely on HKP to exchange public keys between client's and server. In our first implementation, we simply used SKS as installed from the Ubuntu repositories. SKS worked well in some environments, but it didn't scale well to larger environments, where hundreds of thousands of clients running on cloud servers were exchanging public keys in an automated fashion.

Moreover, we envisioned a system where user and host public SSH keys and server public SSL certificates might be exchanged in the same fashion, using the same protocol. We considered trying to extend SKS to improve the scalability and feature set.

In the end, we decided a new HKP implementation, leveraging a modern, high performance NoSQL key-value store -- MongoDB -- and written in modern language -- The Go Programming Language -- would enable us to build a more efficient, type-safe, memory-safe, concurrent, garbage-collected, fast implementation of HKP. We could also extend the feature set with a nice user interface and natively support other public keys.

With the general ideas fleshed out, my esteemed colleague, Casey Marshall, got to work on Hockeypuck -- his implementation of HKP in Golang and MongoDB -- freely available under the GPL. All credit for the development of Hockeypuck up to this point goes entirely to Casey :-) That said, he's really quite interested in outside contributions and help at this point, so if you're proficient in Golang and looking to contribute to an awesome security project, here's your bogey!

We at Gazzang are hosting a reference Hockeypuck server at:

• http://hockeypuck.gazzang.net
• https://hockeypuck.gazzang.net/

But you don't have to use our Hockeypuck server ... we're absolutely delighted that Hockeypuck has been accepted into Ubuntu's 13.04 (raring) distribution in Universe. It's as easy as:

$ sudo apt-get install hockeypuck

in Ubuntu 13.04 to get your Hockeypuck server up and running. For other Ubuntu releases, Casey is publishing backports to a stable and an unstable PPA.

This server has successfully imported the world's current public key ring -- that's 4GB of OpenPGP public key information! Casey's still working on the synchronization, which is based on SKS's "recon protocol". Again, if you're into hard core polynomial math, can read and understand OCaml, and are interested in re-working that algorithm in Golang, get in touch with us :-)

We're really, really interested in your feedback at this point! You can file bugs against the project and packages here. We're also looking for your feature requests... How would you like to use a public key server? Would you find it useful to import your SSH server or host public keys from a key server? Would you find it useful to see "badges" by keys, indicating that key's level or trust? Or perhaps that a key has been "verified"? What about linking public keys to OpenID or OAuth logins? Or what about [insert your idea here!]...

Comments? Bring 'em on!

Cheers,
:-Dustin

It will be a lovely week next week! Valentines Day is next Thursday, February 14th, of course. Make sure you have chocolate and beautiful flowers for your sweetheart.

Near and dear to my heart, I'm personally excited that Gazzang will be introduced as one of the newest card-carrying members of the Linux Foundation! I've been an individual member of the Foundation for years, and have attended nearly a dozen LF events. We're extremely, extremely proud to add Gazzang to LF's very impressive list of active corporate members. What excellent company! I feel that we at Gazzang are differentiating ourselves from our competitors with comprehensive offerings around big data security, enterprise class encryption, and innovative key management -- all built exclusively in and on top of Linux.

And in celebration of all this love, Gazzang's fabulous marketing department has created a special Valentine's Day card for Linux, on behalf of all the enterprises and consumers far and wide that are just head over heels in love with the Penguin :-) Enjoy!

If you attended my talk about Entropy at the OpenStack Summit in San Diego earlier this month, or you read my post and slides here on my blog, you noticed that I had a few suggestions as to how we might improve entropy in Linux cloud instances and virtual machines.

There's one very easy approach that you can handle entirely on your end, when launching an instance, if you use Ubuntu's cloud-init utility, which consumes the user-data field from the metadata service.

You simply need to use ec2-run-instances or euca-run-instances with the --user-data-file option.

Cloud-init supports a directive called write_files.  Here, you can specify a path, ownerships, permissions, encoding, and content of a given file, which cloud-init will write a boot time.  Leveraging, this, you can simply "create" (actually, just append to) the psuedo random device that the Linux kernel provides at /dev/urandom, with is owned by root:root and permissioned rw-rw-rw-.  The stanza should look like this:

write_files:
-   encoding: b64
    content: $SEED
    owner: root:root
    path: /dev/urandom
    perms: '0666'

Now, you'll need to generate this using a script on your end, and populate the $SEED variable.  To do that, simply use this on your host system where you launch your cloud instance:

SEED="$(head -c 512 /dev/urandom | base64 -w 0)"

This command will read 512 bytes from your locale system's /dev/urandom and base64 encode it without wrapping lines.  You could, alternatively, read from your local system's /dev/random if you have enough time and entropy.

Using the recipe above, you can ensure that your instance has at least some bit (actually, 4096 bits) of randomness that was collected outside of your cloud provider's environment.

I'm representing Gazzang this week at the Ubuntu Developer Summit this week in Copenhagen, Denmark pushing for better security, entropy, and key management inside of Ubuntu's cloud images.

Cheers,
:-Dustin

I gave two presentations today at the OpenStack Design Summit in sunny San Diego, CA, as we prepare for the Grizzly development cycle.

In this presentation, I spent about 40 minutes discussing several research papers over the last 6 years showing the problems with entropy and randomness in cloud computing.  Namely:

1. The Analysis of the Linux Random Number Generator (2006)
2. The iSEC Partners Presentation at BlackHat (2009)
3. Minding your P's and Q's (2012)

There's two pieces of the entropy problem in OpenStack and cloud computing that I'm interested in helping improve:

1. Better initial seeds for the psuedo random number generator at instance initialization
2. Better ongoing entropy gathering throughout the lifetime of the instance.

To the first point (better seeds), I suggested a series of technologies that could significantly improve the situation in OpenStack in the near term:

1. The hypervisor could provide a random seed through a block device to the guest
2. The hypervisor could expose a urandom device through the metadata service
• Actually, I'm sitting next to Scott Moser right now, who attended my talk earlier today and merely hours after my talk, he has already hacked this into the OpenStack metadata service :-)  His merge proposal is here.  This is why I love open source software...
3. The user can pass their own locally generated seed to the instance through cloud-init and the userdata
4. Additional seed data can be assembled through the aNerd protocol
• There's lots more to say about this one...I'll have another post on this soon!

As for improving the ongoing entropy gathering...

1. Eventually, a new wave of cloud servers with modern CPUs will have Intel's DRNG feature and leverage the new rdrand instruction
• Unfortunately, we're probably a little ways off from that being widely available
• Colin King has benchmarked it -- really impressive performance!
2. KVM's new virtio-rng driver is pretty cool too, allowing a server to pass through access to a hardware random number generator
3. HAVEGE simply rocks, and should be installed in every cloud instance
4. Gazzang's zTrustee encryption key manager also supports a secure, authenticated entropy service (as a commercial offering from my employer)

Enjoy!

Here are the slides from the first of my two presentations on behalf of Gazzang today at the OpenStack Summit in San Diego, CA.

In this presentation, we started out examining the landscape of security within cloud computing, what's changed, and why encryption is essential to the integrity of the industry.  We talked about the types of sensitive data that need protection, and looked at some astounding statistics about data breaches in the past 3 years that could have been easily thwarted with comprehensive encryption.

We then discussed the several layers where encryption is essential:

1. Network layer
2. Filesystem layer
3. Block storage layer

Within the network layer, SSH seems to be well understood and practiced, though we did talk a little about some tools that can help with SSH public key management, namely ssh-copy-id, ssh-import-id, and DNS records for SSHFP.  We also talked about TLS and SSL, the fact that many applications and services support SSL, but that it's rarely configured or enabled (even in OpenStack!).  PKI and key management tend to be the hard part here...

At the block storage layer, we discussed dmcrypt, and how it can be used to efficient protect entire block devices.  We discussed several places within OpenStack where dmcrypt could and should be used.  We also discussed some of the shortcomings or limitations of dmcrypt (single key for the entire volume, hard to incrementally backup, all-or-nothing encryption).

I then introduced overlayroot to the OpenStack crowd, as convenient way of all local changes and data within an OpenStack guest.

At the filesystem layer, we discussed per-file encryption with eCryptfs, as well as Gazzang's commercially supported distribution of eCryptfs called zNcrypt.  We compared and contrasted eCryptfs with dmcrypt, and discussed the most appropriate places to use each within OpenStack.

Finally we touched on the piece of technology required to bring all of this together -- key management. To actually secure any encrypted data, you need to safely and securely store the keys necessary to access the data somewhere other than on the same systems having the encrypted data.  We talked a little about OpenStack Keystone, and how it does and doesn't solve this problem.  We also introduced Gazzang zTrustee, which is our commercially supported key management solution that Gazzang offers as a turnkey solution in this space.

The Endeavor Space Shuttle passed over the Gazzang world headquarters in Austin this morning. The Statesman has the full article. This image is pretty spectacular without my grafitti (but I couldn't resist)!

At last weekend's Texas Linux Fest, at the end of my presentation, Data Security and Privacy in the Cloud, an attendee asked a great question. I'll paraphrase...

So... What's the actual threat model? Why are you insisting that people encrypt their data in the cloud? Where's the risk? When might unencrypted data get compromised? Who is accessing that data?

A couple of weeks ago, an article from ComputerWorld made the front page of Slashdot:

Wall of Shame' exposes 21M medical record breaches New rules under the Health Information Technology for Economic and Clinical Health Act, By Lucas Mearian, August 7, 2012 06:00 AM ET

Here are a few absolutely astounding numbers from that article, which were pulled from the US Department of Health and Human Services Health Information Privacy website.

Since the data is publicly available, I was able to import the numbers into a spreadsheet and verify ComputerWorld's article.  I can confirm that the Mr. Mearian's numbers are quite accurate and quite scary.  Since September 2009:

• 21+ million people have had their health care records exposed
• 480 breaches have been reported

The top 6 breaches all affected more than 1 million individuals:

• 4.9 million records: TRICARE Management Activity, the US Department of Defense's health care program, exposed 4.9 million health care records when backup tapes went missing
• 1.9 million records: Health Net lost 1.9 million records when backup hard drives went missing
• 1.7 million records: New York City Health & Hospital's Corporation's North Bronx Health Care Network reported the theft of 1.7 million records
• 1.22 million records: AvMed Health Plans reported the loss of a laptop with 1.22 million patient records
• 1.02 million records: Blue Cross Blue Shield of Tennessee exposed 1.02 million records with the loss of an external hard drive
• 1.05 million records: Nemours Foundation (runs children's hospitals) lost 1.05 million records with missing backup tapes

Such breaches are very costly, too.

• $4.3 million: Cignet Health of Prince George's County civil lawsuit penalty
• $1.5 million: Blue Cross Blue Shield of Tennessee penalties
  • have since encrypted all of their hard drives, 885TB of data
• $1.7 million: Alaska Department of Health penalty
  • due to theft of a thumb drive, stolen from an employee's car

Running a few more reports on the public CSV data,

• Across 480 reported breaches, these were the top reasons given for the incident:
  • 55%: Theft of devices or physical media
  • 26%: Hacking/Unauthorized access
  • 12%: Lost devices, disks, tapes, drives, media
  • 5%: Improper disposal of devices
  • 3%: Other

The most disappointing part, to me, is that 72% of those breaches stem from theft, lost devices, and improper disposal -- a total of 15.6 million individual's health records.  This means that the vast majority of these compromises were easily preventable, through the use of comprehensive data encryption.  And I'd argue that many of the remaining 28% of the breaches attributed to hacking, unauthorized access, and other disclosures could also be thwarted, slowed, or deterred by coupling encryption with advanced key management, access controls, and regular auditing.

So here are a few tips, that I've been espousing for several years now:

1. Encrypt your data.
2. Help your colleagues, friends, and families encrypt their data.
3. Insist that your employers institute thorough security policies around encryption.
4. Ask hard questions of your health care providers and financial services professionals, about the privacy of the data of yours they have. Hold them accountable.

There are a wide range of tools available, from free/open source, to paid commercial offerings.  On the free/open source side, I'm a proponent, author, and maintainer of both eCryptfs and overlayroot (which uses dmcrypt).  These can help protect your home directory and your private data in cloud instances.

Gazzang, sells an enterprise-class encryption product called zNcrypt, and I've architected Gazzang's cloud-compatible key management system, zTrustee. I have no doubt that the combination of these two technologies -- comprehensive data encryption and a robust key management solution -- could have prevented the compromise of millions of these records.

Key management in cloud computing presents a brand new, unique, and distinct set of challenges which are in many cases disparate from the traditional set of key management problems system administrators have been dealing with for decades in physical data centers. In fact, this very topic, in conjunction with data security and privacy, is the subject of two presentations I’m giving in the next 30 days at:

• Texas Linux Fest -- San Antonio, TX -- August 4, 2012
• Linux Foundation’s CloudOpen -- San Diego, CA -- Wednesday August 29, 2012

How are you managing your most sensitive information stored in the Cloud? Are you encrypting that data? Where are you storing your cryptographic keys and certificates? And who has access to them? If you have a stake in your organization's security, these questions may be keeping you up at night.

Cloud storage and Big Data present significant opportunities for enterprises, but those opportunities bring several huge challenges. In this session, we’ll explore:

• What's not secure, not acceptable, not working --- but totally pervasive!
• Where encryption makes the most sense around Cloud and Big Data applications
• Key sprawl in the cloud
• The strengths and weaknesses of various key management options
• Easing the pain - Recent innovations for managing keys and company secrets
• Real-world use cases – from web servers to encrypted file systems to big data to SSH to SSL

I hope you’ll join me for one or both of these talks!

:-Dustin

Howdy all!

It's hard to believe that it's been almost 7 years since many of you helped conceive, design, implement, develop extend, test, document eCryptfs back in the IBM Linux Technology Center's Security team, with many more contributions following from Canonical, Red Hat, Intel, etc.

Over those 7 years, we have helped many of the rest of you reading this secure your private data. Today, there are actually millions of eCryptfs users -- through Ubuntu (and other Linux distros) encrypted home directories, Google's ChromeOS, Synology and Seagate encrypted NAS devices, some Android phones, Gazzang's cloud and big data encryption products, among others.

With the 100th release of the user space eCryptfs utilities (ecryptfs-utils), we'd like to take the opportunity to invite any contributors or users of eCryptfs for a celebration!

Gazzang is providing drinks and snacks on its rooftop (and air-conditioned lobby) in downtown Austin, at 8th and Congress, with great views of the Texas state capitol and all of downtown. Please join us at 804 Congress Avenue, Suite 400, from 5pm-8pm on Thursday, August 2, 2012. This is the day before the Texas Linux Fest in San Antonio, so perhaps we might see a few out of town guests join us... I happen to know at least one guest of honor who will be in from out of town for this event ;-)

Hopefully you'll join some of your colleagues around the Linux, open source, security, encryption, key management and data privacy space for an informal get together here in central Texas!

Cheers,
Dustin Kirkland
eCryptfs co-maintainer

Gazzang has built an interesting business around Transparent Data Encryption, building on top of eCryptfs, adding some mandatory access controls and policy management in a product we call zNcrypt. 

With the addition of zTrustee to the Gazzang product portfolio, we have entered an even more interesting, ambitious, new space in modern Cloud computing -- Opaque Object Storage.

eCryptfs and zNcrypt provide "transparent" data encryption, in that when the encrypted filesystem is mounted, allowed users, applications and processes (possessing the appropriate keys) are allowed to seamlessly read and write data as regular files and directories to the mounted storage filesystem.  No user, application, or process needs to know anything about encryption -- they simply read and write data "transparently" from and to files and directories.  Input/output operations are trapped in the Linux filesystem layer, and eCryptfs handles encrypting and decrypting files as necessary.  Assuming you have safeguarded your keys appropriately, an offline attacker with physical or remote access to the disk would not have access to mounted filesystem and instead only see the cryptographically protected data.

zTrustee was designed from the ground up to store and serve the keys necessary to make eCryptfs and zNcrypt work properly.  But we implemented it in a manner in which we can store and serve keys, certificates, files, directories, and data of any type -- similar to some object storage systems.  However, we added our considerable security expertise to our implementation, and use encryption yet again to our customer's advantage.  Each of these objects stored in zTrustee are actually encrypted and signed with the public GPG keys of the client storing and/or retrieving the data.  This means that even an administrative user with full root access on the zTrustee server will not have introspection into the contents of the data blobs stored as deposits on the zTrustee server.  For this reason, we're calling these deposits "Opaque Objects", and noting that our zTrustee server provides "Opaque Object Storage". 

Moreover, the fine-grained security policies that govern the release of these deposits further differentiate zTrustee from various other object storage products.  Beyond the individual encryption of each zTrustee deposit (object), the policy by which an object is released can:

• limit the  (time to live) TTL
• limit the number of times it can be retrieved (e.g. Mission Impossible message)
• be disabled (and later enabled)
• be purged entirely
• required 0 - N  authenticate and "vote" or "sign off" on the release trustees
• be retrieved by an authenticated, signed/encrypted client, or
• be retrieved anonymously using a nonce URL