WebHostingTalk Database Breach, Credit Cards Stolen: Lessons Learned for Web Hosts Everywhere

A very wise man, George Washington, once said “If we don’t learn our history, we’re doomed to repeat it.”  This quote is certainly true in the security industry, as you must always be watching and learning – adapting as situational changes occur all around us.  It is essential to look at the mistakes of others and learn from them.

I would like to make clear that we are not interested in propagating rumors or beating this issue to death.  There are very serious issues that arose here that can be used as a learning experience for all of us in the web industry and it is vital that these lessons be brought out.

It is fair to say that this situation has shown the ideal way not to handle a data breach incident.  There have been numerous failures among many different individuals along the road, and some inexcusable negligence on the part of those involved.  This should be used as a learning experience, guiding all of our incident response plans to better our reaction to these issues in the future.  This example is exactly why we must always have these plans in place, refined, and practiced in case they are ever needed, as it is more a matter of when, not if, we will have to use them.

Background

Roughly two weeks ago, it was first made public that  the popular site WebHostingTalk.com was compromised from a very unique attack – by having their backup servers exploited and destroyed.  At the time, it was reported by the parent company, iNet, that absolutely no credit card information was involved in the breach, but close to 5,400 pages of account information (username, email address, and encrypted passwords) were stolen.

This week, however, we reported that there was credit card information stolen from this same database and it was posted on various BitTorrent sites, forums, and other places around the Internet with the complete table of unencrypted credit card information.  WebHostingBuzz’s COO Matt Russell was affected and notified users in our forums and on the blog as soon as he could.  We did a little additional investigation and found some very troubling information as to what data was taken and the type of data that was involved.

The data stolen included:

• Complete credit card numbers (totaling 9,561)
• CVV2 / CCV identification number
• Cardholder’s name
• Expiration date
• Bank of cardholder
• Credit cards that were “removed” by the user but not deleted from the database

As I reported yesterday, storing this unencrypted data violates the PCI DSS (Payment Card Industry Data Security Standard), and storing the CVV2 / CCV number – whether encrypted or not – is strictly forbidden in all circumstances.

What followed the initial breach was a disturbing series of missteps, misunderstandings, and underestimates on the part of WebHostingTalk and iNet, leading to even more problems and a Public Relations disaster that was equally mismanaged.

Overall, there are many lessons to learn from this situation – hopefully those of us in the web hosting industry never have to repeat them.

The Breach Itself

On March 21, WebHostingTalk’s backup servers were attacked by a presently unidentified hacker, who exploited their database system and exported a copy of the information in the database before completely wiping the entire hard drive of not one, but every single hard drive containing a backup of this information.  The attacker later compromised the main WHT server and was able to destroy most of the information there too.

An overview of the damages:

9,561 total credit cards and CVV2 / CCV codes were exposed publicly, breakdown as follows:

• 318 confirmed valid credit cards with CVV2/ CCV
• 1,800 confirmed expired credit cards with CVV2 / CCV
• 7,443 credit cards with CVV2 / CCV that are presently in unknown status

Over 5,400 pages of user names and password hashes were posted publicly.

Lesson 1

Do not ever retain credit card information in plain text. Ever. If possible, do not store credit card information, period: encrypted or otherwise.

Public Response

The WHT community has several moderators with the User Titles of “community liaisons” and “community leaders” who are not technically paid staff of iNet, but who are in a position of responsibility and can be perceived by their title to be employees of WHT.  This caused some confusion and anger on the part of those clients of WHT who were demanding answers from iNet and saw responses from moderators who did not address their questions. This caused some anger in several of the threads relating to the incident.

Lesson #2

It is always key when handling emergency response to have one clearly designated and official point of contact in a central location where all queries can go through. This way, inconsistencies are significantly reduced and communication is standardized. Other staff members must stay away from the discussion and allow the point of contact to answer all questions.

The information that was coming from the iNet point of contact was heavily based on assumptions and not technically justifiable facts. This lead to confusion, as the first report about the breach said:

“This only affected the post, thread and user tables. Only.”

Another report on the type of data stolen said that absolutely no credit card information had been stolen in this incident. Unfortunately, iNet seems to have gone back and modified or removed their announcements in the archives that stated this explicitly.

Lesson #3

Only report absolute and total facts to customers because what might make you look less-bad in the short term might make you look like a real idiot later. Be completely up front about what happened and only release what you know to be fact, not what you believe to be true.

In fact, as we would find out two weeks after the initial report, the breach included significantly more information than just the post, thread, and user tables. It, in fact, contained unencrypted credit card information, which was explicitly rejected earlier.

In several instances, there was some sarcasm and blatant rudeness on the part of the iNet staff toward paying customers. In my opinion, this is not acceptable behavior by any company.

Lesson #4

Sarcasm in Public Relations is a Bad Idea

Here is one example of the public interaction between customers and iNet. The question asked by WHT Client:
“Is this really a 24 hour job?”

Response from iNet:
“No. We decided to take our sweet time about it. After all, why would we want the forum on line?

Listen; if it could have been turned back on in 30 seconds, it would have been. There’s no reason for us to needlessly wait.”

This is clearly unacceptable – and this is only one example of places where the discussion crossed a respectful boundary between business and consumer and turned needlessly personal.

The handling of the public relations during the entire course of this incident was inconsistent and, at some points, down right unprofessional. This is never a good way to handle a public disaster such as this.

One thing that bothered me about the whole series of incidents was that WHT never sent an all-user email notifying all customers of the breach. Had Matt never shown me the thread, I would have never known anything about it – and my username and encrypted password is sitting on someone’s desktop somewhere ready to be attacked. I do not appreciate that – let alone being one of the paying customers whose credit card details were exposed. Several customers were reporting being on the list of credit cards stolen, but had not been contacted by iNet several hours after the incident.

Perhaps future incidents should use email instead of public forums to communicate information to the clients? Posting a public FAQ (which iNET did do) that is updated frequently is much more efficient than having users blasting questions away in a forum and then getting treated disrespectfully as a result of their questions.

The notification of the attack should have been immediate – whether or not there were user names, passwords, credit cards, or anything else. Customers have a right to know immediately if there is any possibility that their personally identifiable information was compromised.

Technical Response

The technical side of things I have to be less critical of because I am not privy to the information the techs had at what time during the events. I am not making the case that because WHT was hacked that their technicians are incompetent – no system is fully secure. I have been hacked. If you haven’t been hacked, you will one day be in their position.

But as an Information Assurance student, it is taught that we must always be prepared to respond immediately and precisely with a strategy that is effective. The response to the attack is where the lessons are learned from this perspective, so you can respond better in the future.

However, there are standard security practices that were not followed or ignored before, during, and after the attack that made the situation even more difficult on iNET.

Lesson #5

You must know where your information is at and know what your databases contain. Perform regular audits to ensure data remains where it should be, at the proper level of business classification and in compliance with regulations.

Let’s say you take your 3 year old child to a water park and instead of watching them, you lay down and read a book and your child later drowns without you knowing it (you laugh, but I’ve seen it happen all too often.)  This is the equivalent of what happened with information in the databases – there seems to have been no care as to where sensitive information was stored or how it was stored, allowing for the unencrypted credit card numbers to even be stored in the first place. Just like letting a 3 year old swim without you watching, this was unthinkable.

Lesson #6

If a system is compromised, assume all data was compromised, not just what you’ve found pieces of around the internet.

It was stated repeatedly in the initial phase that “Only certain tables were affected.” This statement was completely ignorant of all security practices. Just like if a virus penetrates a classified network, you treat that classified information as potentially released and you respond accordingly.

The fact that WHT did not know the CC numbers existed in the database is no excuse for the claim that only certain tables were affected. Private messages were a prime target for anyone who would have been looking for sensitive information – and of course, why wouldn’t a targeted attack look for something like this? Better yet, if the attacker took bits of several tables, wouldn’t it be safe to assume that the attacker took a dump of the entire database?

Lesson #7

Lock down your systems after they have been compromised.

The backup systems were initially compromised on March 22 and detected a short time thereafter. From the information that was gathered, the credit card information was taken from the database on March 25. Not only that, but the hacker could identify how many users had changed their passwords (just over 1,500) since the first notification.

This is also unacceptable. If you know you have been compromised, pull the network cable out until you get a handle on what is going on and who you are dealing with. If you don’t – continued access can lead to even more significant damage. It is more acceptable to be down for a few days than be repeatedly and consistently compromised over and over.

Conclusion

It is clear that WHT has made many embarrassing missteps over the past few weeks and their actions have hurt their customers and their image. The difference between a customer leaving a business and a customer forgiving a business is not whether or not the business got hacked, but it is the quality of response after they did get hacked.

This situation provides a learning experience for all of us in the web industry. We must take the lessons learned over the past few weeks and apply them when it is our turn to face this type of crisis.