My Biggest IT Mistake: Too Much Security

Security is like horsepower. If you don’t have enough, you lose credibility. But too much security, like having a 500 horsepower engine that gets six miles to the gallon, has a cost. This is a lesson I learned the hard way.

In the middle to late –’90s, I worked at a Web hosting company that ran Web servers on Windows NT 4.0 Server and, later, Windows 2000 Server. Customers needed to be able to log on to their servers to perform basic management tasks such as copying files and creating users. I didn’t want a customer to accidentally break a server, so I made a rule that customers couldn’t be members of the Administrators group.

That mistake probably cost my company hundreds of thousands of dollars.

Because customers didn’t have administrator access, they had to call the support center to make many common configuration changes, including simple tasks like adding a virtual directory. Each call cost my company in a couple of ways:

• We had to pay Tier 1 and Tier 2 support people to make these straightforward changes.
• Customers got frustrated with how long it took to make changes. Many of them cancelled their service, preferring to host their Web servers themselves.

I also learned an important lesson: Forcing overly restrictive security measures on legitimate users forces the users to circumvent those measures to get their jobs done. While auditing computers to see if hackers had gotten into our systems, I found evidence that they had. But the hackers weren’t outsiders, they were our customers. Our own customers had used hacking tools to elevate their privileges so they wouldn’t have to call our help desk to get their work done.

Years later, once I had more experience, I ate crow and decided to give customers administrative privileges. Most of the other engineers thought I was insane — after all, that would be less secure! They were right, but they weren’t seeing the big picture. While giving customers administrative privileges was a little less secure, it was much less costly. Sure, sometimes customers accidentally broke their own servers — but fixing those few mistakes took just a fraction of the time it took to make administrative changes on their behalf. Most important, the customers were happier.

Here are the lessons I learned:

• More security isn’t always better. The cost of reduced security is risk — something bad might happen in the future that would be costly. The cost of increased security is money and inconvenience. A wise engineer finds the sweet spot. Management might be impressed by the new fingerprint scanners, but are they worth the trouble?
• If security is inconvenient to users, they’ll figure out a way to bypass it. Lock out user accounts after three typos, and you’ll find users sharing their passwords so they can log in. Require users to change their passwords every seven days, and you’ll see sticky notes on every monitor with this week’s password.
• Responding to a security event can be cheaper than preventing it. Security is never perfect. Instead of spending hundreds of thousands of dollars trying to make it impossible for an attacker to hack your Web site, it might be cheaper to monitor it closely and restore it from a backup if you are successfully attacked.

Inexperienced engineers turn on every security feature, while experienced engineers weigh the costs and turn on only the security features they really need. If you don’t think the benefits of a security measure outweigh the costs, be prepared to support your argument. Arguing against security is counterintuitive — but remember, you have a limited amount of time and money, and your organization might be better served by directing your energy elsewhere.