SPOOKY STORYTIME!

It's Monday morning. You just got to work, and something seems. . . off. You notice the keystrokes first. The staccato beat is too fast and too hard. No one is making eye contact, and an aura of fear permeates the air.

Your lead engineer rushes over. “We started getting support calls this morning, our customer-facing site is taking over a minute to load the homepage.”

Not great news, but definitely fixable. You have a rock-solid engineering team, and this is just a small speed bump. A quick email to keep everyone in the loop, a few minutes for the engineers to get their head wrapped around the issue, and everything will be back to normal.

Ten minutes later you have a diagnosis. “It’s not the server. It appears our site is loading up hundreds of non-visible pop-ups that are executing malware in our customers’ browsers.”

Your coffee goes sour in your mouth – Your site is delivering malware into your customers’ browsers? The muscles in your neck and shoulders start to contract. That itching on your scalp – it’s your hair turning white.

Thirty minutes later, it’s become clear that someone has remotely changed the site’s database, appending malicious code to most of the data fields. Your site is now the victim of a SQL Injection attack. Site uptime monitors didn’t trigger because nothing ever went down. Over 10,000 visitors have been exposed to malware that your website delivered. Worst of all, the customer information in your system is presumed compromised. It’s your worst nightmare.

You have some very uncomfortable meetings ahead of you.

You could spend the time to shotgun regular expressions in PERL through millions of database rows, making sure all of the hostile JavaScript in the database are scrubbed. Or you could read on to learn how to maximize web application operations to amplify security posture.

What is a ‘secure’ website?

Everyone has their own opinion about what a secure website is, but opinions are tough to audit. Our aim is to create a system that demonstrates due diligence in web security to site stakeholders. The most important components to securing a site are roles, processes, and controls that log events for auditing.

“You can't manage what you can't measure” - Peter Drucker

The compliance standard referred to as SSAE 18 (sometimes called a “SOC” report) was built for this purpose. SSAE 18 is used to regulate how companies conduct business and defines how companies report on compliance controls. These reports are called SOC 1, SOC 2, and SOC 3. Complying with these standards forces you to document your roles and processes, and then build reporting mechanisms to keep information about those processes honest.

This post isn’t about becoming SSAE 18 compliant though, it’s about never having a data breach in the first place.

First, ask yourself these questions:

• Who has read and write access to your data? Can you prove it?
• What roles on your development team have access to releasing code into your application? Can you prove it?
• Do your systems each have a single responsibility? How do you audit access?

Your first instinct might be that you have most of it right, but do you have proof? After really digging-in, you might find out that you have ex-employees credentials in places you didn’t expect, loose compliance with who can release code to production, or a host of other issues you didn’t know about. Having the right rules in place is one thing, but it’s another to verify that they are being followed. All it takes is one ajax page with a bad query to make you and your customers the victim of a SQL injection attack.

To guard against a breach you have to build your development workflow to resist complacency and manual errors.

Your site’s security starts with your architecture.

To guard against a breach, development workflows need to be built to resist complacency and manual errors. Organization is key. Whenever I start a new project, I have a blueprint in mind of what I’m looking to establish for baseline operations. This architecture roadmap and list of features ensures the success of the final product and minimizes failure points.

Here are three things to keep in mind when making design decisions:

1. Develop for Maintenance. Change can be 100x more expensive once a product is in its maintenance phase vs. its initial planning and build phase. Design your functionality to be flexible, and make your product easy to change down the road.
2. Keep it Simple. Simple and easy to understand software architecture speeds up on-boarding and development at every stage. A new engineer should intuitively understand what you are doing and why. Otherwise, the next person to work on your project is going to stomp all over your elegant systems if they can’t understand them fairly fast.
3. Don’t trust anything. Build systems that expect failure and processes that remediate them.

Establish a workflow management process.

Start thinking about software development process as an assembly line. Build processes into development culture for a seamless transition as pieces get handed from one team to another. It’s easier to have clearly defined processes and workflows where people can see their progress and self-correct, than it is to spend your time manually directing individuals. Generally speaking, your engineers want to have a high level of autonomy. It’s easier to have clearly defined processes and workflows where engineers can see their progress and self-correct, than it is to spend your time babysitting, and manually directing individuals. Project ticketing systems like Jira, Trello, Asana, and ActiveCollab are all optimized for organizing workflow. And I shouldn’t have to mention this, but you should be using a code repo like Git or SVN too.

Now that you have auditable project logs, it’s time for some controls. How do we make sure that we aren’t pushing security flaws into production?

You should adopt a modern and widely used and supported framework for your application architecture. Imarc primarily implements Laravel, Craft CMS and a host of other frameworks on LAMP stack web applications. From a security standpoint, it doesn’t really matter which one you pick, as long as it’s a current version and still supported within the web industry. The greatest benefit of using a modern framework, from a management perspective, is that it forces code to be written by the same set of rules and standards. This gives you a platform to start writing best practices that are easy to understand and adopt.

What roles impact security in the pipeline?

The Systems Administrator, Release Engineer, and Code Reviewer are the gatekeepers for all of your processes.

Once a developer finishes their work on a code branch, a second set of eyes should review it before going live. That second set of eyes should be a designated Code Reviewer or someone who is well-versed in Web Security Principals. The Release Engineer should package and review the code branches, and define what is being pushed to production and when. If a release goes south, they should be competent enough to realize it fast and resolve the problem immediately. You should also have a comprehensive operating system and application security update policy that gets logged, reviewed, and executed at least once a month.

Hardware Architecture

Generally speaking, where servers are concerned, I tend to architect around my Disaster Recovery Plan. Most companies that house data and infrastructure have an official recovery plan for acceptable downtime outages, and procedures in the event of a major catastrophe. If you have a closet full of hardware, and there’s a building fire, is 2-7 days an acceptable downtime? If you’re responsible for a revenue-generating enterprise application, probably not. Create a Disaster Recovery Plan to make sure that you have some agility when it comes to quick restoration of services in the event of disaster.

From a security perspective, you should be familiar with the single responsibility principle for site production. In an enterprise web app, that generally means having one server for web applications and one server for databases, each with separate credentials.

If you have a lot of contractors working on a project, and enforcing code quality standards is a concern, you can also add another layer of abstraction to this mix. I recommend setting up an API server and configuring it to be the sole access point to the database server. Then, have the website access data via the API instead of a direct data connection. As long as you have all of the servers using local IP addresses, there shouldn’t be much of a performance hit. Additionally, the site will be immune to SQL injection attacks and have the added bonus of being headless.

The Recipe for Security

1. Break down development pipelines into an assembly line with formal handoffs between contributors.
2. Define Key Process Indicators (KPIs) in the workflow.
3. Set up formal source code change control processes that can be audited.
4. Set up formal roles in the pipeline, then restrict and log access to production systems based on those roles.
5. Set up a ticketing/workflow system that can run reports on all KPI’s.
6. Update (or create) the Disaster Recovery Plan, and make sure you can get back on your feet within 20-30 minutes, not days or weeks.
7. Make sure the systems architecture conforms to the Single Responsibility Principle. If there’s uncertainty in the development process, abstract access to the database through an API server.
8. Enjoy your next vacation knowing that you have a secure, transparent, self-managing development workflow that is unlikely to be compromised, and which gives your business stakeholders confidence in your leadership.