Operational and Security Tips, Tricks and Best Practices
In Gartner’s Top 10 Strategic Technology Trends for 2016: Adaptive Security Architecture, they argued that
“Security must be more tightly integrated into the DevOps process to deliver a DevSecOps process that builds in security from the earliest stages of application design.”
We ultimately need to move to this model if we are going to be successful and continue to reduce the dwell time of cyber criminals who are intent on compromising our applications and data. But how do we get from this:
To this?
Easier said than done. To answer this question, I sat down with our CISO and IANS Faculty Member George Gerchow about what it means to implement and maintain a DevSecOps approach in the cloud – and what operational and security best practices should organizations follow to ensure success in their move to the cloud.
Below is a transcript of the conversation.
DevSecOps seems like a buzz word that everyone is using these days. What does DevSecOps really mean?
George: It is really about baking security in from Day 1. When you’re starting to put new workloads in the cloud or have these green field opportunities identified, start changing your habits and your behavior to incorporate security in from the very beginning. In the past we used to have a hard shell soft center type approach to security and in the cloud there is no hardshell, and we don’t run as many internal applications anymore. Now we’re releasing these things out into the wild into a hostile environment so you gotta be secure since day 1. Your developers and engineers, you have to have people who think security first when they’re developing code, that is most important take away”
What does it really mean when you say baking security in….or the term shifting left, which I am starting to hear our there?
George: It is about moving security earlier into the conversation, earlier into the software development lifecycle. You need to get developers to do security training. I’m talking about code review, short sprints, understanding what libraries are safe to use, and setting up feature flags that will check code in one piece at a time. The notion of a full release is a thing of the past – individual components are released continually. There also needs to be a QA mindset of testing the code and micro services to break it and then, fix accordingly through your agile DevSecOps methodologies.
Sumo Logic is a cloud native service running in AWS for over 7 years now – why did you decide to build your service in the cloud? Can you describe a bit about that journey, what was it like, what obstacles did you face, how did you overcome them? And lastly, what did you learn along the way?
George: Our company founders came from HP Arcsight and new full well of the pain in managing the execution environment – the hardware and software provisioning, the large teams needed, the protracted time to roll out new services. The cloud enabled us to be agile, flexible, highly elastic, and do this all securely at scale – it is at a level that was just not possible if we chose an on-prem model. The simplicity and automation capabilities of AWS was hugely attractive. You start setting up load balancers to be able to leverage tools like Chef to be able to do manage machine patching – it gets easier – and then you can start automating things from the very beginning so I think it’s that idea of starting very simple and leveraging native services that cloud service providers give you you and then looking for the gaps. The challenge initially was that this is a whole new world out and then the bigger challenge became getting people to buy off on the fact that cloud is more secure. People just weren’t there yet.
What does Sumo Logic’s footprint look like in AWS?
George: 100PB+ of data that is analyzed daily, 10K EC2 instances on any given day, 10M keys under management! We have over 1,300 customers and our service is growing by leaps and bounds. At this stage, it is all about logos – you wanna bring people in and you can’t afford to have bad customer service because this is a subscription-based model. When you think about the scale that we have, it’s also the scale that we have to protect our data. Now the challenge of quadrupling that number every year is extremely difficult so you have a long term view when it comes to scalability of security. 10,000+ instances it’s a very elastic type environment and auditors really struggle with this. One of the things that i’m the most proud of…if you look at hundreds of petabytes processed and analyzed daily, thats insane…thats the value of being in the cloud.
10 million of keys under management…thats huge…really??
George: It’s a very unique way that we do encryption. It makes our customers very comfortable with the dual control models…that they have some ownership over the keys and then we have capability to vault the keys for them. We do rotate the keys every 24 hours. The customers end up with 730 unique key rings on their keys at the end of the year. It’s a very slick, manageable program. We do put it into a vault and that vault is encrypted with key encryption key (KEK).
So What Tools and Technologies are you using in AWS?
George: Elastic load balancers are at the heart of what we do…we set up those load balancers to make sure that nothing that’s threatening gets through…so that’s our first layer of defense and then use security groups and we use firewalls to be able to route traffic to the right places to make sure only users can access. We use file integrity monitoring and we happen to use host sec for that across every host and we manage and that gives us extreme visibility. We also leverage IDS and snort and those signatures across those boxes to detect any kind of signature based attacks. Everything we do in the cloud is agentless or on the host. When you’re baking security in you have it on ALL of your systems, spun up automatically via scripts. We also have a great partnership with Crowdstrike, where threat intelligence is baked into our platform to identify malicious indicators of compromise and match that automatically to our customers logs data – very powerful
So how are you leveraging Sumo to secure your own service? Can you share some of the tips, tricks and best practices you have gleaned over the years?
George: Leveraging apps like CloudTrail, now we are able to see when a event takes place who is the person behind the event, and start looking for the impact of the event. I’m constantly looking for authorization type events (looking at Sumo Dashboards). When it comes to compliance I have to gather evidence of who is in the security groups. Sumo is definitely in the center of everything that we do. We have some applications built also for PCI and some other things as well to VPC flow logs but it gives us extreme visibility. We have dashboards that we have built internally to manage the logs and data sources. It is extremely valuable once you start correlating patterns of behavior and unique forms of attack patterns across the environment. You need to be able to identify how does that change that you just made impact the network traffic and latency in my environment and pulling in things like AWS inspector…How did that change that you made have an impact on my compliance and security posture. You want to have the visibility but then measure the level of impact when someone does make a change and even more proactively I want to have the visibility when something new is added to the environment or when something is deleted from the environment. Natively in AWS, it is hard to track these things”
How does the Sumo Logic technology stack you talked about earlier help you with Compliance?
George: Being able to do evidence gathering and prove that you’re protecting data is difficult. We’re protecting cardholder data, healthcare data and a host of other PII from the customers we serve across dozens of industries. We pursue our own security attestations like PCI, CSA Star, ISO 27001, SOC 2 Type 2, and more. We do not live vicariously through the security attestations of AWS like too many organizations do. Also, encryption across the board. All of these controls and attestations give people a level of confidence that we are doing the right things to protect their data and there’s actual evidence gathering going on. Specifically with respect to PCI, we leverage Sumo Logic PCI apps for evidence gathering -nonstop- across CloudTrail, Windows and Linus servers. We built out those apps for internal use, but released them to the public at RSA.
There are a lot of threat actors out there, from Cyber Criminals, Corporate Spies, Hacktivists and Nation States. How do you see the threat landscape changing wrt the cloud. Is the risk greater given the massive scale of the attack surface? If someone hacked into an account, could they cause more damage by pointing their attack at Amazon, from within the service, possibly affecting millions of customers?
George: It all starts with password hygiene. People sacrifice security for convenience. It’s a great time for us to start leveraging single sign on and multi factor authentication and all these different things that need to be involved but at a minimum end users should use heavily encrypted passwords…they should not bring in their personal type application passwords into the business world…If you start using basic password hygiene since day 1, you’re gonna follow the best habits in the business world. The people who should be the most responsible are not…I look at admins and developers in this way…all the sudden you have a developer put their full blown credentials into a slack channel.
So when you look out toward the future, wrt the DevSecOps movement, the phenomenal growth of cloud providers like AWS and Azure, Machine learning and Artificial Intelligence, the rise of security as code, ….What are your thoughts, where do you see things going, and how should companies respond?
George: First off, for the organizations that aren’t moving out to the cloud, at one point or the other, you’re gonna find yourself irrelevant or out of business. Secondly, you’re going to find that that the cloud is very secure. You can do a lot using cloud-based security if you bake security in since day one and work with your developers…if you work with your team…. you can be very secure. The future will hold a lot of cloud-based attacks. User behavior analytics…I can’t no longer go through this world of security and have hard-coded rules and certain things that I’m constantly looking for with all these false positives. I have to be able to leverage machine learning algorithms to consume and crunch through that data. The world is getting more cloudy more workloads moving into the cloud, teams will be coming together…security will be getting more backed in into the process.
How would you summarize everything?
George: “You’re developing things, you wanna make sure you have the right hygiene and security built into it and you have visibility into that and that allows you to scale as things get more complex where things actually become more complex is when you start adding more humans into it and you have less trust but if you have that scalability and visibility from day one and a simplistic approach, it’s going to do a lot of good for you. Visibility allows you to make quick decisions and it allows you to automate the right things and ultimately you need to have visibility because it allows you to have the evidence that you need to be compliant to help people feel comfortable that you’re protecting your data in the right way.
George Gerchow can be reached at https://www.linkedin.com/in/georgegerchow or @georgegerchow
