037. Security Mini Series – Proactive Cyber Security


Chris: 00:30

Welcome to EECO Asks Why, today we’re going to be talking about how you can be proactive in enhancing your facility’s position with cyber security. And to break this down for us, we have an expert Mr. Bill Metcalf, who was the director of information systems at Global Process Automation.

So how are you doing today, Bill?

Bill: 00:46

Hi, Chris. I’m doing great. Thanks for having me. 

Chris: 00:49

Oh, we’re very excited to have you, sir, looking forward to breaking this down for us, because this is a very hot topic, particularly as things have gone to more and more remote type work and to maybe to get us started, can you give us a little explanation of when we hear the term industrial control systems, cyber security?

What is that?

Bill: 01:08

So it’s cyber security, basically around industrial controls. And it’s a little different than a typical IT-based cyber security. If we think about industrial control systems or OT networks, typically we’re thinking about things that are on the manufacturing floor and these things interact with the physical world.

So in IT. A cyber security concern may be confidential information or bank account information. That type of thing. When we come to an ICS cybersecurity concerns, it’s the control of boilers. It’s the control of our safety systems, our environmental controls, that kind of thing. So again, the priority is a little bit different, when we start thinking about industrial control systems and cyber security, as opposed to typical IT type cybersecurity  engagement.

Chris: 02:03

Okay. That definitely helps, we often hear with, to get started from an industrial standpoint, security assessments, a good place. So what should industrial end users expect from a typical IT assessment?

Bill: 02:19

A typical assessment engagement comes down to several different things. We look at the cyber end of things, which courses, the networks, firewalls, we look at assets. What kind of systems, what is it that we’re actually trying to protect from? Typically we use standards to go by.

One of the ones, and there are lots of different standards. I triple unites the standard, Sans Institute has a standard. The one that’s the generic one, that is open and free to all is the NIST. Or it comes from the National Institute of Science and Technology. And basically it’s a set of guidelines that are developed in cooperation, with the federal government, with Homeland security and the industry owners, right? So the people who actually own this equipment collaborate together and they set out a set of standards and best practice that are followed.

So typically we’ll use that as a baseline. And, we’ll engage, maybe vote off of a series of workshops and talk about things like policies and procedures. It’s surprising as you dig into it. The little things that people don’t think about where they fall in line with the supply chain or, are there government standards that they’re required to follow?

Everybody knows about NERC CIP in the power industry. Those are federally mandated standards that critical infrastructure is required to follow.

Recently, water and wastewater has gone into the Clean Water Act, which now is putting requirements of water and wastewater providers, or I guess mainly water providers, on what they should be doing with security.

 The regular manufacturing things that are not considered critical infrastructure. These are guidelines. These are not requirements, but they’re recommendations. And we all know that we’ve seen on the news that things happen, viruses happen. There are people out there attacking systems, whether it’s to disrupt the manufacturing process.

So we’re seeing more and  more cyber extortion, if you will. So the bad guy is looking to gain money and they don’t really care whether it’s a soccer mom, a hospital or a manufacturing facility. Their main goal is to basically hold your system hostage and they use ransomware to do that. And so we’re seeing a lot more of that taking place.

A lot of times when I’ve talked to customers, they’re like, “You know I’m just a small manufacturing company, out in the middle of nowhere USA. Nobody even knows that I exist,” and we always refer to that as security through obscurity. What we’re finding is the techniques in ransomware and things like that are so widespread that you can no longer hide behind being a small company or, in some worry about location.

If you are connected to the internet, you have that possibility of becoming a victim.

Chris: 05:28

Security through obscurity. That’s a new one. So when I think through,  an assessment says, does this type of assessment from a cyber security standpoint differ very much from the actual work on the ground in a, just a standard network assessment that you would typically go in and do for an industrial on the OT.

Bill: 05:48

So, we break it into two different categories. There’s a cyber assessment and the cyber assessment is looking at things like policies, procedures, and it can cover things like backups, disaster recovery, what happens after an outage occurs type of thing. And we look mainly from a network and technology standpoint, we’re focusing at that edge of the manufacturing network.

If we do just a standard network assessment, we’re actually focusing, there’s always the focus on cyber security and what’s going on at the edge, but we tend to focus more down into the network technologies that used redundancy.

We run into a lot of times, OT networks tend to evolve organically. And so there’s not been like the master plan, if you will. Things just we add a switch here and we add a switch there and there are networking protocols, like standing tree and things like that. That can cause unplanned outages. So I would say a network assessment is more about reliability and a cyber assessment would be more about  securability.

Chris: 06:59

Thank you so much, Bill, that really helped, particularly, to connect the dots for me on the differences there. So once you have that baseline from that assessment, what steps should the end users take to enhance their strategy and start working that roadmap?

Bill: 07:15

So you brought up roadmap and I think that’s an excellent way to explain that.

 We’d like to follow the  NIST standard because that’s an open standard that is open and available to everybody. And part of the initial release of this that was very dry. It was very text and a lot of people were having a hard time trying to eat the elephant, if you were. So what this did was that they developed a framework and the framework basically is five steps to being able to manage your cybersecurity risk.

The first step in any of these frameworks, is identify. And that’s to understand what your risk is, what the consequences are and have an understanding of where you are as far as policies and procedures. Do you have those in place? What are your assets? That type of thing. And so that’s step one in the five step.

Then the next step is the protect.

And so typically we think of the cyber security assessment. Is that step one, right? So we’re going to identify all of your assets, what it is we’re trying to protect. We’re going to identify, what you have in place and where those deficiencies are. And then when we get into that protect phase, then we’re going to start looking at things that we can do to improve security.

A lot of times when, when I go to a customer’s location again, because they’ve grown organically, there may not be a clean edge between the business network and the manufacturing network. And a lot of times. People will, read, go online. There’s a ton of resources and our kind of tool in the toolbox for making that clear edge is a firewall.

And sometimes they tend to jump ahead if you will. And so they say, everybody’s saying firewalls thing to do so they’ll go and they’ll put it in a firewall, but they’ve missed that piece about policies and procedures, or there are assets out there that they don’t know about. So what they end up doing is they get that Swiss cheese approach to cyber security.

They’ve got some very solid pieces, but there’s a lot of holes in it. So we always say, start with identify, and then make a definite plan, that matches your facility and addresses your specific concerns. So from that roadmap, you’re going to identify, protect, detect, and you move through those steps.

Chris: 09:42

Okay. That definitely helps a lot. Now after that’s done, I’m assuming sometimes there can be headwinds. What are some of the typical ones that industrial users may see after getting that assessment completed?

Bill: 09:55

So I think probably the biggest resistance is unfortunately, Cybersecurity is expensive, right? And there’s no ROI in putting in a firewall.

There’s no ROI in buying switches or segmenting networks. So a lot of times the headwind is a lack of understanding. So even though there’s not a direct return on your investment, there is risk mitigation. For the most part, when we think about cyber security, it’s about mitigating that risk. Oftentimes we spend hundreds of thousands of millions, of dollars to engineer our systems to work in a specific way.

If we think at the very base term of what cyber security or what an attacker does, the attacker basically causes our highly engineered system to operate in a manner in which we didn’t plan for. So I think a lot of times the headwind is just getting everybody up to speed, understanding what those risks are.

And a lot of times people, when they talk about cybersecurity, it’s the chicken little approach, right? The sky is falling, “we saw on the TV news last night that a ransomware shut down or facility, or we see on LinkedIn, some paper mill in Canada got hit with ransomware and they got locked out and it cost them millions of dollars,” and they don’t offer up solutions.

And I think the thing is that. If you go to your management, if you go, whether it’s at the local facility and you’re talking with the plant manager and his core team, or if you’re talking at a corporate level, maybe to the board of directors, maybe to a CSO or something like that, having that kind of in your back pocket to layout these are the risks and the potential consequences. And we have a solid plan to mitigate those risks. And I think if you do that it clears the way or, heaps the naysayers or the people that don’t want to get into this, like kind of puts all of those concerns to rest.

Chris: 12:05

Absolutely. We thank you so much. Now inside of a plant who typically owns this process, Bill? I’m sure there’s different types of roles and responsibilities, but you have a lot of experience in this space and kind of curious on the different types of titles and roles that the ownership lands here.

Bill: 12:23

It varies from site to site. So to give my age away a little bit. When I started, GPA it was the mid nineties and a lot of the OEM were going away from those proprietary systems and going more to what we see today, off the shelf networks switches, PCs, windows operating system, that kind of thing.

And there was a bridge kind of burnt there because as the OEMs kind of went to this new model, the mill says, “How are we going to manage this?” And the IT managers not knowing the operational differences of OT. And IT says, “I see Cisco switches and I see desktop running windows operating systems and servers running windows operating systems. We do this all the time. We can manage that,” and so they started off of that approach and they tried to run it like it was an IT system. And so things like patches and updates. I was usually a really bad day if at two o’clock in the morning on patch Tuesday, all of your servers go down for two minutes while they reboot and applying patches.

And so a lot of times. The facilities that have lived through that typically there’s a strong disconnect between process control and IT. A lot of times we always say, we work to bridge that gap and sometimes it’s mending fences and trying to get people to sit down on the room and have a conversation.

So in those facilities, usually it’s the engineering manager it’s process control manager. If you’re looking more at a corporate environment, it’s the CSO, who’s the chief security officer, maybe for a corporation or something like that. Those are the folks that are worried about protecting, the OT assets. Typically in a corporate environment, those are the folks that have the overall responsibility to the corporation or to the individual facilities.

Chris: 14:27

Now, a lot of times we’re defensive so many times, but say we want to be offensive. We want to step our game up. So what can we do as owners to enhance and be proactive in our cyber security position?n

Bill: 14:40

I think there’s a couple of things that need to happen as we move forward there’s the promise of industry 4.0 digital transformation. And all of this means connecting our systems together and communicating more and more data. So on the plant floor, which is unprotected into our IT environments, which is protected, realistically speaking process control people are typically process control people.

They’re not IT engineers. They’re not Cisco certified administrators, network engineers, things like that. They’re specifically focused on keeping the machines running and programming their DCS or the TLC. So I think at some point. That merge has come back into place and we do need to get IT involved to help us with things like networking infrastructure, understanding the liability and the management.

And I think to a certain extent, IT also needs to understand that the expectation is different in the OT environment than it is in the IT environment. And as a, as an owner operator, I think that it is imperative that they get their groups to work together. And I also think that it’s imperative, not only between the OT and the IT, but education, so if the end users understand a little bit of cybersecurity awareness, they understand don’t click on the links from emails that you don’t know. Some of those basic things that sound really trivial, very minor can go a long way to help protecting the system. With the coronavirus, we’ve got a lot of people who are working from home, maybe even using the home computers, they can actually go a long way to help protecting their systems by making sure that their computer that they’re accessing their facility is patched and updated.

Make sure that, litle things go away from default passwords on your wifi systems and things like that. So there are lots of things, but I think the key to that is communications and education.

Chris: 16:51

No doubt. I mean, I love what you said about creating that synergy between IT and OT and education piece, not having your password, being password, the basic stuff like that we see, but we’ve een it in our company too, just being in front of education and taking the lead on that I think is so important right now.

Things will happen and what should be the typical response if something does happen in a plant. And then that from a recovery standpoint, can you give a picture of what that looks like as well?

Bill: 17:25

Sure. So I guess when an incident occurs, there’s a couple of things to keep in mind.

So there’s been a lot of studies that show that if a intruder gets into a network, oftentimes they’re in the network for maybe six months or even longer before they’re detected. So some of the things that we can do is number one is start watching our systems a little closer and again, through education, understanding what that may look like.

 I know that I have seen situations where. And intruder has been in the network. And just because they’re knocking around and they’re not familiar with maybe some of the proprietary industrial networks, there’s been like some unplanned outage. And when they go through that root cause failure analysis, they’re looking at everything from there was a bearing that caused the load to cause the motor to trip and all of this. And they never thought that, maybe it was a network problem and maybe it wasn’t necessarily a life cycle type of thing, where the switches just wore out and quit. Maybe somebody was in poking around.

Chris: 18:32

Great. Now, you, you said something that struck me that I hadn’t, I have not heard talked about it in this space yet.

So the typical time could be six months or more that a threat is actually already in your system before you would start experiencing, is that I hear that correctly.

Bill: 18:49

Yeah. So depending on the attacker and what their goals are, oftentimes we hear about phishing and different things like that.

Sometimes that phishing is just to get a foothold. And then once they have a foothold on a computer, they use that. They start to look at the environment that they are in, as I said earlier, a lot of times it’s extortion. And so when they cast that very wide net, they have to figure out what they’re going to charge in order to basically return your system to you.

So a lot of times they’ll actually get that foothold and they’ll look around to see, is it just my home network where I see a couple of smart TVs and maybe a couple of lights and a laptop or something or am I seeing something more like a manufacturing system?

Any of the ransomware messages that I have seen in the message of the pops out. They never give you the price. It’s always contact us. For us to unlock your systems. And so a lot of times they get that anchor point and then they start looking, they start watching how many machines are on the network. We talked about a lot of times the industrial protocols are not as secure as say an enterprise type of environment.

So a lot of the times they’re actually seeing the data, whether it’s, operator interactions with a PLC or controller or, the information coming out of the system so can build out and understand the magnitude of what they have control of. That’s a common thing for an intruder to be in there for an extended period of time before they’re actually detected.

Chris: 20:35

Wow. Wow. That’s crazy. And so from a recovery standpoint, after that, can you touch on that?

Bill: 20:42

Yeah. So from a recovery standpoint, so once you figured out that you’ve been compromised and like I say, most cases we see a lot of ransomware, so machine blocked up, you can’t do anything. There are several things that have to happen.

And initially, when we started, we talked about then this framework and part of that first step is making sure have policies in place. And so hopefully you have a policy in place that tells you what to do. If not. There’s going to be a bunch of people running around with their hair on fire, pointing fingers and trying to figure out what’s going on in many cases, in for example, ransomware, it is a crime.

And so you need to report that crime. A lot of people don’t know this, but in a manufacturing facility, if you are hit with cyber security, You don’t dial nine one one, you call the FBI. The FBI has a team that works with NIST and others. They call it a CERT team and it’s a cyber emergency response team.

They will actually come to site and they’re going to collect forensic evidence to try to catch the person that infected your systems. They’re going to try to catch and prosecute. Typically the FBI  tells the end user not to pay the ransom because we don’t know where that cash is going and a lot of times it’s criminal enterprise.

So it could be going for drugs, weapons who knows what that money that you spend is actually financing. And so typically the law enforcement recommendation is don’t negotiate with terrorists.

There are cases where you have no choice. Again, if you don’t have the proper procedures and policies in place to restore from backup, but if that’s your only set of running code that’s been infected, you may have to pay the ransom in order to get your data back.

Ideally, if you’ve done everything correctly, as soon as it is detected, you want to isolate the system. Again, that’s where network segmentation comes in. You can isolate to a small area rather than impacting the whole facility so that it doesn’t spread. And if you have good reliable backups, You’re going to start from scratch and you’re going to restore backups on the systems that were affected.

And then obviously there was a hole that they got in. You want to plug that hole up so that they don’t just turn right around and get back in. So depending on how good your backups are, depending on how quickly you respond to this through notifying law enforcement, that type of thing, it may be downtime for a day or so.

I have seen situations where basically somebody got in, they pivoted, they did their searching of the network. They figured out that they were on the corporate network and actually affected facilities worldwide. Some of those take months to get back to normal. So that cost and the risk associated with that can be quite extreme.

And then even once it’s happened that the road back. Yeah, it’d be long and difficult.

Chris: 23:57

Now thinking about from an outside resource standpoint and a lot of our industrial end users, they love to have outsourced type support. So from this space, what outside services could they explore after they get that network back in place to ensure and enhance their reliability from a security standpoint?

Bill: 24:20

I hate to keep going back to the identify step of the NIST framework, but it actually calls out as part of your planning. So before the incident happens, you want to involve, as you’re doing your cyber security, you want to involve the OEMs or the integrators, the distributors, people that know that system inside and out, you actually want them part of your team so that if something happens you’ve done and followed what the manufacturer’s recommendations are. You’ve done due diligence to follow best practices. And you also have resources and allies to help you so that you’re not dealing with this by yourself.

You’ve got those outside people coming in and sometimes just having third-party validation is immeasurable because somebody is looking at it from a clean slate and it doesn’t have a predefined expectation, if you will, when they look at cybersecurity, when they look at the way that you’ve tried to implement different policies, procedures, and different strategies to protect your equipment,

Chris: 25:29

Absolutely fresh set eyes, and it’s very important. Bill, we call it EECO Asks Why we always love to get to the why. So why is being proactive in the cyber security approach important for our industrial end users?

Bill: 25:44

If you think about what happens in a cyber event, right? So we’ve got our highly engineered systems that are no longer working the way that we designed them to.

If you think about somebody who was standing next to a piece of equipment, and that equipment comes apart, there is a chance that somebody could get hurt. There’s a chance that somebody could get killed. If you look at the potential impacts. So these systems are managing our safety systems, they’re managing our environmental systems.

If things are non-operating as we designed them, we can impact the environment. So whether that’s local to the facility or maybe even spread out to the community, even if the very always say, this is the best case scenario. Is you take a hit to your reputation, right? You about to tell your board members or your stock investors, “hey, we had a cyber incident and we lost this,” you know, you’d take a ding in your reputation, but in the grand scheme of things, that’s the best case scenario. If that is the only impact that you have. So if you call it life safety, environmental safety. Everywhere I go, every manufacturer that I go to before you go in, you’ve got to take the safety training and all of that.

And every company that I go to, one of their high priorities, they’re their cornerstone of their business is life safety. And if you think about the potential impact of a cyber event can be dangerous to life safety. A lot of times I think that. If people thought of it in that term, it would actually rise maybe even a little higher on our priority list to understand that the actions that they take to protect their systems also protects the life safety of their employees, their contractors, and the surrounding community, the environment, that type of thing.

Chris: 27:40

No doubt Bill. Thank you. I mean life safety point, that is the why, you drilled it. You really unpacked a lot to be being proactive when you’re talking about the map, identifying and protecting, detecting doing that assessment. Bill, you covered a lot of ground, a lot of good resources.

We’ll try to link some of the things that you referenced, on the podcast for our listeners to go to and research more. And Bill just thank you so much for taking the time with us and going through all detailed  information to make our industrial end users stronger and safer against cyber security in the future.

Bill: 28:18

Thank you all. I hope somebody will take something away from this and, you know, go ask a question, start a conversation, you know, make an impact on your facility’s cybersecurity.