059. Season 2 Recap Transcript

00:00 Chris: 

Welcome to EECO Asks Why today we have an idea episode. We’re going to be talking about something that comes up more and more frequently lately. I hear people talking about this all the time. So we said, you know what? We just need to tackle this head on. And we’re talking about IT/OT convergence, and to have that conversation we brought in an expert, we brought in Michael Manzi, who was the manufacturing information systems practice lead at Feyen Zylstra so welcome. 

00:30 Michael: 

Thank you, Chris. 

00:31 Chris: 

I’m excited to have you, how you doing? 

00:32 Michael: 

I’m just doing ducky. Oh, I’m feeling good today. Feeling like I’m ready to talk about IT and OT. Exciting topic. 

00:38 Chris: 

But exciting topics. It’s really coming up more and more. I was just in a plant yesterday and this topic came up and, I have some younger kids and one of them is in middle school and maybe help me if I needed to explain it to her, she’s a sixth grader about what IT and OT is. How would you start defining that?

00:58 Michael: 

Well, I would say, to start off just in general, IT deals with more of the business side of the company. Where OT, operational technology, deals with the machinery that runs on the plant floor and actually making the product very similar, symbiotic uses a lot of the same technologies, but to put it into terms that a middle schooler would understand, think the administration at the middle school, like a principal, the secretary they have a job to do, and they’re in charge of the teachers and the teachers are in charge of teaching. The administrators don’t teach but the teachers do so OT would be the teachers. They’re the ones actually running the machines or teaching the kids versus the administrators, which need to run the building. And part of that is making sure the teachers are hitting the marks that they need to hit. So they need reports from the teachers of how is the class doing is a class on track? What’s the average? They don’t need to know minute by minute, is Johnny sitting at his desk is Sally paying attention? They just need to know key process indicators, like such as average, the average grade of the class, what was the attendance today and to that degree, right?

02:06 Chris: 

I think that’s the best analogy I’ve ever heard of comparing IT/OT. That painted such a great picture. And I know we’re talking about, you live in the OT world a lot as well, and I’m just trying to maybe help our listeners understand that picture of how does IT see what you do in process control? What is their view? 

02:25 Michael: 

We’re the wild west of networking. There are no rules. They see it like I mean, it’s an old reference here, Thunderdome. And to a large degree, they’re not incorrect. Cause in the OT world, our machines have to run, we can’t afford downtime. And a lot of us learned IT. We didn’t get formal training. We just had this machine that all of a sudden was capable working on a network and we network it to another machine. Hey, there’s this thing called a switch? Well, I’m going to get this unmanaged one because I don’t want to take the time to learn how to work a managed switch. I just want my machines to run. So you’ll see a lot of organic growth and process control networks, which is a common term for operational technology networks. And within that, it’s just, there’s no structure very poor record keeping, poor security practices. They just want things to work.

They want to be able to get help when they need it. So there really is no thought into how people access that network, limitations on it, how it’s structured and in a modern world, it’s led to some problems because now the OT world needs to interact with the IT world. Everybody has heard about the internet of things and manufacturing 4.0. And that’s where the convergence is now happening. 

03:35 Chris: 

Since you guys are the wild west. If you’re in that. What’s their perception, right. Speak to their side, how they viewing you? 

03:43 Michael: 

Well, it’s actually how we view them. Cause they, they view us as, almost ignorant to IT practices and they are formally trained on networking, quite often, they hold, certifications from Microsoft, from Cisco and they understand how data packets really move, how switches are set up, how to limit, cyber storms how to patrol the remote access in a safe way. When OT looks at them though, they’re overly restrictive because, in our mind, the machines have to run. And if you’re not enabling us to run the factory, you’re not helping the factory. So a common thought processes, I would use to sum up how I saw IT is they think the factor is made for their networks and that they weren’t enabling the plant floor to be successful. And you know, we started having these conversations 15 or 20 years ago and it was oil and water. And it was two people talking a similar language with dissimilar interests and some of the terminology where we thought we were saying the same thing actually meant different things. 

04:44 Chris: 

So it dates back that long ago to when it all started hitting the plant floor? 

04:50 Michael: 

Well, once you had a MES systems, manufacturing execution systems, come into play probably in the late nineties, early two thousands business people said, Hey, I can actually get these reports, and see up within the hour within the minute, what my plan for is doing. So everybody thinks this IOT stuff is brand new manufacturing 4.0 is brand new. It’s just the same stuff re-termed and packaged differently. You see a lot more IT presence on the plant floor. You see a Microsoft come down on the plant floor. You’re seeing ERP systems come down to the plant floor and a lot of other third-party players trying to redefine the detonating known data flows of the control systems. Just a new terminology for what is essentially the same thing that we’ve been doing for 20 years. 

05:35 Chris: 

Now when I was doing some research to get ready to talk with you. You’re an expert. I want to make sure we go where you want. I ran across some things called the CIA triad and how that is viewed, from the OT world and an IT world there’s definitely different priorities. And you’ve already spoke to some of that. So maybe that break that down for our listeners so they can understand what you mean the priorities are different for IT and OT, what are you actually referring to? 

06:01 Michael: 

There are several areas where we differ on how we view our networks. And I’ve stressed earlier that our machine’s got to run. And one of the things I pose to young engineers or when I go into a new factory or something, I said what do we make here? And if I was at, a paint company, the first answer I expect to hear is paint. If I’m at a machining company, drill bits hammers, or if I’m at a car company, cars, I tell them they’re all wrong.

Every plant makes the same thing. They make money. And they make product to make money and the machine makes the products. So if the machines are not making products, the company is not making money. If the company’s not making money we don’t have jobs. So on the plant floor, our top priority is availability. The machines need to run. On the IT side, they’re connected to the outside world. They’re worried about confidentiality, security, and availability is their third priority. Where you know, availability is a top priority of the OT side, where confidentiality is the bottom priority. And when you don’t understand the cultures, then you go to war without understanding why, right? Because we’re both not helping each other because we have different interests. 

07:19 Chris: 

Big time, different interests there, so it was the middle layer of integrity. Is that viewed the same equally for IT and OT or is that even considered? 

07:29 Michael: 

Yeah. Availability for process control is probably the highest. And then, integrity is kind of in line with that is the system has to deliver good data, reliable data. And that we actually do have common ground and that’s a good place to start and, you got to come up approach the conversation and not come at it as oil and water. You’re going to need to do this for us or else, right. Or we’re not going to do this for you. It’s just, Hey guys. And that’s why I start off with the, we all, we want the company to make money. And we’re all playing in the same sandbox. We’re on networks. So you do things better than we do in certain aspects. And we understand what we need to get done. We need you to help us enable ourselves to be successful.

And you need to understand on your end, you know what our priorities are. That these machines need to run. And I can’t have you randomly patching stuff causing downtime on my network. So we have to come up with a mutually agreeable solution that protects both our interests. 

08:27 Chris: 

Very much. So now, when you talked about the plants, the industry 4.0, smart manufacturing. We’re moving so much data now. That data traffic, maybe that could be an issue that causes some conflict there. So what are the types of data that you guys are looking at moving around and managing and how are they different from the IT world versus the OT? 

08:47 Michael: 

And now you’re at a big crux of the argument. I’ve crossed this bridge many times. So let’s say company A CEO, he wants to get some of this industry 4.0, he went to a seminar. He wants to modernize, Hey, I want this IOT. And he looks at, and he sees a computer. He sees reporting stuff, he sees data collection. And who does he know that does. Hey, my ERP guys, my IT guys, they fix my iPhone, they fix my computer, they know the internet stuff, they know how to do data.

So they asked their IT, or, the CIO or the IT, this isn’t everybody I’m just taking just a general use case here. There are some people that are more savvy in some areas and it’s changing over time. People are getting more educated, but the IT guys deal with the business side. So when they look at a reporting system, their work they’re typically dealing with an ERP style system, a business reporting system, and those systems are used to receiving data every hour, maybe once a day, no more than once a minute or every 10 minutes. So when it goes to process, a daily report, it’s looking at hundreds of points of data to compile.

And usually those systems are telling you what happened yesterday. What happened last week, rear view looking back, right? Yes. When we deal with an OT system, the MES systems, and even to the IOT systems. Now this is where the differences, it’s the amount and velocity of the data. Say like a press control system. I have written monitoring systems and analytics systems on that where I’m collecting data at 10 to 15 milliseconds that can be millions of points a day and that’s just one data point. So let’s say press speed. It could be a million data points a day. So a business system, many of them, when they try to process that amount of data will time out or we’ll return the report six hours later. They’re not built to handle that magnitude and velocity of data. 

10:45 Chris: 

So when I also, one thing that keeps coming up and even in our company, I’m getting this more and more the topic of cyber security. And I’m curious from your standpoint, it seems like that’s a really big deal for our IT engineers out there that cyber security topic, is it that big a deal for the OT? Just wondering where they come together there.

11:04 Michael: 

It wasn’t. And this is one of the ways that OT needs to listen to IT because they’ve been dealing with outside connections for a long time. In the OT world, you were generally air gapped. You had, a machine had some network internal, but it wasn’t connected anything. Maybe you connect that to another machine, and it wasn’t connected to the outside world. And then now it is in a modern mature system. So when we look at that, you know, we’re like what’s the danger? I put a password on it. The data moving’s encrypted, I’m safe. You’re not. There’s people that spend their whole day figuring out how to decrypt things, how to get through passwords. And you look at typical cybersecurity stories, I always point to the one from target, 10, 15 years ago they had an HVAC system, had a remote access system on it so that the HVAC supplier could help them maintain the HVAC system. And that’s where the hackers got in. They got into an HVAC system, into a fan, basically got on the network and stole everybody’s credit card information. It’s crazy. 

That’s a routine issue, you need to own your remote access. You need to monitor it. You need to be in charge of it. There’s a lot of third party remote access systems and they’re convenient. They’re easy to use. And when you look at good, better, best is kind of a term I used. It’s probably good, it’s got password protection, you open the connection. You can close it yourself, but ultimately you don’t own the connection. Some third party app out in the cloud somewhere or somewhere else is brokering that connection for you. And if you’re going to be proactive in your cybersecurity, you need to own that connection. 

12:45 Chris: 

Good point. Has this started being an increasing relevant topic with COVID? Work from home? People wanting to get data from the manufacturing floor? 

12:54 Michael: 

Everybody wants to enable remote access, not only for their own users, but for remote support. These machines are getting more complicated, more technical harder to support. In some cases, OEMs will refer to their machines and a s intellectual property. So you’re not allowed to access the PLC programs. You’re not allowed to see what’s going on. You were reliant on them for support. They all offer their own remote access package. Some of them even, we got our own cellular technology, we’ll just add it on there and will instantly be able to connect to your machine. And maybe in some small factories where it’s machine not connected to a network. That’s great. In a mature factory with it all networked and that many networks connected to your business network. You’ve just created a cyber problem. 

13:41 Chris: 

Right. Good point. I mean, we’re hearing more and more OEMs, just like you said. That’s their preferred method of support, to have that remote connectivity, versus the old days of flying a technician out to somewhere to reset something. These OEMs are really pushing that remote. So your advice is to own that process, no matter what. 

13:59 Michael: 

The ratio of machines to website capable engineers that, that ratio is getting higher and higher and not in a good direction. How do you, with a finite level of resources and of engineer’s capable of supporting and an ever-increasing number of machines to support. Remote access is critical and a must have, and you need to do it correctly. And another term I like to use when I talk to people is cybersecurity doesn’t happen by accident. You have to think through this, there is a whole bunch of phase zero stuff in a modernization project of how am I going to structure my network? Where do I place my firewalls? How am I segmenting my network? How do I allow access? Who can access? What level of security?

It’s not just, you put a antivirus program out there and in a lot of cases, OT networks don’t like antivirus programs because they shut down the network, they shut down machines. And again, it goes against our top priority of availability. There’s a whole separate suite of OT monitoring products and also on OT systems, you will see that windows updates are not done to the second.

You’re lucky and in a good system, maybe they’re done quarterly, six months, a year? There’s some systems out there that haven’t been done ever. And they’re afraid that the update is going to break the software. And in many cases it has in the past. And, once you hit yourself in the head with a hammer, once to do a second time, I believe it’s called insanity or something like that.

15:27 Chris: 

That’s right. Now let’s, take it back to the plant for a second as well. So say something fails on the OT network and conversely let’s talk about something that fails on an IT network. How are they treated differently? So far as replacements, upgrades and things like that?

15:43 Michael: 

So that’s another, a high area of difference. And then the big thing there is meantime to repair. And again, it goes back to the availability of the network. It’s critical on the OT side within the OT side minutes count, seconds count. So anything over minutes or hours is devastating. You shut down production. What’s the cost per hour of not producing stuff? You’re costing the company money. On the business side so you didn’t get your email and it’s not as nearly as critical. So that’s one thing IT has to learn from OT is the criticality and how to keep up the resiliency of that network. Things like ring topologies, Dundon stars having spare parts on the shelves, having a switch, hot switch on hot standby are pre-configured ready to be plugged in to replace another switch, power cleaning UPS’s. These things need to be part of phase zero design. What is the criticality of this network? And you have to think through it as to what is the proper architecture of it. 

16:46 Chris: 

Right. So you’re saying to the IT, sometimes that speed to resolve. It could be delayed a little bit. Cause like I said, you just, you may miss an email or things like that whereas the OT doesn’t have that. Is that the ultimate rub between the two is that speed to actually make the upgrade enhancements and put those in place? 

17:05 Michael: 

One of them. That is a big one. It’s just probably on the top five, top 10 lists. Another big one is, controls engineers need to be empowered on the network. They need to be able to handle the data traffic, have machines talk to each other, be able to structure their databases, install software, make changes and on a typical IT network, you tell IT I want administrative rights on the network and I want to be an administrator on my PC and be able to do whatever I want to do on it. See what reaction you get. 

17:33 Chris: 

You know they’re going to laugh you right after. 

17:36 Michael: 

That’s right. So I like a one place. I actually installed a SQL server database and the IT guy made me uninstall it. He had to install it. And then I had to give him the schema for him to set up within it. And then I had to test it. And if something wasn’t right. I had to submit a ticket for him to make a certain change. A day later that change was made. I had a test it again and then okay. That this part works, but now this part doesn’t so I need to make this change. So for an operation that should have taken 10 minutes, ended up taking 10 days.

And in that case, what we ended up doing is we had a big IT/OT summit and that’s really what every medium to large company should do if you have IT and OT departments. And you might not know you have an OT department, but you probably do your maintenance supervisor, your controls engineer, your plant engineer they might not have the official title of it, but they’re responsible for those machines running. You got to sit down and talk about how are we going to handle this? One case we separated the networks we put in a DMZ zone, which I think is kind of funny because I talked about how conflict happens between IT and OT. So we create this third network in between called the demilitarized zone where we can pass data back and forth to each other firewall on each side.

But we went so far as to create a whole separate domain on the process control network. That was the responsibility of the process control engineers. It was maintained by IT, but they empowered the process, controls engineers to what I would call pseudo administrator. So we could add machines, remove machines, install programs, but we didn’t have that power on the business network then since it was a separate domain.

19:16 Chris: 

Wow. So the summit that you’re talking about. That sounds like a great opportunity because my question was at that point, it sounds like there’s no really common grounds of trying to figure out how do we get common ground. And I guess that is just get people in a room, brainstorm, get them together, collect those ideas, share with strong leadership.

19:36 Michael: 

It takes a willingness from both sides and strong leadership and, you really need executive sponsorship really to walk it through the process. And because OT is busy, IT is busy. And, they kind of look at this as a hindrance to what their daily routine is, and they got to understand, this is the future of the company. We have to figure this out. 

19:57 Chris: 

Now, those summits you’ve mentioned phase zero several times today. So those summits and those meetings and those engagements, is that a great place to start trying to lay some foundational work between the two? 

20:10 Michael: 

Absolutely. I mean, you know, when you’re having those summits, you’re really going through the needs and priorities of both sides and OT needs to learn how to be structured and IT needs to learn how to let go of some of the control and to enable the machines to be available and running. And in there you’ll have to sit down and design, the topology and in governance. So there’s two sides to it. There’s infrastructure and governance, right? One is just, how are we going to architect it? And who are the machines? where the switch is going? What type of cabling are we going to use? And that’s a difference too, because office grade stuff versus industrial grade stuff is two different things, right? And so it was two different suppliers to different players in those fields.

And then the governance side is how do we administrate this? And you got to figure out where it makes sense. And what I found is from an OT side, a good approach when talking to the IT, is how do I make your job easier? Because a lot of it, they’re looking at this extra network that they have to take care of and if they give us freedom on it, then we’re going to ruin it and it’s going to create more work for them, but if you make it, and you don’t have to lie about it, it’s, this is inevitable. So we can even part be partners with you and help you and reduce your workload. Or we can continue just to be a hindrance to you. And is this is going to happen. 

21:29 Chris: 

Now let’s stay right there. Let’s say just a quick analogy. You got your, IT who’s the Yankees, you got your OT who’s the Red Sox. They’re probably never going to get along. So what point do you need to bring in a third party? And then when that happens, who do you bring in and, what is the desired outcome of that, kind of conflict resolution to bring the groups together?

21:50 Michael: 

I was going Yankees and Rebels. Go blue and gray. Right? There you go. Yankees, Red Sox works too. Whichever is more politically correct, I guess. So you’re right. So you got this strong leadership, you’ve got this IT group, you got this OT group and they really don’t know how to talk to each other. And so we were successful at and when I was a corporate engineer at global engineer for a couple of companies is we did bring in a third party. That had OT specialists it and OT specialists that came in and arbitrated the discussion because they knew the areas that would cause friction and had actually gone through this journey multiple times.

And it can really enable the conversation. Think of it as a marriage counselor. Sometimes they know couples can’t talk to each other because they got different priorities and they don’t know how to even engage in the conversation without getting into a fight. And, but you bring in some, an arbitrator who knows what both sides want at the end of the day and lead them through that discussion.

And you want it done in a way that you’re not telling them what they need to do. Every plant is unique. Every company is unique and you need to respect that uniqueness. Sandler sales training will tell you don’t put seagulls on people’s paintings. Tell them there’s something missing in the sky and describe a seagull to them.

23:08 Chris: 

At FZ. Is that, do you guys offer that type of consultation where you can come in and help bring those worlds together? Just know if you had any examples of how you approach that. 

23:21 Michael: 

Yes. That, that is something that we do. We do have consulting level services where we would come in and sit down and go through your user requirements. It’s probably the first thing. This is all phase zero. Why are we doing this? What’s the business case. We got to understand what we’re trying to solve for you at the end. And then once we got those requirements, do you need a recipe system or are you just trying to get just reports from your machines or is this, legally critical? Are you making military grade stuff? Is this an explosive environment? Is there a safety issues? We got to understand the, the requirements of that particular facility in that company. And then from there you can develop a network with a logical framework with modules that would be needed.

And from there you would then go into, the physical structure of, okay, now we logically see this, how do we physically do it? And that’s how you walk through that conversation. But it all starts off with those user requirements. And I think that’s where that arbitration comes in and once IT and OT are sitting at the table going through user requirements, it develops that partnership. They both see that they’re able to help each other instead of hurting each other. 

24:29 Chris: 

Any commonalities from your experience where you’ve seen across multiple disciplines IT and OT. I’m trying to think through something that our users can take away and try to actually apply. You’ve talked physical a few times that physical layer just wondering, are there any red threads that you’d like to throw out there to, Hey, you’ve seen this working across multiple industries, 

24:51 Michael: 

So, yeah, one of the things you’ll talk through logically as a typology and with that, it’s, how am I going to run the level three and level two infrastructure through my plant. And so you would sit together the IT and OT teams talk about benefits and drawbacks of rink typologies and start typologies and mesh topologies. And even the wireless aspect of it as well. And then he would sit there and what buildings did you switches to in? Where do we need to put in cabinets at places? What go, how do we construct these cabinets? Did we have pull ups as in, do we need one ups? Do we need two? Doesn’t need to be a single switch or redundant switch, right? Where do we need patch panels? And this, that definitely takes a collaboration between IT and OT to get that conversation done. And then you have a typology arc. 

25:38 Chris: 

Great examples. And I’ll tell you what, for our listeners out there, you’ve probably filled up several pages of notebook paper, taking notes listening to Michael. This is great stuff, Michael. And we always wrap up with the why, you really unpacked at IT/OT convergence, but let’s get to that.

So why is it so critical for, IT and OT to come together? Cause I mean, we’re all on this path together, right? Towards 4.0, so at some point, we got to learn to play in the sandbox together. So what would be the why there? 

26:06 Michael: 

You know, you owe me money for using that one, that same sandbox, but yeah you’ve hit the nail on the head bit there. And you know, when I was working for end for end users and one of the things I would say, it’s when I felt that we were getting off track and starting to be in conflicting again, is if we don’t do this now, we’ll be doing this later with three different initials above this building.

It’s inevitable it’s happening. It’s modernization. We knew one of our competitors had actually moved significantly down that path and they were beating us on sales numbers across the board. And we’re competing them by 20%. And we did a little bit of math and the level of automation that they had took out about 20% of the cost of them to make the product. If you want to compete in the modern world, modern problems require modern solutions. 

26:58 Chris: 

Well, Michael, this has been wonderful. Thank you so much for our listeners out there. Check out the show notes. We’ll make sure there’s links out there to connect with Michael to FZ the wonderful things that they’re doing that supports you on your journey to smart manufacturing, industry 4.0, so Michael, thank you again and hope you have a wonderful.

27:16 Michael: 

Thank you, Chris. This was a really great conversation. 

27:19 Chris: 

Wow. What a great conversation. I know what impacted me the most was hearing about the summit and bringing all the groups together to make better decisions to move forward. I want to remind everyone about the war stories. You can submit those on Facebook and Instagram, and that comes directly to us. And also if you can help this podcast by giving us a five-star rating or writer, the simple one sentence review. And I want to remind everyone, keep asking why.