When “I’m Fine” Isn’t the Truth (EP 135)

[0:10] Hello, friends. Uncle Marv here with another episode of the IT Business Podcast, the show for IT professionals and managed service providers, where we help you run your business better, smarter, and faster.

[0:22] Well, folks, if you're an MSP, we've heard all about that alert this week that is at the top of your list, the two vulnerabilities for PHP, being called PHP to Shell, and it's turned what might sound like a website problem into a broader infrastructure concern. The issue isn't whether you just simply build a website with PHP, it's whether PHP is running somewhere else inside the systems, appliances, and services you manage. And of course, that's where it gets interesting for us service providers. So I wasn't going to address this at all because I don't do websites. But a client question actually made me start to think about it in a different way. I got multiple questions this week about what we need to worry about because, of course, the news outlets have really turned this up and they actually brought up the, Interesting fact that PHP could affect up to 70% of websites.

[1:38] Web browsers, applications that have a GUI interface, and it has been talked about that pretty much anything we connect to through a web portal could be affected, and that led the question to come to me as to, well, what do we have, that we would need to worry about? And at first, I'm like, I don't know. So I had to go back and do a little digging. I told the customer I would. And then, of course, a second vulnerability came out with the ConnectWise and their cloud backup and the third party access problem that it also brought up. Now, if you don't know about the ConnectWise issue because it's been overshadowed by the PHP, it actually started before PHP, but ConnectWise identified, unauthorized access involving application credentials associated with its cloud backup service, which was formerly known as SkyKick.

[2:36] So ConnectWise suspended cloud backup, and as a precaution, they also suspended related SaaS security, cloud management services, while it investigated its response. And the second part of that story that apparently didn't get as much, attention is that security researchers also reported activity involving application access to Microsoft 365 tenants, which most of us manage in some form or degree. And that included behavior consistent with attempts to establish persistence and increase access. So again, I thought, you know what, this is something where.

[3:19] The narrative probably needs to be changed because vendor security all of a sudden is becoming our security because even though we may not manage it, our clients are looking to us to, one, explain what is going on, and two, for us to make sure that it's not happening to anything that they use. So I did some quick searching, and of course, I was happy to see that a lot of the vendors that I use had put out PHP statements. In fact, Action 1 was probably the first one that I paid the most attention to because Action 1, you know, temporarily shut down their patching service. And I think it was shut down for, was it an entire day? I, maybe, maybe not. But they shut it down and they pushed out a whole bunch of notices that they were aware of recently disclosed PHP vulnerabilities, and.

[4:19] They made a comment to ensure the security of our customers and follow cybersecurity best practices. We have temporarily paused our services as a precaution until a patch is available and the vulnerabilities are remedied. And then, of course, they actually did let us know that they found nothing and resumed services and all of that stuff. There was also one of the web gurus that I use because again, I don't use, I don't do websites. I sub that out or my clients have their own people that do it.

[4:53] And I was happy to see my primary web guru sent out an email to clients saying that they wanted to personally reach out and let them know that they were aware of it. And they actually explained a little bit more. And they said, while these vulnerabilities may not affect every website, we're checking all sites as a precaution and making updates where needed to ensure they're running supported security patched PHP versions. And they even said that if your website was hosted on these two particular platforms they've already confirmed, that websites hosted on those servers were already protected, and they even put in big underscore no action is needed on your part we're handling the necessary reviews and updates as part of your security package so that was great so action one, my web guru of course we saw.

[5:50] Statements come out from a lot of other vendors. I won't go down the list because you should have gotten them from your vendors, but it did bring up a point of saying, you know, do we know everything that is running PHP? And as I mentioned earlier, no, I don't think that we all do. And some of us that are actually doing websites that are actually developers may have a better handle on it. But the one thing that, For those of you that aren't doing anything, well, we need to understand PHP can be bundled into a hosting platform, a management platform.

[6:27] A portal, or a vendor appliance. So while you may not be responsible for securing the product, or let me rephrase that, you might be responsible for securing the product without ever having installed PHP yourself. So if one of your firewall vendors says that there's a PHP vulnerability, they may send you an alert saying, hey, here's the patch, you need to install it. So if you've not done any research on this at all, there's two vulnerabilities that they were focusing on this week. Both of them start with CVE-2026, and then the two differentiator numbers, 109618 and 109619. So of course, check and see if those need patching. Both of them can allow unauthenticated remote code execution.

[7:21] And PHP released fixes itself on October 21st. But getting those patches into your environment depends on who supplies and who maintains the affected software.

[7:34] Now again this brought up to mind what should we do in terms of knowing what's out there well one start an inventory and that's what i actually ended up doing um i didn't do it until Friday, but i started doing it i started looking at public facing websites i started looking at client portals web servers um none of my people have self-hosted applications but I m running virtual machines. So I looked at all of that.

[8:04] And then I will, over the next week, start to expand that review to all of the appliances that are run by vendors, hosted servers that may include PHP. So a lot of that is going to be me. If I haven't seen my vendor send out a notice, I'll probably send an email saying, hey, what does this have to do with you guys? Are you running PHP? Is it affected? Who owns the update? Because that's going to be a big thing. One of the other customers that came to me and asking, hey, can you help us, make sure that we're protected from this? I actually had to tell them no, because they are not being maintained by any of my web subcontractors. They either had, you know, a friend of theirs do it, another company do it, their kid or whatever. And I'm like, you know what? No, that's something we don't manage. You didn't want us to manage it. You need to contact those people. And luckily, none of them are running a web server on any of our systems. So from that standpoint, I feel a little bit comfortable. But again trying to find out what is running inside our environments is something that needs to be done so getting an inventory is key, for vendor appliances of course the vendor's firmware or software update.

[9:29] Is what's going to you know be needed to look at this is not a do-it- yourself PHP replacement for managed hosting or software with service, get confirmation from your provider that the affected systems have been addressed. So... I've started to put together a list of which vendors we should be watching, but I thought, you know what, that's not fair.

[9:55] I don't want to do that, but I do want to go a little bit beyond that where I want to say, don't stop at patching. So for exposed systems, we've got a review for unexpected processes, newly created PHP files, look for unusual outbound connections, and unexplained web server crashes. So if all of that has happened, that may be something to look at and alert the people that it needs to be looked at. Installing an update closes the vulnerability, but it does not prove that nobody used it beforehand. So again, just because you patched it doesn't mean it's fixed. So the takeaway, of course, is, you know, don't just send your team a ticket saying, you know, patch PHP. They need a scope. They need a way to verify completion and there needs to be a way that every system is documented as you know affected and patched or not affected, the fix was verified and all of that stuff so that's what needs to be looked at there and then this gave me a different perspective and i mentioned.

[11:10] And the way that I titled this show is to say that, you know, vendor security is our security. And again, the customers don't care. Who the appliance is from or what product is being run. They just want to know that it's being taken care of. And usually they're going to look to us first for that. So that got me thinking, you know, even if, the estimate is high, that 70% of, web, you know, portals and appliances and stuff could be affected, even if it's high, even if it's 60 or 50%, you know, So the bigger lesson is that every vendor connection is a potential attack path, which is why I would say, you know, always vet your vendors and make sure that they are staying up to date. You need to know which vendors have access to your systems and to your customer systems and then maintain an inventory probably from the one you just did to see who was affected by PHP. Now, this also includes API keys. And includes remote access connections. The bottom line is if you don't know what's connected, you can't properly access the risk, which is something that I had to come to grips with myself this week.

[12:35] Another thing I decided to look at was applications that we do allow or trust and specifically with the ConnectWise thing in the M365 portals, all those apps that show up, you know, that we either give access to or that get access that we don't even know about it. And for example, if you were to go into your 365 portal, and I have done that, to my main customers over the last couple of days, is look at what shows up in app registrations, ones that you have put in yourself. That's probably going to be something like AppRiver, Cork, those types of products where when we go to set them up, they give us all of the stuff that we need. There's usually a connector that you have to do when you set up the API. It will give you a list of, hey, this needs read access, this needs read write access. But if you go under enterprise application, there's a lot of apps listed in there that you probably, one, had no idea were there, or two, had no idea that it was installed because it didn't need your help. I'm talking about applications, and I'm looking at one of my client portals now, Adobe Identity Management.

[14:01] It. I looked it up. It's one user. I have no idea what they did, but it's one user that has that in there. Uh, there is monday.com. Now I know that I've got a couple of clients at this site that uses Monday. Zoom has got two in there. There's a regular Zoom and then a Zoom social login. There's a FedEx office. Didn't know that was in there for this client. There's group me. Uh, there's actually an old one in here. This is a client. Apparently I used Backupify a long time of ago. I have not used them in years, but it's still there. So this is one that I have to take out. So my summary there for that, if I were to rate these bigger lessons under vendor security is your security. One, every vendor connection is a potential attack path. Two, trust doesn't mean unlimited access. Just because a vendor needs access doesn't mean it needs administrative access to everything. Treat them like users, give applications the minimum permissions they need, review those permissions periodically, and remove access that is no longer necessary. That's what led me to go look at these enterprise apps.

[15:20] Third, monitor applications as well as people. One thing that I started doing recently when I would list the number of user accounts that I was billing for, it wasn't just employees. So every account that was considered active, which included service accounts, admin accounts that I created, even if I created a separate admin account for backup purposes or for something else, those are counted as users under the way that I bill, and I monitor those as well. But what I've not been doing is monitoring these applications. So we have to monitor those just like we monitor users.

[16:07] Number four, and this goes specifically to the ConnectWise breach, because that was a cloud backup. And if you just assume that, well, it's backed up, the backup was successful, everything's good, that's not a good enough strategy. So the backup itself can be intact, but the service that you need to access it may not be. So test your restores and understand recovery options and I m going to now change my backup testing schedule, some of them are once a quarter some of them are worse than that so I ve got to figure out a way to i don't know if i can get it to monthly but something that will test or restore process and even if you use, one of those great Cadillac products that will spin up a, startup screen to show you that it works. I don't know if that's enough anymore. I may be wrong. You can let me know. Number five, have an incident response plan for your vendors. And this is probably a tough one because a lot of times we may not be able to have one if our RMM goes down. Do we have a backup RMM? But the idea is don't wait until something happens to figure out who to call, especially if your vendor is not available 24 7 or it's only a email and you may get a response back in 24 to 48 hours.

[17:35] That may not be acceptable anymore if a client goes down especially if they have something that is in their system managed by one of your vendors that you don't watch, so, again, going back to that inventory.

[17:51] Not only do we need to know which applications they are we need to know which customers might be affected, in some cases, how to disable that integration if we need to move to a different one.

[18:05] So all of this, to summarize it, is, again, vendor security is basically a customer relationship issue. I said earlier, customers don't necessarily care which vendor caused the problem. They care whether their business is protected and whether you can explain what's happening. So, of course, the first thing is be honest. That's absolutely the first thing. Um, the fact that I was able to tell my customers, look, I don't know, but I'll look into it. They've been with me long enough to know that that's true. Uh, I'm not going to lie to them. I'm not going to BS them and say, Hey, look, it's all comfortable. We don't have anything. Um, verify that. So, um, it's about, um, what you don't know and what you're going to do to investigate. So don't promise that everything is safe until you have evidence to support that conclusion, so that was it i didn't want to you know get too much into this only to look at it from a different perspective because i know that you know if you look in the forums and the reddits and stuff everybody was talking about you know uh.

[19:14] We're fixing this our vendors fixing this but nobody mentioned from a client's perspective what this was i did see it which is a pretty good i did see two questions on reddit one of them is you know as an it person what should i be doing, and then there was another one that says I m just a regular person what can i do so at least those questions were brought up um, Sorry, I'm just flipping my notes here to see here. But like I said, I saw those questions and read it. I did not see a lot of things from a customer perspective. So the bottom line is this. The PHP vulnerabilities remind us that we need to know what we're running, keep it patched and verify that our security controls are working. The ConnectWise incident reminds us that protecting our own infrastructure sometimes isn't enough. We have to understand the vendors we trust, the access we give them, and the risk that come with those connections.

[20:17] Oh, I've got this I came up with here. You can outsource software. You can outsource backup services. You can outsource security monitoring. But you cannot outsource your responsibility to understand the risk you're accepting on behalf of your customers. So that I think is where I want to leave it today thank you very much for tuning into this Twitch show and stay tuned for other episodes coming up later that's going to do it folks we'll see you soon and until next time holla!