SolarWinds Breach – May 2021 Update

As with all things security in the IT space, the issue with the SolarWinds Orion breach is that it’s a slow burn that keeps on giving. Given we’re several months post breach now, does that change anything for system admins? We had the chance to catch up with Fabio Viggiani from Truesec once again to talk about the new information that has come out over the past months since the initial video.  

It’s no surprise to find out that while there has been little talk in the news about the issue as of late, certain threats and other issues remain. With the scope of this attack being as large as it is we’re poised to be talking about this and pulling back layers of the onion for some time. So, we’ll be keeping you updated on insights, as we continue to find out more! 

In this video Fabio and I discussed a number of different things as it relates to the SolarWinds hack and IT security in general, including: 

• While it may “feel” like we’re seeing an uptick in security issues over the last couple of months, that may not necessarily be the case.  
• The increasing boldness of attackers by going after build servers and cyber insurance companies. 
• Nation state actor implications 
• A brief timeline of key events leading up to the large, reported Orion breach 
• Why MSPs shouldn’t trust their customer’s environments 
• The urgent need for security monitoring tools 
• And a lot more! 

Without any further wait, you can watch the latest interview with Fabio below.

1:26 – Have we seen an increase in security incidents in the last couple of months? 

6:18 – What new information has come out regarding the SolarWinds breach since the end of December? 

13:32 – Is there any indication that the “perceived” increase in security issues these last couple of months can be tied back to the initial SolarWinds breach? 

16:07 – Does any of this new information change the security playbook for service providers?

Follow-up Interview with Fabio Viggiani on the SolarWinds/SunBurst Hack

————————

I always have mixed emotions bringing you content surrounding a security issue or breach. On the one hand, I’m glad I can bring you (hopefully) useful content. On the other, I often find myself saying…  “Haven’t we been here before?” While that last question did come up while I was preparing to write this article, this one feels different. The scope is larger, there are many lingering questions, and there is the potential for heavy impact.

Whether you keep up on tech news or not, it’s unlikely you’ve gone this far without hearing about the recent breach of SolarWinds Orion in some way shape, or form. Just in case you haven’t though, it’s a doozy. The attack includes very targeted methodologies, information gathering, maliciously signed binaries, and more. On top of that, with the scope and design of the breach, we have the potential to be dealing with the fallout of this attack for some time. That leaves a lot of private IT departments and managed services providers wondering, what is their next move? Are they compromised? How do they tell their customers if they are affected? How do they patch the breach? Many of these questions have left some online communities, such as Reddit’s MSP subreddit, abuzz with questions like this and many an MSP wondering what they do next.

To help provide some information on this subject, I recently reached out to some colleagues at Truesec (Experts in Cyber-Security) and they connected me with Fabio Viggiani who is the Security Team Leader at Truesec. We initially talked about doing a quick 5-10 minutes interview, but Fabio is such a wealth of information on the topic that we ended up discussing the breach for nearly 30 minutes!

If you find yourself asking any of the questions mentioned here, I highly suggest watching my video interview with Fabio below. We discussed the following topics (timestamps can be found within the youtube video)

1:50 – What is SunBurst – the SolarWinds Hack?

4:20 – Who else has been targeted?

6:32 – What happens next? (Attack Stage 2)

8:28 – How are general IT admins affected?

10:51 – How do I know if I’ve been affected?

14:00 – Can the hack change the IT industry?

16:55 – What can companies do to protect themselves from future attacks?

20:03 – How should Service Providers respond?

Interview with Fabio Viggiani on the SolarWinds/SunBurst Hack

Additional Resources

As mentioned in the video, here are some additional resources on this subject

• Truesec Website
• Fabio’s Blog Outlining the Attack in GREAT detail
• Fabio Viggiani on Twitter

Wrap-Up

As I mentioned at the beginning, while it bums me out that I’m writing about yet another security vulnerability in the industry, I’m wildly excited that I can share information, tips, and tricks from a seasoned security expert like Fabio with you! Hopefully, the interview allowed you to take away something that will help your own organization deal with the fallout of this recent breach. That said, was there anything that stood out to you specifically? Any tips that you found extra helpful? If I had to choose one myself I’d say I found Fabio’s advice on not forgetting about the basics to be a REALLY good point. The basics of security continue to be important. Don’t forget about them by being hyper-focused on attempting to guard your organization against a highly complex supply-chain attack like this.

What about you? Anything you felt to be super important? Anything you feel was missing from our talk? Let us know in the comments section below!

The post How the SolarWinds Hack Could Change Data Security Forever appeared first on Altaro DOJO | MSP.

One of the most common types of cyberattacks is one where cybercriminals seek to compromise the victim’s web credentials. Using email-based phishing attacks and increasingly convincing social engineering techniques, victims are tricked into providing their user ID and password for a wide range of cloud-based platforms and applications.

According to Ponemon’s 2019 Global State of Cybersecurity in Small and Medium-Sized Businesses report, over half (57%) of SMBs have experienced phishing/social engineering attacks in the last 12 months, and nearly one-third (30%) have experienced credential theft.

What makes online credentials so appealing to cybercriminals is the access these credentials provide to online banking, Office 365, Azure apps via Azure Active Directory, financial applications, customer data, and more. Gaining access to these kinds of applications and data can be detrimental to SMBs – potentially even causing them to shut their doors.

So, how can you as an MSP help protect your customers from this kind of cyberattack?

The answer lies in Multi-Factor Authentication (MFA).

Before we continue, if you or your customers use Office/Microsoft 365, we have an upcoming webinar you simply can’t be missing. On May 27, Microsoft experts Andy Syrewicze and Symon Perriman are giving a free live demo of the advanced security features of O365/M365 you should be using. Read more here and save your seat.

Now let’s get onto some MFA basics and then talk about how you can incorporate this security control into your service offerings.

What is Multi-Factor Authentication?

Multi-Factor Authentication (MFA) is a security method that uses multiple identifying “factors” to verify a user’s identity, instead of simply relying on the traditional username and password. MFA requires additional factors to identify and authenticate the user. These factors include:

• Text messages to the user’s smartphone
• Sending codes to an alternate email address
• Asking additional security questions
• Using secondary authentication to trusted 3^rd party sources
• Biometrics (such as fingerprint or retina scan)
• Facial recognition
• Security hardware token device
• Security token app on a user’s smartphone
• Certificates

Additionally, depending on the MFA solution being used, details about the when and from where the authentication request can come into play, including location, day/time, IP address, requesting device’s MAC address, etc.

All of these factors – in one form or another – fall into one of three generally-accepted authentication factors:

1. Something you know – This can be information relevant to authentication that the user themselves knows already such as passwords, answers to security questions, etc.
2. Something you have – These are generally represented by physical items the user possesses such as a smartphone, security token, or RFID badge.
3. Something you are – This is where biometrics and facial recognition come into play. This factor uses any part of your personality that can help uniquely identify you.

Office 365 2 Factor Authentication Mobile Sign In

How Does MFA Work?

First off, notice we’re discussing multi-factor authentication. The focus here is for you to use multiple factors with your customers. Why? Because each of these factors on their own can be (and in many cases, have been) hacked or spoofed. Mobile devices have had their SIMs swapped for an attacker-controlled device, passwords can be cracked, even fingerprints have been shown to be spoofable using 3D printing.

With MFA, the user authenticates by providing a number of factors – how many depends on the level of security needed, the individual’s role within the organization, etc. In general, the user first provides their usual username and password. Once provided, they are then presented with one or more additional challenges where the implemented factors mentioned above need to be satisfied.

Multi-Factor Authentication Steps

Where Do You Find MFA?

There are dozens and dozens of software vendors offering MFA. In many cases, it’s offered as part of a larger Identity and Access Management solution – which may be too complex for simply implementing MFA for your SMB customers. Microsoft offers Azure Multi-Factor Authentication to secure access to Azure Active Directory, Office 365, Azure-based VMs, applications, and data, as well as be a trusted authority for third-party cloud applications and platforms. This service is simple enough to scale down to an SMB’s needs. And, as mentioned, there are a number of vendors offering MFA solutions that are simple and cost-effective enough for an MSP.

Office 365 2 Factor Authentication Desktop Sign In

Why is MFA No Longer an Option for MSPs?

Your customers are equally at risk of cyberattacks, data breaches, ransomware attacks, and more. It’s imperative that any kind of external access to applications, platforms, and data is protected via MFA to ensure that cybercriminals can’t leverage compromised credentials to steal data, send emails containing malware, access business details, or commit fraud. Should an attack successfully trick a user into giving up their ID and password, the cybercriminal doesn’t have the additional authentication factors to do anything with the compromised credentials. In essence, the credentials are worthless to the cybercriminal.

It’s important to note that MFA isn’t just for select users at a customer; it’s necessary that every single user is enrolled in and mandated to use MFA to keep your customer organization’s secure.

How to Go about Offering MFA to Your Customers

There are a few ways to do this. The first is to simply absorb the cost of setting up MFA and offer it at no charge. Microsoft Azure MFA has a free version that is a very viable option. If you are offering either Managed Office 365 services or Managed Security services, I’d suggest bundling it in as part of those services. For those SMB customers that are on the larger side and need MFA integration with single-sign-on access to multiple cloud applications, you’ll want to look at vendors like Okta who focus on the integration of their MFA with thousands of existing cloud products and services.

It’s Time to Secure Your Customer With MFA

Multi-Factor Authentication needs to be an embedded part of your service offerings intent on keeping your customer’s applications and data safe from cyberattacks intent on gaining access. By implementing MFA in your customer’s environments, you’ll help to minimize the risk of successful cyberattacks focused on credentialed access.

The post Why MFA is No Longer Optional for MSPs appeared first on Altaro DOJO | MSP.

Windows Server 2008 and Windows Server 2008 R2 reached end-of-life (EOL) on January 14^th, 2020. I don’t know what sort of guidance and pointers that you’ve read or listened to, but it seems to me that most material does a lot of hand-waving and over-simplifying of a complicated topic. I want to offer something with more detail and nuance.

Before we continue, if you run or manage virtual environments that have one or more physical machines or legacy servers that have not been virtualised you can now use Altaro Physical Server Backup to protect these physical machines and keep them safe. Altaro Physical Server Backup is a free server backup freeware solution created to satisfy this need, with the added bonus that it’s free. Download Altaro Physical Server Backup now

What It Means That Windows Server Reached End-of-Life

In its simplest form, end-of-life for Windows Server means:

• The product will receive no further updates, including security fixes
• Microsoft will no longer maintain its list of compatible hardware
• Microsoft will stop testing products for compatibility on EOL operating systems
• Microsoft will no longer maintain components and documentation in software development kits and similar tools. They might remove them entirely.
• You will not have access to support services from Microsoft for the product
• You might lose the ability to exercise downgrade rights to the EOL version

If a product has reached end-of-life, that also means that it has passed its end date for mainstream support. In the case of Windows Server, mainstream support ends five years prior to end-of-life. The end of mainstream support means:

• Microsoft will not develop new features
• Microsoft will not certify new hardware
• Microsoft will not accept further warranty claims
• Microsoft will not issue hotfixes

A few other differences exist, but few that affect a majority of licensees. You can review Microsoft’s somewhat more detailed explanation. If you have more questions, talk to your Microsoft Technology Account Manager (TAM). If you don’t have a TAM, then the differences probably do not apply to you anyway.

What Does NOT Happen Now Windows Server Reached End-of-Life

The above leaves room for a lot of questions. Three major assurances:

• Windows Server 2008 and 2008 R2 will continue to function indefinitely
• Product-specific licenses will not expire
• Third parties can continue to support the products

I will explore further in upcoming sections. If you wondered, “Can I continue to use Windows Server 2008 or Windows Server 2008 R2 after they reach end-of-life?” then I want to assure you that you can.

The Windows Server Support Cycle

Each version of Windows has a clearly defined lifecycle. When Microsoft releases a new product, they publish its milestones: the date of its beginning, the date that its mainstream support ends, and the date that its extended support ends. For some products, they also include an end date for service pack support, although differences in product support practices can make that chart unintuitive. For Windows Server versions through 2008 R2, the product without Service Pack 1 had a different lifecycle than the same product with a Service Pack. As a result, you will see some EOL dates for these two products in 2011 and 2013. As long as you have installed Service Pack 1 on either of these operating systems, the January 14^th, 2020 date applies. You can check the table yourself.

Typically, Windows Server has a five-year mainstream support span followed by a five-year extended support phase. Microsoft can change this for new releases if they want, but they adhere to the published plan once a product ships. This gives every customer a clear opportunity to understand exactly what to expect in terms of support for any given product. They will not have surprises.

Microsoft does make occasional exceptions, but almost always as extensions. These extensions usually have special conditions attached. For instance, they might choose to patch an especially nasty security flaw in an EOLed product. They might provide extended support for an EOLed product, but only in strictly-defined conditions. For very large customers in very special circumstances, they may offer full support for a defined length of time. These things happen most frequently for desktop operating systems. Do not expect any special treatment for Windows Server 2008 or 2008 R2 – almost none of us will get it.

Addressing the EOL of Windows Server 2008 and 2008 R2 in Practical Terms

As usual, the looming date has brought out the Internet punditry in full force. The confusion and uncertainty create a ripe environment for all sorts of fear-mongering, wild speculation, and blanket judgment. The situation deserves a more well-rounded response.

The Pundits are Wrong

I’m sure that you have read at least one article that tries to make you feel like the worst administrator in history if you haven’t eliminated all of your Windows Server 2008/R2 systems by the EOL date. We saw the same ones for Windows XP and Windows Server 2003. Probably had some for the 2000 series products as well, although I don’t personally recall any. First of all, if your major sin is running a server operating system instance past its lifetime, then you’re not even in the bottom 50%.

Most of these article writers have never worked in an internal IT department. Very few have ever worked for a general-purpose outsourced IT provider. Of the handful that have done either, most worked strictly with technologies built for technologists, meaning that they had little or no intersect with line-of-business applications. To them, an “IT Generalist” is a person that has to maintain the SharePoint environment and the Exchange environment, and sometimes even the Windows Servers that they run on! They literally have no idea what an authentic generalist deals with, meaning that they have absolutely no clue what you put up with from your vendors.

When You Must Play Along

To get the responsible guidance parts out of the way, those pundits are not entirely wrong. Some systems really need the upgrade and I don’t expect to see many valid excuses to the contrary. A few things come to mind:

• Windows Server installations that use only Windows Server features (file, print, domain, etc.)
• Installations running only Microsoft application servers
• Servers hosting applications by a vendor that provides migration or upgrade guidance

If you can make the upgrade, make the upgrade.

When You Have to Make Do

Let’s turn to the harder situations. If you’ve never had a vendor that required you to stay on a ridiculously out-of-date version of something to continue using their mission-critical product, then count yourself lucky. At this point, the pundits tend to jump in with proclamations of, “Vote with your wallet,” or, “It’s your fault for not choosing another vendor,” or, “You can just wave your magic ‘the customer is always right’ wand and all the vendors will suddenly do your bidding,” and other ignorant platitudes. Those of us in the trenches know that reality does not bend that way. Personally, I have some major vendors that have no alternatives, and if one ever shows up, switching to them would literally require architects, helicopters, and multi-month construction projects.

Pundits often act like the providers that refuse to move on are insignificant outliers. Obstinance among vendors is common. Just look at the struggles that Microsoft is having with convincing the software manufacturers of the world to stop relying on SMB1. Take a look at the company names on that list; I’m certain that you recognize at least a few. I’m equally certain that you understand the importance of those players in their respective markets. Some of the names that you might not recognize have even greater presence in their verticals. If pressure from Microsoft can’t get these organizations to move off of a thirty-year-old protocol with demonstrably superior alternatives that have been available for well over a decade, what chance does a typical small business have for convincing anyone to abandon a ten-year-old operating system that still functions adequately?

Vendors generally stick with old technologies for a simple combination of two reasons: some technological advances require significant changes, and no company makes significant changes without an equally significant incentive. In 2010, two of my vendors required us to use a Windows NT 4.0 system to integrate with their hardware. The changes to the hardware abstraction layer in Windows 2000 Server placed substantial demands on legitimate driver builders. The products that these vendors made were crucial to our daily operations and they had no competition at all, so forcing us to stay on Windows NT was the path of least resistance for them.

I do not personally know of any comparable technology changes that favor remaining on Windows Server 2008 or 2008 R2. Windows Server 2008 was the last release of Windows Server that still allowed for 32-bit only installations. I can imagine that some software makers want to continue with that. I know of a few that require customers to install their application server software on Windows desktop installations specifically because they still have 32-bit architecture offerings. I wonder if they have even tried to work with a 64-bit installation; the AMD64 architecture became king specifically because it runs both 32-bit and 64-bit code natively. 64-bit Windows on an AMD64-compatible chip should easily handle anything that would run on 32-bit Windows. However, the vendors make the rules for their software, and customers rarely have any choice.

It’s not always a matter of technology. Sometimes it’s laziness or greed. A vendor might not support migrations unless they perform them. If they publish migration guidance, customers might not have the necessary expertise on staff. A provider might prioritize new customers over existing customers, refusing to show up to help out. More commonly, a vendor will perform the migration, but only for an exorbitant charge. Cash-strapped businesses (or cheap business owners) might opt to take their chances with old systems.

Anyway, it doesn’t really matter why you must continue using old operating systems. It happens, vendors don’t care about the plight of their customers and their systems administrators, and we must deal with the consequences.

How to Move Forward from Windows Server 2008 and 2008 R2 EOL

To get started, you must perform your due diligence. Research the upgrade and migration paths for all of your systems. Move the ones that you can. If you need to make a business case to convince your employer, then do that. You have the knowledge that you need. Build a persuasive presentation from the facts:

• Leaving security holes unpatched makes you vulnerable to breaches. Or, perhaps more convincingly, a breach earns you a featured, but distasteful, spot in the news. Using unsupported software makes it that much worse.
• The greater the distance between the installed OS version and the current OS version, the more difficult upgrade and migration paths become
• Hardware failures become unrecoverable
• Support structures become unavailable (e.g. backups, external or replacement expertise)
• Maintenance costs increase
• Old systems might hold other systems back (e.g. connectivity and compatibility)

If you have no way to move forward, then you must hunker down. Essentially, you must build the most impenetrable wall possible.

• If available, use hardware firewalls and VLANs to isolate the system
  • Alternatively, enable and tighten Windows Server’s built-in firewall
• Leverage the isolation techniques native to hypervisors
• Reduce membership in the Administrators and other privileged groups to the minimum
• Maintain locally-installed antimalware for as long as possible
• Institute the strongest possible border protections
• Implement regular monitoring and auditing practices

Most of these suggestions describe general best practices for all server installations. You can find the primary difference in the first bullet point. Even in higher-security organizations that normally segregate server systems, servers typically reside in a flat network with other servers. That creates opportunities for network-aware attacks to move laterally. We use other defenses to combat that, such as firewalls, anti-malware tools, intrusion detection and prevention systems, etc. We also implement operating system patches as quickly as feasible. Once the operating system reaches end-of-life, we lose patches and the antimalware and IPS vendors lose interest. So, the risks of keeping old systems in the same network continually increases. So, regardless of your organization’s overall security requirements, the effort to maintain those old systems also increases.

The Upshot

I assume that only a handful of systems administrators will keep their 2008 and 2008 R2 systems out of sheer laziness. I doubt they do enough research that they’ll even find this article, so I won’t address anything to them. I expect that if you did arrive here, that you have a legitimate problem that requires you to either forge forward without help or stay on an old system against your will. I understand your plight; I’ve been there. My best advice: be a persistent nag. Every single time that you talk to any representative of the vendors that plague you, ask when you can modernize. Until that day comes, hang in there.

The post Windows Server 2008 End of Life appeared first on Altaro DOJO | MSP.

Do you currently use the same TV, phone, movie player, music player or computer that you had 30 years ago? Probably not, since it would be slow, unsecure, and not designed for today’s highly-connected society. Would it surprise you that many companies are still using a network communication protocol that was designed at the same time? If you or your clients are using Server Message Block version 1 (SMB1), turn it off and replace it! Just like your VHS player, SMB1 is also outdated and not designed for the modern Internet, making it highly vulnerable to attacks. In fact, Microsoft has explicitly warned its users against using it since 2017 and has stopped developed any services that depend on it since 2014. If you are a managed service provider (MSP) then part of your responsibility should be to protect your clients, so you need to take this recommendation seriously, patch any vulnerabilities, and bring your customers to the modern era. This blog will help you understand why SMB1 is unsafe, how to detect if it is still being used, and show you ways to mitigate the risks.

Understanding the Server Message Block (SMB) Protocol

The first Server Message Block (SMB) networking protocol, also known as the Common Internet File System (CIFS), was designed in the mid-1980s by IBM, and first adopted by Microsoft in the early 1990s as part of its LAN Manager products. SMB is an application and presentation layer protocol which is used by organizations to share files, printers, ports and other resources. A fundamental problem with SMB version 1 (SMB1) is that it offers no native encryption. This means that a malicious user with a password can see the messages on the network in plain text, giving them the ability to alter the communications sent between servers. They can steal information, reroute traffic to their own servers, crash a server, or inject code which could execute remotely and take control of a machine. Since most networking hardware manufacturers use default passwords, it became easy for bad actors to hijack network traffic from organizations that did not change their passwords.

Microsoft introduced SMB2 in 2006 with Windows Vista / Windows Server 2008 which supported features like larger block sizes, request compounding, caching of properties, durable handles, security signing, greater scalability, symbolic links, and more. In Windows 8 / Windows Server 2012, Microsoft released SMB3 which enabled many new distributed computing scenarios, such as cluster support, transparent failover, SMB direct, directory leasing, network multichannel, and finally native encryption. SMB2 and SMB3 are designed for the modern era and should be used instead of SMB1.

SMB Version 1 Security Vulnerabilities Leaked

In early 2017, the urgency to drop SMB1 became dire as a new series of cyber attacks were exploiting its vulnerabilities. It turned out the National Security Agency (NSA), one of America’s most trusted government organizations, had discovered several critical weaknesses in SMB1 but decided not to report them to Microsoft or other vendors. The NSA was then hacked by a group which leaked dozens of these exploits, and new classes of malware and ransomware began appearing. These vulnerabilities could create kernel pool corruption which would crash servers, or even allow remote code execution (RCE), which effectively let a malicious user take over a server or computer. Some of the most notorious ransomware which used these SMB1 vulnerabilities included WannaCry, NotPetya, Retefe, and EternalBlue, and they have been estimated to have cost businesses billions of dollars in damages.

These vulnerabilities are documented here:

• CVE-2017-0143
• CVE-2017-0144
• CVE-2017-0145
• CVE-2017-0146
• CVE-2017-0147
• CVE-2017-0148

Microsoft publicly criticized the NSA and US Government for its practice of “stockpiling vulnerabilities” which they claimed the government had intended to use for their own benefit. Microsoft says that if the authorities had disclosed these issues to Microsoft in advance, then they could have proactively started patching SMB1 clients before the hackers began exploiting its weaknesses.

Is your Customer using SMB Version 1?

Microsoft has been campaigning their partners and customer to stop using SMB1. They have even challenged many of the top ISVs and IHVs to stop using it by listing vendors still using SMB1, including Apple, Aruba, ASUS, barracuda, Belkin, Bitdefender, Canon, Cisco, Citrix, Dell / EMC, F5, Fujitsu, Hitachi, HP, IBM, Linksys, Linux Kernel, McAfee, NetApp, NetGear, NVIDIA, Oracle, RedHat, Sony, SUSE, Tintri, Toshiba, VMware and Western Digital.

The following section shows a few ways to check if your customer is still using SMB1 so that you can disable it, as documented in Microsoft Knowledge Base Article 2696547. Be aware that after turning off SMB1, a restart of the operating system will usually be required.

Disabling SMB Version 1 on Windows Server

From Server Manager, you can view the list of features which are installed to determine if “SMB 1.0/CIFS File Sharing Support” is installed. From here you can use the “Remove Roles and Features Wizard” to deselect it and remove it from the system.

Figure 1: SMB Version 1 on Windows Server

This can also be done through PowerShell using the following cmdlets.

Disabling SMB Version 1 on Windows Client

Checking for and disabling SMB1 is available on Windows client through the “Turn Windows features on or off” interface.

Figure 2 – SMB Version 1 on Windows Client

This can also be done through PowerShell using the following cmdlets.

Disabling SMB Version 1 at Scale

For MSPs working with larger environments, they can use desired state configuration (DSC) to determine if any server or clients in the environment are still using SMB1. These steps are documented in this Microsoft blog by Ralph Kyttle.

The easiest way to disable SMB at scale is through Group Policy, which will globally change the registry setting for any computers using it. These steps are documented in this Microsoft blog by Troy Arwine.

Microsoft Critical Security Bulletin MS17-010

There are some organizations that must continue to use SMB1, for example, if they are still using older (and often unsupported) versions of Windows, their legacy management software relies on this protocol, or their printers have firmware which cannot be updated. If your business is being forced to use this unsafe protocol, then pressure your vendors to update their software to support to SMB2 or SMB3.

If your customers are still stuck with SMB1, then it is critical that you install Microsoft Security Bulletin MS17-010 – Critical. This is a Security Update for servers and clients with following operating systems:

• Client: Windows Vista, Windows 7, Windows 8.1, Windows RT 8.1, Windows 10
• Server (Full and Core installations): Windows Server 2008, Windows Server 2008 R2, Windows Server 2012, Windows Server 2012 R2, Windows Server 2016
• Older version of Windows client (XP, ME, 2000) and Windows Server (2003) are also vulnerable, but since Microsoft no longer supports these operating systems, there are no public patches available for them. This is a great example of why customers should not use unsupported operating systems like Windows Server 2003. There are rumors that updates are available to some organization who pay for expensive custom support agreements with Microsoft.

You can verify whether this patch has been installed by checking for its KB number, file version, or using PowerShell or WMI. These steps are documented here by Microsoft support.

Now that you understand why using SMB1 is risky, you can appreciate the urgency to upgrade to a newer version as quickly as possible. If you must still use SMB1, but you cannot patch the systems, then make sure that the networks are isolated or they’ll remain vulnerable to ransomware attacks.

The post Security Bulletin MS17-010 | Why MSPs Need To Turn Off SMB1 appeared first on Altaro DOJO | MSP.

Last year we were all dealing from the ramifications of patching the Spectre Meltdown vulnerability. It was many hours of countless “patching maintenance” windows to get all devices in a secure state. This year is the year of Zombieload – a new speculative execution variant that Intel recently reported. The new vulnerability exists so far only on Intel chips leaving AMD and ARM unaffected and can affect chips as old as 2011. The vulnerability was apparently discovered by security researchers and can be used to perform various attacks such as extracting AES keys, accessing data from another VM used by the active or sibling logical CPU. It can even be used to access data from another user session in applications.

A video was recently uploaded to youtube demonstrating how Zombieload can be used to obtain web browsing data from another VM that’s using Tor the anonymous web browser:

Zombieload Vulnerability Patches

Security researchers have revealed a white paper on Zombieload describing the vulnerability in great detail. In their conclusion for remediation, they state that “disabling hyperthreading is the only possible workaround to mitigate Zombieload on current processors.” So expect many of the vulnerability patches to contain various changes to microcode and disable hyper threading.

Intel also reported that their patches will have a minimial performance impact. Still, as IT Pros, these patches MUST be tested before rolling out. Last year there were reported issues with the Meltdown/Spectre patches and Zombieland will most likely go through the same.

Below are some of the patches that the major vendors have released

VMware Patch  – Updates have been released for the following VMware products affected:

• VMware vCenter Server (VC)
• VMware vSphere ESXi (ESXi)
• VMware Workstation Pro / Player (WS)
• VMware Fusion Pro / Fusion (Fusion)
• vCloud Usage Meter (UM)
• 
Identity Manager (vIDM)
• vCenter Server (vCSA)
• vSphere Data Protection (VDP)
• vSphere Integrated Containers (VIC)
• 
vRealize Automation (vRA)

MacOS Patch – Already deployed fix to the latest Mojave and Safari browser updates.

Microsoft Patch – Recommended action is to install OS updates and possibly disable Hyper-threading in order to be completely protected.

HP Workstations Patch – HP is in the process of releasing fixes for each product that is affected by the vulnerability.

Google Patch – Each Google product has its own recommendation:

• Google Infrastructure that runs google products like Youtube, Search, Maps – No customer action needed
• Android – Only Intel-based Chrome OS devices are affected, updates are handled by Chrome OS
• Google Apps/G Suite – No customer action needed
• Google Chrome Browser – Follow OS vendor instructions and keep Chrome up to date
• Chrome OS – No customer action needed
• Google Search Appliance – No customer action needed
• Google Wifi/OnHub – No customer action needed

Citrix Patch – Recommends that all Citrix Virtual Apps and Desktops customers update their OS and hardware. For Citrix hypervisors, there is an update here.

Dell Patch – EMC servers have a BIOS update that addresses the vulnerability.

Linux Distributions

Redhat Patch

Ubuntu Patch

SUSE Patch

Debian Patch

Public Cloud Service Providers

Amazon AWS Patch – “AWS has designed and implemented its infrastructure with protections against these types of bugs, and has also deployed additional protections for MDS. All EC2 host infrastructure has been updated with these new protections, and no customer action is required at the infrastructure level. ”

Azure Patch – “Microsoft has deployed mitigations across all our cloud services. The infrastructure that runs Azure and isolates customer workloads from each other is protected. This means that a potential attacker using the same infrastructure can’t attack your application using these vulnerabilities.”

Google Cloud Patch – “The infrastructure that runs Google Cloud and isolates customer workloads from each other is protected against known attacks. But for some Google Cloud products, clients will need to patch their runtime environments.”

Microsoft PowerShell Speculation Control Module

One of the most popular PowerShell modules that exist on the PowerShell gallery is called Speculation Control.

To install the module run the following command on an administrative PowerShell console to install the module from the PowerShell Gallery:

Install-Module -Name SpeculationControl

After the module is installed on the machine to check, run the following command to receive a report:

Get-SpeculationControlSettings

In the screenshot above, there are several patches required to squash these vulnerabilities. As an MSP, manually running this against each client device would take hours and hours. Thankfully we can use PowerShell Remoting to script this out. Copy the following to a notepad and save it as a .PS1:

[CmdletBinding()]
 
param (
 
[parameter(Mandatory=$true,Position=0)]
 
[string[]]$computer,

[parameter(Mandatory=$true,Position=1)]
[pscredential]$credential

)


$report =@()

Foreach($computer in $computers ){

    $status = Invoke-Command -ComputerName $computer -ScriptBlock{

        Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force | out-null

        Install-Module -Name SpeculationControl -force | out-null

        Get-SpeculationControlSettings

      

    } -Credential $credential

    $report += $status


}


$report

Now, we can open up an administrative command prompt, save the credentials to a variable, we will use this to pass through the script parameter. This credential must have access to the devices we want to run this script against. So for a bunch of devices on a domain, you could input the domain admin credentials:

$credential = get-credential

Now that we have our credentials, we need a list of computers to run against. If you have a lot of devices, you could make a list in a notepad and then grab the contents of the text file using Get-Content cmdlets. However, in this example I’m just going to create a list by using comma separated strings:

$computers = "192.168.0.13","192.168.0.15"

Now that we have the two variables for our required parameters, we can run our script by simply typing out the path to our .PS1 file in an administrative PowerShell console. Type in the following syntax using both variables:

C:\Scripts\SpeculationControlRemote.ps1 -Computer $computers -Credential $credential

Each device will install the PowerShell package manager NuGet if it doesn’t already have it, then install the SpeculationControl module from the PowerShell Gallery and run the command to create the report. We get a nice output like the following with the PSComputerName property to distinguish each node:

This can be extremely useful for providing a report to a client for assurance that their systems are completely patched and taken care of.

Recommended Actions for Managed Service Providers

With the evolution of IT, we are starting to see the same song and dance each year where a new vulnerability comes out and IT Pros must spend hours patching them. For Managed Service Providers this is a great situation to show clients the worth of hiring an MSP to manage and protect their IT environment so they don’t have to hassle with issues like this. Take the time to educate clients on the vulnerability and the risks associated. Then explain to them the work involved so they can see the value of what they are paying for. I highly recommend taking the next week or two to test out each vendor patches before rolling them out into production, especially for clients. Last year there were bugs in some of the patches for Meltdown and Spectre so we should be cautious in the beginning. From a cost and profitability perspective of running an MSP, the more time and maintenance a client’s environment requires, the more engineering resources they consume which means less overall profitability. Because of this, taking the time to automate the vulnerability patching for each customer can become a huge cost saver. It’s only a matter of time until the next security vulnerability is discovered and we’ll have to do this again, so automate the remediation steps as much as possible.

What about you? Did you have good success patching the spectre vulnerabilities last year? Do you expect this one to create some challenges? If so what challenges do you foresee? Let us know in the comments section below!

Thanks for reading!

The post Zombieload – The New Vulnerability MSPs Need to Worry About appeared first on Altaro DOJO | MSP.

Did you think ransomware had gone away? Think again.

It has perhaps declined since last year but is still very much alive and well, and as always, attackers have new tricks up their sleeves. Consider the below impacted items.

• Infected Endpoints – For every workstation that ransomware can infect, the cybercriminal has that much more able to find and encrypt data valuable enough that the ransom will be paid. According to a recent KnowBe4 report, the average ransomware attack infects 16 workstations, which clearly demonstrates the effectiveness of phishing campaigns. (Another reason to train your customers how to spot them!)
• Infected Servers – While servers are less susceptible to infection, because they are generally not used for reading email (a top entrance medium for ransomware), they too are often caught up in an attack. According to the KnowBe4 data, the same ransomware attack infects 5 servers in addition to the workstations.
• Impacted Data – In the end, ransomware authors program their malware to seek out the data they believe will be valuable to the victim organization. It can exist on the infected workstations or servers, or even exist across mapped drives or SMB connections on other systems that are, otherwise, infected. According to KnowBe4, 97% of ransomware targeted Office-related files, regardless of the system they were on.
• Even Services like O365 can be affected – Consider how many organizations today are using exchange online. A vast majority of MSP supported customers fall into this category and the email data of these organizations has become a large target or ransomware. While less likely than on-premises infrastructure, there have been reports in the last year or two of Office 365 being affected from time to time, adding to the headache caused by ransomware.

Ransomware’s reach within an organization is quite remarkable. It affects a large number of endpoints and servers, and material amounts of data, both on-prem and potentially in cloud storage. This level of potential impact warrants being ready to address an infection, should it occur, and occur it will eventually.

There are several mitigation techniques available today to help deal with the effects of a ransomware attack, but in the end, your ace-in-the-hole will always be your backup and restore operations.

So, how do you plan a ransomware recovery strategy?

The largest unknowns are what endpoint will be the entry point and where the data is that will be impacted. Depending on the size of your customer’s organization, this may be pretty much impossible to determine. Instead, what you need to do is to instead look at this from the business perspective and work backward. In addition to the usual backup targeting exercise you’ll go through with your customer, below are 5 steps to take to properly leverage backups as part of a ransomware response effort as well.

1. Identify What Data is Important – Instead of reacting to a ransomware scenario and hoping to find that you have a very recent viable backup from which to recover, identify those data sets that are most valuable, critical, proprietary, compliance protected, etc. now. Determine what kind of recovery objectives that data requires and put a backup plan in place to be able to restore in the case of a ransomware infection. According to the KnowBe4 report listed above, 61% of organizations recovered server-based data from backup. You don’t want to be part of that other 39%
2. Identify What Endpoints are Important – Even if you pay the ransom, you still, technically, have malware sitting on one or more of your endpoints. Every single infected endpoint needs to be put back into a known-good (read: pre-ransomware infection) state, and more realistically, fully wiped and redeployed. By identifying the ones owned by users whose productivity is critical to the organization (e.g. your executive team), you can build a backup plan for those machines to facilitate a fast recovery from backup as well.
3. Identify What Servers are Important – This should really already be in place due to the critical nature of the systems. If you already have a backup strategy that allows for fast recovery of critical systems, simply walk through a ransomware infection and/or encrypt scenario and ensure you have the ability to recover. You are already doing regular quarterly test restores right? If not, this should be added incentive to do so!
4. Determine How You Can Recover Non-Essential Endpoints – For those endpoints deemed non-critical, you still need to have a plan on how you will bring them back into the same known-good state. It’s either restore, reimage, or rebuild. You may also have a method of restoring or reimaging a gold configuration and update using scripts or a systems management solution, whatever solution you have here, it needs to be defined now, and not during the aftermath of a ransomware attack.
5. Protect the Backups Themselves – Most backup systems on the market today use disk as the most common medium. The days of tape are long gone. With that said, an online medium such as disk can make backup data a prime target for this type of attack. With that in mind, you need to make sure you’re following the best practice of (Just-Enough-Admin) for your backup operations, in that only the rights needed to get the job done should be assigned to the user, and service accounts in use to provide the backup services.
6. Have an Offsite Copy – Should the worst happen, and your on-prem backups also become encrypted, it will serve you well to have an offsite, disjointed, location that contains a copy of the backups. The big thing here is that it should not be easily accessible from your production network. Proper use of application service accounts, file/folder permissions, and network topology should make it much more difficult for an attacker to reach these offsite backups, let alone find them.

Wrap-Up

The concept of ransomware recovery isn’t really that difficult; it’s more about proactively having an understanding of what’s important to your customer, following industry best practices, and putting a plan in place to recover, just like in any other disaster scenario. An ounce of prevention is worth a pound of cure.

To learn more about creating a solid Backup-as-a-Service to offer customers read this free ebook

Are you using similar steps in your ransomware recovery strategy? Has it worked well for you? Let us know in the comments section below!

The post 6 Steps to Using Backup for Ransomware Recovery appeared first on Altaro DOJO | MSP.

KeePass is one of the most widely used password management tools used today in IT. It’s simple to use and secure which meets the needs of most businesses. Also, the fact that it’s open source goes a long way for the community to trust its code and ensure it’s not stealing customer data. It is one of the top free tools an MSP can use for password management. To top it all off, there is a PowerShell module available for automating and managing KeePass databases. This is incredibly useful for MSPs as they usually have a tough time tracking all the passwords for their clients and ensuring each engineer is saving the passwords to KeePass for each project. Being able to automate the deployment of an ESXi host, generate a new password, then automatically store it without any human intervention is a godsend. I can’t count how many times I’ve run into an issue where a password was nowhere to be found in KeePass or was typed incorrectly into KeePass. Quite simply, if you’re an MSP managing a large number of systems, you need KeePass, and you need to automate it. Here’s how.

Getting Started Automating with KeePass

If you don’t have KeePass, you can download it here. Also, you can get the module from the PowerShell Gallery here. Or on a Windows 10 machine open up an administrative command prompt and type:

Install-Module -Name PoShKeePass

Now when we run a quick Get-Command search, we can see we have our new KeePass functions and we’re ready to go!:

get-command -Name *kee*

Connecting With KeePass

First, we need to establish our connection to our KeePass database. I have a KeePass database created in a folder, notice the .kdbx file extension which is the database extension used for KeePass:

Now we will use the New-KeePassDatabaseConfiguration cmdlet to set up a profile for the connection to the .kdbx file. We will also use the -UserMasterKey parameter to specify that this KeePass database is set up to use a Master Key. There several different ways of configuring authentication to a KeePass database, but for the purpose of this demo we are going to make it simple and use a MasterKey which is just a password that is used to access the database:

New-KeePassDatabaseConfiguration -DatabaseProfileName 'LukeLab' -DatabasePath "C:\Temp\KeePass\Database.kdbx" -UseMasterKey

If I run Get-KeePassDatabaseConfiguration I can see my database profile is set up and points to the .kdbx:

Generating Passwords with KeePass

One really cool feature of the KeePass PowerShell cmdlets is that we can use KeePass to generate a password for us. This is very useful when setting unique passwords for each server deployment. This can be done with regular PowerShell coding, however, the KeePass modules allow us to generate a password in 1 line and on the fly without any extra coding. We will create our password variable and use the New-KeePassPassword cmdlet to generate a password. The parameters are used to specify the type of password complexity that we want:

$password = New-KeePassPassword -UpperCase -LowerCase -Digits -SpecialCharacters -Length 10

Now when we inspect our $password variable we can see that its a secured KeePass object:

Now let’s upload a new entry to our KeePass database. Let’s say we are deploying an ESXi host and want to generate a random password and save it to our KeePass database. We will use the New-KeePassEntry cmdlet and specify our profile “LukeLab” that we set up earlier. We are also going to use our $password variable as the password profile that we want to use for the password complexity requirements. Then we get prompted for our Master Key password:

New-KeePassEntry -DatabaseProfileName LukeLab -Title 'ESXi1' -UserName root -KeePassPassword $password -KeePassEntryGroupPath Database/ESXi

Now, when we open up our password database we can see our new entry and a randomly generated password:

Updating A KeePass Password

Password rotating is now becoming a normal standard within IT. Security is bigger than ever and the need to change passwords every so often has become a necessity. Luckily, with KeePass and PowerShell, we can create scripts that automate the process of changing our ESXi host password and then update the new password in KeePass. We start by collecting the current KeePass entry into a variable by using the Get-KeePassEntry cmdlet:

$KeePassEntry = Get-KeePassEntry -KeePassEntryGroupPath Database/ESXi -Title "ESXi1" -DatabaseProfileName "LukeLab"

Next, we use the Update-KeePassEntry cmdlet to update the entry with a new password.

Update-KeePassEntry -Title 'ESXi1' -UserName root -KeePassPassword (New-KeePassPassword -UpperCase -LowerCase -Digits -SpecialCharacters -Length 10) -KeePassEntryGroupPath Database/ESXi -KeePassEntry $KeePassEntry -DatabaseProfileName 'LukeLab'

Now when we look at our password entry for ESXi1 we can see it has been updated with a new password:

Now let’s update our ESXi system by obtaining the secure string from the new Entry and changing the password on the ESXi Host.

We save our entry to a variable again:

$KeePassEntry = Get-KeePassEntry -KeePassEntryGroupPath Database/ESXi -Title "ESXi1" -DatabaseProfileName "LukeLab"

Now we have our password as a secure string. If we look at the properties using $keepassentry we can see the secure string object is there:

So now we can use this variable to create a credential object and pass that along to a script and change the ESXi password to this new password:

$newcreds = New-Object System.Management.Automation.PSCredential("Root",$keepassentry.password)

And, just to prove it, I can use the GetNetworkCredential method and show the decoded password is the same that is in our KeePass:

Wrap-Up

This is a very powerful module as it allows IT Pros to automate the way they manage and enter passwords. There are many more capabilities than the KeePass module allows us to do and the module author is expanding on more and more advanced features. This is an exciting time to be in IT where we are able to use an assortment of PowerShell modules to accomplish feats that were near impossible only a few years ago.

What about you? Do you see this being useful for your engineers? Have you used KeePass for password tracking before? Let us know in the comments section below!

Thanks for reading!

The post Building PowerShell Tools for MSPs: Automating KeePass appeared first on Altaro DOJO | MSP.

At a time in our industry’s history where sharing any kind of personal data inappropriately can put a company out of business, the thought of automatically sending data to a vendor can understandably raise a few red flags. And yet, at the same time, the idea of being able to collect data around usage, issues, and feedback is easily achieved and is highly desirable by vendors. Often referred to as telemetry data, this information can be invaluable to a vendor to get key insights into their products in an effort to make them better.

So, where exactly should you stand on the topic of sending telemetry data?

On the one hand, you definitely want the products you use to rectify issues you’re having and to improve overall. But on the other, there’s a necessary concern around what kind of data is being sent and how is it being used.

Depending on the product in question, each vendor may implement a vendor telemetry program a bit differently. In general, you may be surprised that most large vendors have a hard time getting enough meaningful telemetry, and a lack thereof will make it difficult for that vendor to make improvements and fix bugs for certain use-cases. In this article, I’m going to use Microsoft’s telemetry program as an example, but keep in mind that, before you turn on the sending of any telemetry data, you should do your due diligence around what’s being sent and to whom by the vendor in question.

1. Telemetry data is (generally) non-identifiable – Because the goal of most vendors is to get information about their product, the data is usually focused squarely on their product, it’s current state, errors, issues, etc. In the case of Microsoft (who uses 4 telemetry levels), they specifically attempt to avoid collecting personally-identifying data. Think about it; it becomes a legal nightmare should that data be breached.
2. Telemetry data can include personal data – In certain circumstances, additional data is necessary to provide proper context. Take the case of an application crashing. If you were to want the vendor to understand the nature of the crash, they may need a copy of the data file you were working on. In Microsoft’s case, the use of the Full telemetry setting gives them permission to collect “extra” data as part of the investigation (which can include personal documents, according to their documentation).
3. You should look for ways to control the process – The simplest way to control telemetry is to decide whether to participate or not. It doesn’t affect your use of the product, nor your support; it’s merely a way for vendors to better understand how their product acts in the field. If possible, look for ways to specify what kind of data is allowed to be sent. Microsoft provides some Group Policy settings to manage this so that security and privacy can be maintained as necessary within your organization

Remember: Pay Special Attention to Compliance – While most big vendors like Microsoft will try to fashion their telemetry gathering in a way that doesn’t violate privacy laws like HIPPA, the onus is on you to verify that is actually the case for each and every vendor that you or one of your customers uses. If there are any questions whatsoever on this bullet point, it would be safest to just turn off telemetry gathering.

Telemetry: Helpful or Dangerous?

In concept, the idea of sending telemetry data is rather benign and even helpful in the long run. But it’s up to you to review each vendor’s stance on telemetry privacy looking specifically at 6 aspects of their program:

• What specific kinds of data are being sent?
• Is there ever a scenario where personal data or documents are sent?
• Can you limit/control the process?
• What vendor teams have access to the data?
• How is it being used?
• Are there compliance concerns?

Telemetry programs are designed to be helpful, so it’s good to at least consider participating. But, be certain to ask questions about the nature of the data collection and its use. Based on the answers to these questions, you can quickly determine whether you wish to participate and to what degree.

Resources

To aid you in finding relevant information regarding telemetry, I’ve placed a few useful links in the section below. These links will either explain how a given vendor handles telemetry data, or how to configure it for yourself or your customers.

• Microsoft Diagnostic Data in Your Organization
• Information Contained within VMware Diagnostic Data
• HP Privacy Statement
• Dell Country-Specific Privacy Policies
• IBM Privacy Statement
• Cisco Privacy Statement

Wrap-Up

While the gist of today’s article may give you the sense that vendor telemetry is bad, it really isn’t when monitored and review properly. Again, without proper telemetry, your core vendors have a much harder time improving the products and services you sell and use, so it makes sense to provide telemetry where applicable.

How about you? What are your thoughts on vendor telemetry? Does your MSP have a stance on this issue? Let us know in the comments section below!

Thanks for reading!

The post 3 Important Things You Should Know About Vendor Telemetry appeared first on Altaro DOJO | MSP.

Earlier in the year, Spectre and Meltdown caused a huge uproar within the IT community as people scrambled to ensure they were not at risk from the discovered vulnerabilities – but what about the latest security risk: Intel Foreshadow?

Being a Systems Engineer, I’ve personally spent my fair share of time patching hypervisors and OSes due to these cleverly crafted vulnerabilities. If you’ve been paying attention to the recent IT Security news, you’ve probably heard about the new vulnerability called Foreshadow or the L1 Terminal Fault issue. As an MSP it is critical to patch against these type of vulnerabilities as they are very dangerous, especially for multi-tenant environment services. The speculative execution vulnerabilities can allow one to obtain data from the underlying hardware that is hosting the multi-tenant infrastructure.

What is the Foreshadow Vulnerability?

As a high-level explanation of this newly found vulnerability, Foreshadow is similar to the Meltdown vulnerability in that it allows applications to access the contents in kernel memory. Foreshadow takes advantage of Intels SGX (Software Guard Extensions) feature which is available in new Skylake processor architecture. This new feature enables Trusted Execution Environments, that are secured environments in memory where data can be contained and protected. The Foreshadow attack will attempt to reach this encrypted data and use speculative execution to access it before the processor determines that the attack doesn’t have permission to access it.

MSP Checklist for Internal and Client Environments

As an MSP it is imperative to patch against the Foreshadow vulnerability. You may be wondering where to start and how to proceed with securing your clients’ environments. Below is a checklist that contains recommended steps as well as links to patches for each vendor:

1. Check for Known Risks

Start by inventorying your infrastructure for processors that are at risk to this vulnerability. According to Intel, as of right now, the following platforms are potentially impacted:

Intel® Core i3 processor (45nm and 32nm)
Intel® Core i5 processor (45nm and 32nm)
Intel® Core i7 processor (45nm and 32nm)
Intel® Core M processor family (45nm and 32nm)
2nd generation Intel® Core processors
3rd generation Intel® Core processors
4th generation Intel® Core processors
5th generation Intel® Core processors
6th generation Intel® Core processors
7th generation Intel® Core processors
8th generation Intel® Core processors
Intel® Core X-series Processor Family for Intel® X99 platforms
Intel® Core X-series Processor Family for Intel® X299 platforms
Intel® Xeon® processor 3400 series
Intel® Xeon® processor 3600 series
Intel® Xeon® processor 5500 series
Intel® Xeon® processor 5600 series
Intel® Xeon® processor 6500 series
Intel® Xeon® processor 7500 series
Intel® Xeon® Processor E3 Family
Intel® Xeon® Processor E3 v2 Family
Intel® Xeon® Processor E3 v3 Family
Intel® Xeon® Processor E3 v4 Family
Intel® Xeon® Processor E3 v5 Family
Intel® Xeon® Processor E3 v6 Family
Intel® Xeon® Processor E5 Family
Intel® Xeon® Processor E5 v2 Family
Intel® Xeon® Processor E5 v3 Family
Intel® Xeon® Processor E5 v4 Family
Intel® Xeon® Processor E7 Family
Intel® Xeon® Processor E7 v2 Family
Intel® Xeon® Processor E7 v3 Family
Intel® Xeon® Processor E7 v4 Family
Intel® Xeon® Processor Scalable Family
Intel® Xeon® Processor D (1500, 2100)y

2. Patch EVERYTHING and TEST

You need to quickly protect all aspects of your system that are vulnerable in a tested environment FIRST. Do not skip this step. Last time there were many performance issues due to the Meltdown and Spectre patches so make sure you do yourself and your customers a favor and run any required patches through a test environment before deploying to any production devices. Below are the following major vendors that have released a patch for their systems. Also note, at the time of writing this article Microsoft has stated that Hyper-V servers with hyperthreading enabled can potentially weaken the microcode and OS updates that they released. If you are running Hyper-V be sure to follow this flowchart here along with each mitigation step listed here to best secure your environment based on the Server OS and features enabled:

VMware
Dell
HP
Cisco
Oracle
SUSE
Ubuntu
Debian
Gentoo
Microsoft

3. Patch Production Systems

After thoroughly testing any recommended patches and using them in some sort of “burn-in” test period, you are ready to patch production systems. Make sure to apply the microcode firmware updates for your hardware, as well as the OS patches where applicable.

4. Check Cloud Status

The three major public cloud vendors have already stated that they have deployed patches to their underlying hardware against this vulnerability so any systems utilizing these platforms are considered in very minor risk:

Azure
Google Cloud
AWS

5. Keep up to Date

Stay informed on the issue just in case there are further developments. A site has been created in order to inform people about the issue. As an MSP, it is important to keep your clients aware of the situation and that you are actively protecting their systems by providing regular updates on the current vulnerability status.

Whats Next?

As practitioners of the Professional IT scenery, we are once again tasked with undergoing yet another wave of vulnerability patches. Unfortunately, we are running through the same motions as we did with the Spectre and Meltdown incident. Is this going to become a normal practice for the IT World? If so, we are going to start to see even more and more emphasis on Security as critical vulnerabilities like these consume quite a large amount of time and call for more outage/maintenance windows. By following this checklist above, you will have a good success rate with patching your customers’ environments to prevent yet another speculative execution vulnerability.

Let me know in the comments below your experiences with the remediation of this new vulnerability and whether you run into any issues in doing so! We want to hear about it if you do!

The post MSP Response Checklist for the Intel Foreshadow Vulnerability appeared first on Altaro DOJO | MSP.

Let’s face it, no one likes Windows Updates – least of all Managed Service Providers. However, there is a way to make the process less tedious: through automation.

For MSPs managing Windows Updates for clients is always messy. No matter what patch management solution your using, it is inevitable that Windows Updates will still cause you headaches. Whether there is a bad patch that gets rolled out to a number of clients, or there is the never-ending burden of having to troubleshoot devices where Windows Patches just aren’t installing properly. Luckily, with the magic of PowerShell and the help of the PowerShell module PSWindowsUpdate we can manage windows updates in an automated fashion allowing us to develop scripts that ease some of our Windows Update pains.

How to Install PSWindowsUpdate

PSWindowsUpdate was created by Michal Gajda and is available via the PowerShell Gallery which makes installation a breeze. To install PSWindowsUpdate, all we have to do, if we are running a Windows 10 OS, is open up a PowerShell cmd prompt and type in the following syntax:

Install-Module -Name PSWindowsUpdate

If we want to save this module off and put it on a network share so that other servers can import and run this module, then we will use the save-module cmdlet:

Save-Module -Name PSWindowsUpdate -Path

Using PSWindowsUpdate

Now that we have the module installed we can now run Windows Updates via PowerShell. There are a numerous amount of actions we can perform with this module, but for this post, we will go over the basics. We can use the Get-WindowsUpdate cmdlet to fetch a list of updates that are available for our machine. In my example, I have the module installed on a workstation and I run the following syntax to get the list of windows updates applicable to my machine:

Get-WindowsUpdate

Now that I can see what updates my machine is missing, I can use the -Install parameter to install the updates. If I wanted to install all available updates and automatically reboot and afterward, I would use the -autoreboot parameter and the syntax would look like this:

Get-WindowsUpdate -install -acceptall -autoreboot

if i want to just install a specific KB i can use the -KBArticleID parameter and specify the KB Article Number:

Get-WindowsUpdate -KBArticleID KB890830 -install

In the screenshot, you can see that we get a confirmation prompt and then each update step is listed as it occurs. If we wanted to remove an update from a machine we could use the Remove-WindowsUpdate cmdlet and specify the KB with the -KBArticleID parameter. I will use the -norestart parameter so the machine does not get rebooted after the patch is uninstalled:

Remove-WindowsUpdate -KBArticleID KB890830 -NoRestart

If we want to check available windows updates remotely from other computers, we can simply use the -ComputerName parameter:

Continuous Update Script

PSWindowsUpdate can be used in deployment scripts to make sure Windows is completely up to date before it is placed in production. Below I have created a script that will deploy all available windows updates to a Windows Server and restart when complete. Right after the restart is done, the update process can be started again and repeats itself until there are no more windows updates left. This is a very useful script to use for VM deployments. Unless you have the time to ensure your VM templates are always up to date every month, there are almost always going to be Microsoft Updates to install when deploying new Windows Virtual Machines. Also, the process of ensuring that all available Windows Updates on a system are installed can be a very time-consuming task as we all know Windows Updates isn’t the fastest updater in the world.

This script requires that you run it from an endpoint that has the PSWindowsUpdate module installed, it also should be run with an account that has local administrator permissions to the remote server that it is managing windows updates on. Essentially it will install PSWindowsUpdate on the remote server via PowerShell get and will use the cmdlet Invoke-WUJob which uses task scheduler to control windows updates remotely. We have to use Task Scheduler because there are certain limitations with some of the Windows Update methods that prevent them from being called from a remote computer.

Copy this script to a notepad and save it as a .ps1. I save mine as install-windowsupdates.ps1:

<#
.SYNOPSIS
This script will automatically install all avaialable windows updates on a device and will automatically reboot if needed, after reboot, windows updates will continue to run until no more updates are available.
.PARAMETER URL
User the Computer parameter to specify the Computer to remotely install windows updates on.
#>

[CmdletBinding()]

param (

[parameter(Mandatory=$true,Position=1)]
[string[]]$computer


)

ForEach ($c in $computer){

    
    
    #install pswindows updates module on remote machine
    $nugetinstall = invoke-command -ComputerName $c -ScriptBlock {Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force}
    invoke-command -ComputerName $c -ScriptBlock {install-module pswindowsupdate -force}

    invoke-command -ComputerName $c -ScriptBlock {Import-Module PSWindowsUpdate -force}

    Do{
        #Reset Timeouts
        $connectiontimeout = 0
        $updatetimeout = 0
        
        #starts up a remote powershell session to the computer
        do{
            $session = New-PSSession -ComputerName $c
            "reconnecting remotely to $c"
            sleep -seconds 10
            $connectiontimeout++
        } until ($session.state -match "Opened" -or $connectiontimeout -ge 10)

        #retrieves a list of available updates

        "Checking for new updates available on $c"

        $updates = invoke-command -session $session -scriptblock {Get-wulist -verbose}

        #counts how many updates are available

        $updatenumber = ($updates.kb).count

        #if there are available updates proceed with installing the updates and then reboot the remote machine

        if ($updates -ne $null){

            #remote command to install windows updates, creates a scheduled task on remote computer

            invoke-command -ComputerName $c -ScriptBlock { Invoke-WUjob -ComputerName localhost -Script "ipmo PSWindowsUpdate; Install-WindowsUpdate -AcceptAll | Out-File C:\PSWindowsUpdate.log" -Confirm:$false -RunNow}

            #Show update status until the amount of installed updates equals the same as the amount of updates available

            sleep -Seconds 30

            do {$updatestatus = Get-Content \\$c\c$\PSWindowsUpdate.log

                "Currently processing the following update:"

                Get-Content \\$c\c$\PSWindowsUpdate.log | select-object -last 1

                sleep -Seconds 10

                $ErrorActionPreference = ‘SilentlyContinue’

                $installednumber = ([regex]::Matches($updatestatus, "Installed" )).count

                $Failednumber = ([regex]::Matches($updatestatus, "Failed" )).count

                $ErrorActionPreference = ‘Continue’

                $updatetimeout++


            }until ( ($installednumber + $Failednumber) -eq $updatenumber -or $updatetimeout -ge 720)

            #restarts the remote computer and waits till it starts up again

            "restarting remote computer"

             #removes schedule task from computer

            invoke-command -computername $c -ScriptBlock {Unregister-ScheduledTask -TaskName PSWindowsUpdate -Confirm:$false}

             # rename update log
             $date = Get-Date -Format "MM-dd-yyyy_hh-mm-ss"
             Rename-Item \\$c\c$\PSWindowsUpdate.log -NewName "WindowsUpdate-$date.log"

            Restart-Computer -Wait -ComputerName $c -Force

        }
   

    }until($updates -eq $null)

    

    "Windows is now up to date on $c"

}

Now that you have your .ps1 file created, remember the location where you saved it:

We will open up an administrative PowerShell console and type in the following command in order to deploy all available windows updates on our server “Server3”:

PowerShell "C:\users\Administrator\Documents\install-windowsupdates.ps1" -computer Server3

A remote connection is established with Server3 and we install the module remotely. Then we perform a get-windowsupdate to get a list of any available windows updates. Then we invoke the server to install those updates:

You can see in the screenshot, that after the updates are installed and the server is rebooted, the script picks back up again and starts the process of verifying if there are any new updates available, if there are, it will run through the steps of installing them again. Once there are no more available updates, the script will stop:

Wrap-Up

As you can see, this method is quite useful in several different situations. It’s simply another tool you can add to your toolbox to help your team assist your customers. More efficient operations is a good thing for everyone!

What about you? Do you have Windows Update stories to share? Any particularly favorite workarounds? Has this post and script been useful to you? Let us know in the comments section below!

Also, I’ve written a lot of automation for MSPs here. If you like this, check out the following blog posts:

Automation for MSPs: Getting started with Source Control

Automation for MSPs: HTML Tables

Automation for MSPs: Advanced PowerShell Functions

Thanks for reading!

The post Building PowerShell Tools for MSPs: Automating Windows Updates appeared first on Altaro DOJO | MSP.