Exposure management requires open collaboration across the security ecosystem to solve difficult customer problems. An August 2022 technical issue identified by Microsoft and behind-the-scenes resolution among multiple technical teams over a weekend demonstrates how proactive collaboration can benefit customers.
The work cybersecurity professionals do every day to prevent an event from happening rarely gets the headlines. Yet, it’s just as significant as the work we do to respond to an incident as it’s happening. Such was the case in the story we’re about to share. It’s one of those events in which an ounce of prevention was worth a pound of cure, and speaks to the value of taking a proactive, preventive approach. It also speaks to the interconnected nature of cybersecurity technologies and demonstrates how effective collaboration between vendors ultimately benefits users.
In mid-August 2022, Microsoft tech support had been fielding an uptick in reports of users not being able to access Azure-hosted Office 365 services. By the end of the week, they had identified a common theme among customers that were using both Tenable and Microsoft products in their environments., When Tenable vulnerability scans were run on Windows machines joined to Azure Active Directory (AAD), the machines were being negatively impacted.
Using established partnership channels, Microsoft reached out to points of contact within Tenable Research in the afternoon of Saturday, Aug. 20. By Saturday evening, engineering team members from both Tenable and Microsoft were collaborating on identifying the root cause of the issue and continued communications through the night.
On the next morning’s status call, a shared customer — who was willing to engage on this issue — was identified and all parties collaborated throughout the day to determine the issue and test proposed solutions.
By late Sunday night, a draft workaround was verified in the customer’s environment to avoid the identified file contention issue, and commitments were made by Tenable and Microsoft to have the interim solution and joint communication released on Monday.
The incident was not about a vulnerability, but instead was a file contention situation where the operating system locks the file or, in some instances, deletes the file when competition for a resource occurs.
Within 48 hours of initial engagement from Microsoft, an updated Tenable plugin solution was released to our customers through our automated feed. Throughout the following week, Tenable and Microsoft teams stayed in constant contact to address customer questions, explore engineering alternatives, if needed, and coordinate communications for our shared customers.
Transparency and open collaboration matter
Whilst the whole incident was generally invisible to the user, it displays the reality of what can happen with multiple security tools running in a shared customer environment.
However Tenable was able to remain ahead and abreast of this issue thanks to the established and ongoing relationships with our industry partners. During the whole issue, we had an open communication with Microsoft and were in frequent discussion with them during and after the initial response.
Ultimately, established engineering-level relationships, transparency around the issue and shared commitment to finding solutions greatly reduced the impact of this event. Only a few of Tenable’s customers reported the incident to us, and resolving it was relatively straight-forward.
Coordinated communications from both Tenable and Microsoft provided customers with quick solutions in an updated Tenable plugin to avoid future file contention. Also guidance was provided from Microsoft on how to update the Azure AD BrokerPlugin to restore user access.
This resolution to the situation was primarily achieved because of Tenable Research, our relationships, our ability to respond rapidly and our commitment to providing the best knowledge for our customers. Established partnerships, transparency, and a shared customer-centric commitment to security were the keys to this successful rapid response event. While it’s impossible to measure the value of making sure an incident did not happen, we believe such proactive steps are fundamental to helping organizations around the world reduce risk without sacrificing performance. We hope that by shining a spotlight on the value of preventive security efforts we can provide other security professionals with a means to articulate the value of their own exposure management practices.
Learn more
- For additional details, customer advisories and joint communications, please see the Tenable and Microsoft postings associated with this event at:
- https://community.tenable.com/s/article/Plugin-Updates-to-Address-Windows-Scan-Targets-being-left-unable-to-connect-to-Azure-Active-Directory-AAD
- https://docs.microsoft.com/en-us/microsoft-365/troubleshoot/authentication/unable-sign-in-m365-desktop-apps
