Standing on the Windows platform, anticipating transformation

During the recent week, Sophos engaged in Microsoft’s gathering on Windows Endpoint Security Ecosystem Summit. Reflecting on the recent CrowdStrike incident that led to a global system crash following a kernel-driver update, participants from various sectors, including industry and government, congregated to delve deeply into topics such as kernel architectures, update-deployment processes, and, most importantly, the evolution of the security ecosystem with transparency and widespread community involvement to safeguard the globe. This was an initial conversation, not a policy discussion, but several significant points emerged.

One of the focal points was how the Windows system could advance to limit the necessity for security firms to utilize kernel drivers, user-space hooking, or other methods to seamlessly and actively interact with the platform while denying adversaries a foothold in the system’s core. Collaborative insights across industries, coupled with past successful experiences, are vital to achieve this goal. Another area of focus was deployment – specifically, how software and updates are effectively disseminated to millions of users securely and without causing major disruptions.

During the exchange, Microsoft highlighted our practices and outcomes as a model. In this write-up, we will elaborate on the current integration of Sophos with the Windows platform and discuss, at a conceptual level, potential directions in which the Windows system could develop to recalibrate the techniques and access required for third-party security vendors to collaborate effectively. We will also touch on Safe Deployment Practices (SDP), a subject that both Microsoft and Sophos explored during the summit. As a conclusion to this post, we will recount three episodes of managing foundational changes for both Mac and Linux products, offering them as potential reference points for further discussions within the industry.

This piece serves more as a guidebook than a roadmap, providing a backdrop and general information on the landscape. Defining specific requirements for such extensive resilience and security objectives surpasses the scope of this post, but an overview of the landscape itself is valuable amidst these reflective conversations. Watch this space.

For what reason does Sophos utilize kernel drivers?

Similar to other cybersecurity enterprises, Sophos interacts with the underlying Windows infrastructure using a blend of methods, some of which penetrate deep into the platform’s foundation: kernel drivers, user-space hooking, and assorted techniques. Each security entity adopts its exclusive approach to this. At Sophos, we have previously disclosed details on our methodologies. In a general sense, the system access facilitated by kernel drivers is indispensable to deliver the anticipated security functionalities to modern cybersecurity product users. These functionalities encompass:

Insight

• Delivering highly precise and near real-time insights into system operations

Defense

• Empowering the prevention of malicious or non-compliant activities proactively rather than merely observing them
• Facilitating prompt responses to identified malicious or non-compliant operations and rectifying or reverting them

Tamper resistance

• Instilling trust in the correct functionality of the security product even in scenarios where segments of the operating system have been compromised

Stability / compatibility

• Ensuring the installation of the security product does not undermine the stability of the Windows system or the performance of third-party software and hardware

Performance

• Delivering the above capabilities with a foreseeable and bearable impact on the overall system performance

Efficient power usage and modern standby

• Offering these capabilities even during low-power modes – meaning that if there is any other activity ongoing, the security product will continue to deliver insights and protection
  * Other functionalities of the Windows system should operate smoothly and dynamically resolve dependencies to prevent deadlocks during low-power modes

Present Sophos Windows drivers

At present, Sophos maintains five Windows kernel drivers: an ELAM (Early Launch Anti-Malware) driver, two drivers intercepting file and process activities, and two drivers capturing network activities. Extensive details regarding these kernel drivers have been covered previously, therefore, we will provide a brief overview here. To summarize:

• It is mandatory for security vendors to provide an ELAM driver, like SophosEL.sys, to establish as an endpoint-protection product and disable Windows Defender on user devices
• The two file drivers offer comprehensive process journaling, event recording not available through Windows API currently, as well as anti-tampering features, process interception, and ransomware blocking
• The pair of network drivers facilitate web security, packet analysis for intrusion prevention, DNS shielding, and network stream redirection for zero-trust network access

In the subsequent section, we will briefly discuss Sophos’ approach to injecting DLLs into processes within the kernel and user space. For now, let’s provide an overview of the functions of each of the five drivers, inviting curious readers to refer to the aforementioned post for further details.

SophosEL.sys

Being the ELAM driver, SophosEL.sys plays a crucial role. In alignment with all security vendors operating in the Microsoft Windows environment, Sophos needs to supply an ELAM driver to initiate AM-PPL (Anti-Malware Protected Process Light) services and processes. Only AM-PPL processes can enroll as an antivirus (AV), thus deactivating Windows Defender on user devices. Additionally, AM-PPL processes enjoy inherent protections like being unaltered through the user interface. In detail, SophosEL.sys prevents blocked drivers from loading early in the Windows boot process. Furthermore, this driver encompasses distinctive “fingerprints” of Sophos-specific code signing certificates, empowering Sophos to execute AM-PPL processes and services.

SophosED.sys

Making up one of the file-system drivers, SophosED.sys is the principal anti-malware driver at Sophos; the “ED” designation indicates Endpoint Defense. The functions performed by SophosED.sys include providing events to the Sophos System Protection service (SSPService.exe) through a blend of synchronous callbacks (suspending activities until SSPService.exe returns a verdict) and asynchronous events (queueing serialized versions of events and relevant parameters for asynchronous notifications). Other functionalities of this driver include:

• Maintaining a detailed system of context-inclusive ‘shadow’ process/thread/module tracking
• Documenting low-level system events to the Sophos event logs for forensic analysis
• Safeguarding the Sophos setup and configuration processes against tampering using an independent authentication mechanism
• Establishing an autonomous attestation system for binaries distributed by Sophos
• Injecting SophosED.dll into newly initiated processes
• Ensuring the execution of Sophos native applications essential during boot
• Establishing secure communications between Sophos processes, services, and drivers; constant file hashing; and support for memory
• scanning

hmpalert.sys

The HitmanPro Alert driver is the secondary file-system driver within our five kernel drivers lineup, responsible for enforcing CryptoGuard. Its functions include identification and prevention of mass encryption of files by ransomware, and introduction of hmpalert.dll into recently initiated processes.

sntp.sys

The sntp.sys network-filter driver provides the fundamental network interception functions necessary for implementing network filtering by Sophos; “sntp” represents Sophos Network Threat Protection. This driver’s capabilities cover filtering of HTTP and HTTPS web traffic to establish web security, Data Leakage Prevention (DLP), and enforcement of acceptable use policies through Sophos web protection; analysis and recording of HTTP or HTTPS web traffic, DNS queries and responses, and general TLS stream activity in Sophos event logs and Sophos Central data storage; interception and injection of L2 packets to maintain Sophos’ IPS (Intrusion Prevention System); and pausing/delaying outbound flows for additional examination or coordination activities across systems.

SophosZtnaTap.sys

SophosZtnaTap.sys stands as the second network-filter driver in use; it serves as a Sophos-developed OpenVPN TAP driver. It is employed by Sophos to implement their ZTNA (Zero Trust Network Access) agent. This driver intercepts DNS requests and, if related to ZTNA-protected applications, replies with a tunnel IP address, subsequently directing IP traffic to the applications.

About DLL injection

Sophos integrates DLLs into processes through a unique mechanism established in both SophosED.sys and hmpalert.sys. Currently, there is no officially supported approach in user space or the kernel to request DLL injection. The inserted DLLs enhance visibility and safeguard API calls executed by applications.

Travel this path: Steps for enhanced operation

In the following two segments, we initially outline the decisions that Sophos has embraced in its update and feature deployment procedures, then elucidate (once more, at a broad level) potential advancements on the Windows platform to diminish reliance on third-party kernel drivers, an objective seemingly worth striving for.

Secure deployment: Gradual rollouts and feature toggles

As emphasized earlier, a key topic during the Summit was Secure Deployment Practices (SDP). Similar to Microsoft, Sophos has dedicated substantial efforts to fortifying our software structure to accommodate step-by-step software deployments and feature toggles. Sophos aims to optimize the safety and reliability of our products while granting customers maximum visibility and control where feasible. Sharing our strategies and insights with Microsoft and fellow industry members is anticipated to cultivate a comprehensive set of best practices for the entire Windows community.

As recounted in a previous publication earlier this year, Sophos has refined a robust protocol for introducing new software versions and gradually activating new features for our customer base. This mechanism also empowers Sophos to promptly deactivate features for a single user, a specific software version, or the global user base. Furthermore, Sophos Central provides users a comprehensive overview and authority over software updates and configurations within their entities.

Any security solution, whether reliant on proprietary kernel drivers or leveraging existing Windows platform capabilities, necessitates periodic updates that modify system behavior. Any alterations in system behavior of such nature should be incrementally released to guarantee stability and functionality. The discourse surrounding optimal practices for secure deployment was a standout feature of the Summit, serving as a domain where collaborative ecosystem advancement can cultivate enhanced customer trust in patches and updates – thereby bolstering internet security for all.

Diminishing reliance on third-party kernel drivers

We shall now outline, in broad terms, several functionalities that Sophos executes using kernel drivers. If the Windows Platform were to advance in ways that mitigate the necessity for kernel drivers, as discussed earlier, these functionalities could be valuable additions.

Continued evolution entails a process likely necessitating transparent communication and input from various stakeholders; much like Rome, Windows was not constructed in a day. Moreover, implementing changes will require thoughtful consideration on how malevolent entities could circumvent such changes. We present this information as the starting point for dialogues.

This list does not encompass all current platform functionalities in practice; for this publication, we highlight eight potential advancements based on our own experiences, offering an initial overview of certain features that Sophos believes would be beneficial. These eight points are intended to instigate further discussions and refined delineations. We anticipate and encourage collaboration with Microsoft to elaborate on any prerequisites, ideally through iterative and collaborative efforts.

API to authorize/deny access to files and directories

Including a supported mechanism for security vendors on the Windows platform to assess files and directories accessed by processes and approve/deny such access could be advantageous. This might entail receiving notifications about attempts to open a file, managing decisions for subsequent file access, and overseeing updates and modifications to these decisions.

API to authorize/deny registry access

Enabling a supported mechanism on the Windows platform for security vendors to monitor registry access by processes and authorize/deny such access could be beneficial.

API to regulate process behavior

A supported mechanism on the Windows platform for security vendors to monitor process activities on the system and take appropriate actions could prove valuable. These functionalities would mirror the assistance provided by Windows kernel to kernel-mode drivers (with some enhancements). The details below serve as preliminary guidance and are not exhaustive.

Process Activity Callbacks: A capacity to address events like initiation of child processes, termination of processes, commencement of threads, termination of threads, context setting for threads, scheduling of APC, loading of images, etc., where security vendors can permit or block the operations.

File Activity Callbacks: A functionality to address events such as attempts to create, open, modify, or rename files/directories.

• For instance, Sophos monitors suspicious modifications to documents that could indicate ransomware. Ransomware might attempt to evade detection by encrypting the file in place or creating an encrypted copy alongside the original, then either replacing the original with the copy (deleting the original, renaming the copy) or overwriting the original contents. Writing actions can be executed through standard file writes or memory-mapping the file for writing. The supported mechanism should offer adequate callbacks for thorough analysis.
• In the same context, crafting a functionality to address events like creation, deletion, renaming, linking of Registry keys, access to key/values, modifications, and enabling/disabling such operations can be valuable.
• A feature to address events like installation of new drivers, hardware, or software devices, validating them during the installation phase (explore the unauthorized drivers section below); also, enabling visibility into processes connecting to driver devices and allowing/blocking access can be pertinent.

intricate and could also consist of overlooking the hierarchy of building device stack and filtering devices and procedures releasing IOCTLs to devices.

API for managing network access

A contemporary endpoint protection approach covers network security. Therefore, it might be valuable for the Windows platform to offer a validated system for security vendors to thoroughly safeguard networked devices. This might involve the ability to accept and permit diverse network flows, to analyze and potentially adjust the data within the flow, and to carry out this process before interacting with the destination.

For current zero-trust deployment strategies, this could also involve the capability to capture and reroute traffic through gateways specific to vendors, to screen and reply to DNS queries, to authenticate/permit access to registered applications, and to seize or insert authentication tokens in the rerouted traffic. Discussions on this front would naturally include mechanisms to prevent misuse of such capabilities.

API for approving/preventing kernel drivers

It might be beneficial for the Windows platform to provide a supported system for security vendors to hinder unauthorized drivers. Kernel drivers have the ability to terminate any procedure, including AM-PPL security procedures, and this is commonly used by malicious software campaigns.

It might also be valuable for the Windows platform to introduce a supported user space mechanism for security vendors to stop local and domain administrators from overriding or undermining the security product’s choices, apart from, for instance, by authorizing the behavior, driver, or application using the security product’s API or user interface.

Moreover, it might be advantageous for the Windows platform to offer a supported system for security vendors to receive detailed data about potential kernel drivers (e.g., filename, driver size, hashes, signatures) and to manage the blocking and loading of kernel drivers.

API for linking context with kernel objects (procedures, files, Registry keys, network connections etc.)

It might be beneficial for the Windows platform to provide a validated system for security vendors to uphold an unalterable context about kernel objects, such as files and procedures. The context may encompass information regarding whether an object belongs to Windows, belongs to a specific security solution, or is related to another product; information on whether the object has been inspected, the time of inspection, and the conclusions reached; as well as file hashes or other data linked to an object, like a unique identifier for the object. It could be valuable for this context to retain through reboots as appropriate.

DLL insertion or equivalent mechanisms

It might be beneficial for the Windows platform to provide a validated system for security vendors to insert DLLs and/or deliver functions currently offered by inserted DLLs. Presently, inserted DLLs serve both hooking and low-level protection, for instance as discussed earlier.

Hooking: Inserted DLLs hook various APIs to report details about API calls from procedural code, indicating when the procedure is malicious and when malware is inserted in an otherwise authentic process. Some of these API calls are also covered by Event Tracing for Windows (ETW), but the data gathered via ETW lacks certain parameters necessary for effective protection.

Also, ETW is consistently asynchronous, and it might be valuable to have a synchronous mechanism. Preferably, a security vendor should possess authority over which API calls, the level of detail, and whether a specific event is synchronous or asynchronous. For instance, it could be beneficial for the Windows platform to offer a validated method for intercepting syscalls.

Low-level protection: Inserted DLLs also provide detection/protection mechanisms. Certain instances involve protecting the hooks from unhooks (by malware), preventing hooking by malware, memory page protection beyond the operating system’s provisions, identifying efforts to bypass APIs (e.g., using syscall directly, accessing PEB and related data directly).

It might also be beneficial for the Windows platform to develop new Windows protection mechanisms, like Windows-provided integrity of its own DLLs (e.g., “PatchGuard in user mode”). Another approach could be Windows-supplied asynchronous (like Microsoft Threat Intelligence Secure ETW, already in existence) and synchronous (novel) callbacks about in-process events, encompassing memory allocations, setting thread context, and kernel exception handling — e.g., callbacks about exceptions before passage back into user mode. Naturally, these or equivalent mechanisms should be created while considering their impact on system performance.

Protection from tampering and AM-PPL

It might be beneficial for the Windows platform to provide a supported system for a mechanism to safeguard security processes from being deactivated, terminated, or uninstalled. Currently, this functionality is carried out by AM-PPL (which in turn necessitates an ELAM driver) and by the Sophos driver. Without ELAM drivers, security vendors require some other “root of trust” to enable launching protected processes.

The security offered by AM-PPL presently lacks completeness, as malicious agents can still uninstall or manipulate the security product, unless the security product actively defends itself (e.g., protecting its binaries and Registry keys). It might be beneficial for the Windows platform to introduce a supported plan to secure a security product and its various components and functionalities, like files, processes, registry keys, and IPC.

Ideally, this extra layer of security could only be waived by the security product itself (for update/uninstallation purposes), with some provision for removing the security product through other means if required.

And beyond: Mac and Linux

In this last segment, we will explore three instances where the progression of the Windows platform might draw inspiration from how specific issues have been addressed on, respectively, Linux and macOS.

Sophos on Linux 1: XDR Oversight with eBPF

eBPF is a technology that offers in-kernel observability hooks in the Linux kernel; the initial significance of the term was Berkeley Packet Filter, an early packet-filtering technology, but no longer does. Microsoft has an experimental adaptation of eBPF for Windows.

In Linux, Sophos employs eBPF probes for monitoring process, file, and network activities. These probes collect information and carry out basic stateless filtering; user space operates on the flow of events and evaluates the activities.

A crucial safety element of eBPF is the verification process. eBPF programs must adhere to various restrictions to be compiled into bytecode and loaded into the kernel. For instance, Linux does not provide string pattern-matching functions, and they cannot be implemented in eBPF bytecode due to verifier complexity restrictions. Linux eBPF kprobes function in an atomic context and only have access to unpageable kernel memory.

Such limitations could make it challenging for eBPF for Windows to be the foundation for an “approved/block” interface in user space as described earlier. eBPF for Windows could serve as a solution for dynamically gathering system activity events in the kernel and dispatching them to user space for retrospective analysis.

Sophos on Linux 2: Document scanning with fanotify

Since version 5.1, Linux boasts a fanotify API for intercepting file operations. Initially, Sophos utilized a Linux kernel driver

(Mole) to deploy real-time file scanning, yet transitioned to fanotify as an early adopter (and played a part in refining it into the standard it is today). Present-day advanced Sophos Linux products utilize fanotify to gather file events asynchronously, conducting file scans in the background when necessary, and initiating response actions based on the scan outcomes.

The shift to fanotify demanded a substantial commitment from Sophos. Different Linux distribution providers rolled out kernels with fanotify compatibility at varying release intervals, necessitating Sophos to maintain support for both the Talpa kernel driver and fanotify setups. Modifications to kernels leveraging fanotify had to propagate to the different Linux distributions before Sophos could adopt a consistent interface. Within the Microsoft platform environment, multiple editions of the operating system are in operation. It might be crucial to bear this in mind when contemplating alterations to the Windows platform.

Sophos on macOS: Bid Farewell to kexts? A Big Sur-prise

Apple introduced fresh endpoint security APIs a year prior to enforcing their obligatory use. While Sophos dedicated the year to transitioning from kexts (kernel extensions, in macOS) to the new APIs, clients persisted in using the version involving kexts and continued to receive OS and security services. The subsequent major macOS release eliminated kernel access for all vendors. Once more, the challenges linked with managing updates across distinct OS versions, and facilitating users to smoothly update and configure security solutions following OS upgrades, should be taken into account. Additionally, we provide these retrospective insights with the trust that they will stimulate a graceful evolution of the Windows endpoint ecosystem, irrespective of the direction it takes:

• Initial release notwithstanding, Apple’s endpoint security APIs were unsuitable for substituting kexts in a production environment. This restricted the deployment of the APIs in production and the accrual of practical experience.
• In contrast to Microsoft’s Canary and Dev channels, new releases were simultaneously available for all Apple Insiders.
• Apple refrained from sharing comprehensive plans, suggestions, or developer instructions for their APIs.
• Several crucial endpoint security APIs were unveiled late in the beta phase, with reported flaws necessitating reevaluation with each release to affirm status.
• Apple failed to provide any guidance or prior notification to security vendors regarding the timeline of the general OS release for customers.
• Even though Apple offers the option to still utilize kernel APIs, it mandates customers to deactivate multiple major OS security features concurrently. This has spurred customers and vendors alike to migrate to the endpoint security APIs instead of persisting with legacy kernel APIs. A different strategy of furnishing a single “toggle” for enabling access to those kernel APIs may not have yielded the same outcome.

Summary

Change is challenging. As evidenced by recent cybersecurity incidents and ongoing software patterns, it is also not discretionary. The ultimate result of this week’s Microsoft summit could remain undisclosed for months or years; undeniably, some of the potential changes arising from it might be disruptive as only fundamental change can be. We must also weigh the advantages of having Windows organically offer an expanded array of OS native security interfaces for the whole endpoint security ecosystem to leverage against the monoculture vulnerabilities of swapping the comprehensive diversity of proprietary innovations and controls that we currently enjoy from the endpoint security ecosystem. Having said that, we believe that transparency and open communication are the most effective means to expedite improvements for defenders and customers. Let’s embark on this journey.

About Author