Pacific Rim: Discovering how to consume soup with a blade
Traditionally, security characteristics have not been a top priority for embedded architecture devices like network appliances, but during Pacific Rim, they became the center of an escalating competition – one that blue teamers, including those not solely at Sophos, must grasp.
The positive news is that many of our existing principles translate exceptionally well: Modern network appliance technology is founded on well-understood operating systems such as various Linux versions. The downside is that some of these principles may necessitate adjustments. While technology has advanced, there still remains a substantial number of devices in operation employing outdated, security-oblivious embedded architectures – merely lying idle on racks.
Naturally, Sophos, as an information-security organization, possesses a twofold perspective on security and response; we do not just react to incidents affecting us as a company, but also to incidents impacting our products and services – the “us” that gets introduced to the broader world. Consequently, our incident response protocols extend beyond our internal corporate realm to the very infrastructure we deploy for our clients. This constitutes a particular form of double vision which – we believe – provides us with an advantage in devising strategies to adapt incident-response principles to the present requirements.
Effectively implementing the dual-view system necessitates close collaboration between the teams responsible for developing our products and the team entrusted with handling security issues concerning them, our Product Security Incident Response Team (PSIRT). Given that not all enterprises possess (or require) a PSIRT, prior to delving into our discoveries, it is helpful to elucidate on how our PSIRT functions.
A Day in the Life of the Sophos PSIRT Team
Our PSIRT team diligently monitors numerous channels for information regarding new discoveries related to Sophos products and services. For instance, as previously noted in a recent article which shed light on Sophos Intercept X (a subsequent write-up explored our content update structure), we have been engaged in an external bug bounty initiative since December 14, 2017 – a mere year before the initial indications of what transpired in Pacific Rim emerged – and we welcome the scrutiny and collaborative prospects that this initiative provides. Moreover, our responsible disclosure policy also extends a ‘safe harbor’ to security researchers who divulge their findings in good faith. Alongside external reports, we also execute our own internal assessments and monitor open-source data.
Upon receiving a security incident, the PSIRT team undertakes triage procedures – confirming, assessing, disseminating, and tracking the issue to ensure our response is appropriate, safe, and effective. When necessary, matters are escalated to our Global Security Operations Centre (GSOC), which operates round-the-clock across over a dozen locations, synchronizing efforts on cases incessantly.
Our PSIRT team takes charge of remedial actions, collaborating with our product subject matter experts to deliver technical security advice, and progressing towards resolution in line with response protocols – empowering our clients to efficiently mitigate pertinent risks promptly. Our objective is to communicate outcomes clearly in actionable security advisories and inclusive CVEs – incorporating CVSS ratings, as well as Common Weakness Enumeration (CWE) and Common Attack Pattern Enumeration and Classification (CAPEC) information.
Not only does this align with standard PSIRT practices, but it also supports our dedication to CISA’s Secure by Design initiative. In fact, Sophos was among the pioneer organizations to pledge allegiance to this initiative, and the specifics of our pledges can be viewed here. (An essay by our CEO, Joe Levy delves extensively into our commitment to Secure by Design and how, equipped with insights from Pacific Rim, we plan to uphold that commitment.)
A capable PSIRT team does not solely rely on incoming reports. In the backdrop, besides conducting internal testing and research, the team strives to enhance our product security standards, frameworks, and guidelines; conduct root cause analyses; and consistently refine our processes based on input from both internal and external stakeholders.
All of these tasks shape the ensuing discussion in this article, as we elucidate on what we have gleaned through refining and enhancing our processes throughout the Pacific Rim saga. We will explore principles – several of which we have integrated or are in the process of integrating ourselves – as a springboard for a wider discourse among professionals about the semblance of an effective and scalable response in the realm of network appliances.
Takeaways from our Journey
Data Collection
The foundation lies in the ability to capture state and alterations directly on the device itself. Network appliances are frequently disregarded as standalone devices since their customary role is as cloaked conveyors of network activity. Nevertheless, recognizing this distinction is a pivotal step in delivering observability on the device – a crucial element for response.
Primary challenges:
- Network plane versus control plane. Our aim is not to scrutinize your network (the network plane) in the slightest. Conversely, it is imperative to monitor the device responsible for managing your network (the control plane). This distinction is often conceptual rather than tangible, yet it has evolved into a crucial differentiation to safeguard customer privacy.
- Availability of on-device resources. These appliances still function as compact devices, with constrained RAM and CPU resource availability. The mechanisms for capturing data for telemetry must be streamlined to prevent unnecessary service degradation affecting the device’s core function. (That said, resource capabilities have improved in recent times – albeit making it simpler for intruders to conceal themselves in the commotion. Administrators are less likely to unknowingly expel a trespasser from a device with an inadvertent forceful reboot upon noticing that the firewall is sluggish for the entire network, as the modern firewall can accommodate bulky software and thus does not exhibit the same urgency.)
- Excessive data capture. Network appliances are constructed differently. The /tmp directory might remain relatively tranquil on a user endpoint – meriting proactive surveillance – but it could be significantly more chaotic on a network appliance. Fine-tuning is imperative to prevent the telemetry from being flooded with superfluous data.
Data Transmission
Irrespective of whether the detection takes place on the device or within a rear-end data reservoir (to be detailed further), a juncture arises when the gathered telemetry needs to be dispatched off the device. While numerous of these principles are extensively documented in the realm of security monitoring, network appliances pose some distinctive challenges.
Key challenges:
- Interference from the host system / NIC configuration. Network appliances are inherently finicky in terms of managing network interfaces and how the host system impacts the traffic it carries. The inclusion of an extra data stream output often necessitates considerable reengineering. Making sound technology selections is essential to ensure seamless integration.
- Accumulation versus choice. Amassing the entirety of a user’s network traffic poses significant privacy issues and is an inefficient form of detection engineering. Opting for the most pertinent data using rulesets (which can be crafted, modified, tested, and deployed) is a common method for high-volume accumulation, but it necessitates well-documented (and scrutinized) selection criteria for successful implementation. This differentiation also enables careful implementation of retention policies – longer for selected data and shorter for accumulation.
that generate minimum disruption are crucial for establishing a buffer between response and device functionality. OSQuery serves as an excellent illustration of a technology that can facilitate near-real-time queries while lessening the likelihood of resource impact.
Triggers, alarms, and detections
The subsequent phase involves distinguishing signals from background noise. As cybersecurity professionals, we are often trained to seek deviations from the norm and the presence of abnormalities – yet the interpretation of both varies significantly in network devices.
Principal challenges:
- Telemetry selections + streaming selections = blind spots. Deliberately opting for a subset of accumulation, while essential, results in gaps that must be continuously reassessed on the fly. Omitting /tmp from accumulation may be a sound move to reduce noise, but leaves it as an ideal breeding ground for malware. Practitioners must devise methodologies to monitor these blind spots with less detailed “tripwires” such as file integrity monitoring.
- Establishing detections over selected data. While possessing the subset of selected data is a positive start, it is likely still too cumbersome to process. We discovered that at this juncture, detection engineering practices could then be applied to the selected data – ideally in a standardized format alongside other security telemetry, to facilitate examination.
Response measures
We are dealing with core network infrastructure, which does not respond favorably to aggressive approaches. While terminating a suspected rogue process or isolating a device from a network may seem trivial on a user endpoint, executing either on a network appliance could detrimentally affect the availability of a user network. In our experience, at this stage, having firm boundaries, setting expectations, and halting response activities from exacerbating the incident proved immensely beneficial.
Principal challenges:
- Effects on network availability. The impact of “turning it off and on again” is more pronounced when referring to an entire organization’s internet access. Implementing any response measures – scalable/automated or otherwise – must be treated as a potentially highly impactful business modification and must adhere to a change management process.
- Network versus control plane (once more). It is crucial during data accumulation and equally essential during remediation. Understanding the boundaries between the responder and the network user is vital to restricting the exploitation of response measures and limiting the exposure to any adverse effects.
- Business and legal constraints. At this stage, the dialogue shifts beyond technical response practitioners to incorporate members of the extended response team – particularly Legal and the executive leadership. Some of the queries to pose to those stakeholders include: Who assumes the risk if a response measure disrupts a network? Who assumes the risk if that measure isn’t taken, leaving the network vulnerable?
Conclusion
Necessity drives innovation, and it is safe to affirm that Pacific Rim has demonstrated that there is more to be done in the realm of incident response for network appliances. The application of these fundamental principles has empowered us to safeguard our customers to a degree that we never deemed possible, yet it has also unveiled critical limitations that practitioners need to tackle – some internally within their organizations, some in-house at each vendor, and some industry-wide. Categories such as network availability, data confidentiality, and limitations of accountability, concerning response measures, demand not only technical but also commercial and legal frameworks. As challenging as these topics may be to deliberate, let alone implement, it is a discussion we must engage in across various platforms if we are to keep pace with the evolving threat landscape.
Sophos X-Ops is eager to collaborate with others and provide additional detailed IOCs on a case-by-case basis. Reach out to us via pacific_rim@sophos.com.
For the complete narrative, please refer to our landing page: Sophos Pacific Rim: Counter-Offensive Against Chinese Cyber Threats.
About Author
