Cisco Live Melbourne Security Operations Center (SOC) Review

Splunk

Our Splunk stack comprised Splunk Cloud and Splunk Attack Analyzer. Splunk Cloud included Splunk Enterprise Security (ES) and the Cisco Security Cloud apps. As our security infrastructure involved on-premises appliances like the Firewall Management Center and the Secure Network Analytics Manager, transferring on-premises data to the cloud was necessary. To achieve this, we set up a UCS M3 server on-site. Once the server was up, we deployed a compact Ubuntu virtual machine and installed Splunk on it.

The Cisco Security Cloud app, available on the Splunk base app store, is a unified app to bring data from Cisco Security tools to Splunk. It’s modular, enabling configuration for individual products to ingest data into Splunk, including Secure Malware Analytics, Firewall, Secure Network Analytics, Cisco XDR, and more. The app includes pre-configured dashboards for each product and health monitoring of the app to track data ingestion levels. It transforms ingested data to Splunk’s Common Information Model (CIM), a universal schema for data indexing, facilitating cross-data set visualizations and field searches across telemetry types.

With the Cisco Security Cloud app set up to ingest data from multiple sources, we installed the universal forwarder app to establish connections with the Splunk cloud deployment. The universal forwarder proved highly efficient, forwarding large volumes of data to Splunk cloud without exceeding 30% CPU usage or experiencing significant ingest delays. This enabled SOC analysts to search data in Splunk cloud, where Enterprise Security was also installed. Notable incidents from XDR were automatically populated as alerts in Splunk ES.

Secure Firewall Threat Defense

In the Cisco Secure Firewall (CSF) deployment at Cisco Live Melbourne, we have an Intrusion Detection System (IDS) setup that takes in a TAP from the current network and security infrastructure utilized during the conference. The CSF serves as the ingestion point for other security solutions in use by our Security Operations Center (SOC), collecting important data, producing logs and events that feed into tools like Cisco Splunk and Cisco XDR. Additionally, CSF retrieves files from unencrypted sessions for further analysis in Secure Malware Analytics sandbox.

While operating in passive IDS mode has limitations in terms of visibility, such as the inability to leverage TLS Server Identity for extracting extra details from HTTPS connections and the restriction on general decryption, the firewall still offers essential alerting capabilities. The vast amount of data points captured for each connection played a pivotal role in many investigations, notably highlighted in the sections ‘Sifting Traffic with Secure Firewall’ and ‘Malware Callouts from the Show Floor’.

Regarding geolocation data, Cisco Live attendees predominantly exhibited connections back to the USA, overshadowing other destinations significantly.

Australia, as the host country, also demonstrated a notable presence with twelve million connections. While no other country exceeded a million connections, the rest of the list displayed an expected dominance of regional and global technological spots. This predictability in geolocation preferences of the attendees enabled us to delve deeper into uncommon inbound and outbound geolocation connections, aiding in expanding multiple investigations following the discovery of an initial event. Despite the potential for masking geolocation data for malicious activities via Tor, VPN, or compromised hosts in other countries, traffic aligned with anticipated geolocation patterns still undergoes scrutiny through signature, heuristic, and sandbox analyses. Geolocation is just one of several facets that can expose attack patterns.

Furthermore, we monitor application data at a broader level, alongside individual alerts for malicious domains. Although plaintext attacks and information leaks persist at each conference, their occurrence rate has gradually declined. Notably, at Cisco Live Melbourne 2024, HTTPS was preferred over HTTP at a ratio of 15:1. The adoption of HTTP/3 is also on the rise.

Additionally, the trend of using DNS over HTTPS to obfuscate DNS requests is noteworthy. Despite the prevalence of plaintext DNS requests, the adoption of DNS over HTTPS continues to climb. It is expected that plaintext DNS requests will eventually be overshadowed by encrypted DNS protocols, akin to the transition from HTTP to HTTPS.

Automations

Written by Aditya Raghavan

In the realm of automation, we have introduced three fresh automation workflows to expedite threat hunting for our analysts. Credit goes to Ivan Berlinson, our associate from France, for the first two workflows in XDR automation integrating with Secure Malware Analytics, and to Adi Sankar for the workflow involving Umbrella.

1. Malicious samples submission in Secure Malware Analytics

Our aim is to minimize the number of dashboard transitions our analysts must undertake. Therefore, whenever samples submitted to Secure Malware Analytics are confirmed as malicious (with a threat score > 90) and observed inIn the Cisco Live setup, this automated process would promptly generate an incident in XDR and dispatch a Webex message to the Incidents channel. The above serves as an illustration. While not a routine task in a live environment, it aids in highlighting intriguing paths of inquiry within XDR and Webex for our analysts.

2. Innocuous examples from prevalent document formats

Likewise, at such gatherings, we commonly observe contents being transmitted in plain text. Documents of standard file types submitted to Secure Malware Analytics with a non-malicious outcome (threat score < 85), witnessed in the Cisco Live setup, and falling under the following categories generally contain text in plain sight. This merits an inquiry for our analysts to ascertain if any crucial data unintentionally leaked. This process would automatically spawn an incident in XDR and transmit a Webex message to the Incidents channel for documents of the listed file formats.

• PDF, TXT, XLS, XLSX, XLSM, PPT, PPTX, PPTM, DOC, DOCX, DOCM

3. Generating incidents from Umbrella Security Events

Instances of DNS Security Events in Umbrella pertaining to particular areas of interest would be escalated to the analyst as separate incidents by category. This demonstrates an instance of an automated incident created for the Malware category.

Analyst Narratives

CoinLoader Infection Inquiry

By Christian Clasen

A few days into the conference, we detected multiple block events in Umbrella DNS. These events involved TXT record queries for seemingly randomly generated subdomains linked to ucmetrixsdn[.]info. The queries mimic the domain generation algorithm (DGA) method frequently employed for malware communication.

The DGA technique is a characteristic of a command and control (C&C) infrastructure typically serving one of two objectives: fetching instructions from the malware’s operators or exfiltrating data from the infected endpoint covertly. As this malware is widely recognized (initially identified in 2018), we can leverage public intelligence to compile anticipated behaviors and additional signs of compromise to commence our examination.

The DGA behavior observed here aligns with the CoinLoader malware. Dark Trace published a comprehensive article that provided us with some guidance: https://darktrace.com/blog/catching-coinloader-decrypting-the-malware-hijacking-networks-for-cryptomining-operations. We immediately sought answers to the following questions:

1. What was the current status of the breach?
2. Was there any threat to other participants?
3. Had the user succumbed to the infection while on the conference network?
4. Who was the user operating the infected device?
5. Were there any linked infections at the conference?

CoinLoader serves as an initial dropper intended to download additional malicious payloads such as ransomware, data thieves, and crypto miners. It seemed that this specific infection was likely in its nascent phase, and Umbrella effectively curbed further infection stages by obstructing the C&C traffic. No interactions were logged between this device and other attendee IP addresses, nor any reconnaissance activities, indicating that the risk to other participants was presumed to be minimal.

CoinLoader lures victims by posing as cracked or pirated variants of legitimate software. To verify if the malware was acquired on the conference network, we scoured our SOC tools (including Secure Malware Analytics and Firewall file events) for RAR and ZIP file extensions, and any instances of filenames containing “keygen” or “crack.” No evidence suggested that the malware was downloaded during the conference. Due to the encryption of attendee traffic, this certainty remained unattainable.

To pinpoint and inform the device owner, we employed standard fingerprinting methods. DHCP logs and traffic patterns proved invaluable in deducing the OS and device category. In this instance, MDNS queries originating from the device divulged both the OS classification and hostname. The hostname included the attendee’s first name. Leveraging data from the wireless infrastructure, we successfully pinpointed the device location on the exhibition floor.

Following user notification and device evaluation, we proceeded with an extensive search for related IOCs elsewhere on the network. Our hunt included:

• A specific phrase in the Issuer field of the TLS certificate
• A specific ASN and publicly routable IP range situated in Eastern Europe.
• Additional C&C domain names and URLs.

By utilizing Splunk, we efficiently combed through all our log sources for these IOCs and found no other instances of this infection.

Approaches to Identify Clients on Public Wireless Networks

Nathan Quentin

In real-world scenarios, practical implementations often deviate from the idealistic configurations envisioned by manufacturers. Restrictions such as events, financial limitations, and time constraints frequently thwart the deployment of comprehensive security infrastructures. Faced with these challenges, analysts must resort to correlation methods to maximize the insights obtainable within the Security Operations Center (SOC) environment. One particular constraint encountered at the Cisco Live SOC was the absence of Umbrella Virtual Appliance (VA) integration, resulting in a blind spot in client-side IP visibility. Despite the lack of VAs, analysts managed to attribute potentially malicious or dubious DNS queries to client IP addresses on the public Wi-Fi network by leveraging their understanding of Umbrella operations.

Umbrella functions as a recursive DNS resolver that harnesses the capabilities of the global DNS system to enforce security protocols and acceptable usage policies. The public IP addresses assigned during the conference are associated with an Umbrella organization, facilitating the attribution and enforcement of suitable policies for DNS queries. Due to Network Address Translation (NAT), all IPv4 queries are traced back to the public address that serves all conference attendees. In an ideal Umbrella deployment, internal recursive resolvers (VAs) would be in place to provide internal IPv4 attributions. Regrettably, the internal resolvers used at the conference lacked this functionality, resulting in only public IP address attributions in Umbrella alerts.

The most straightforward solution to this dilemma would involve incorporating the internal recursive resolver logs into our Security Information and Event Management (SIEM) and Security Orchestration, Automation, and Response (SOAR) systems. Although this solution was in the works and undergoing active development, it was not immediately accessible during the initial phases of the conference. So, how could this gap be closed to ensure the availability of the most precise information for these incidents? The answer is uncomplicated for those familiar with Umbrella’s functionality.

When Umbrella identifies a query targeting a malicious domain, it does not merely reject resolution or return an NXDOMAIN response. Instead, it directs the query to dedicated IP addresses owned by Cisco and awaits the subsequent connection to present a blocking page. For HTTP/S connections, this approach effectively communicates to the end user why their connection was thwarted. Umbrella reserves distinct IP addresses for categories such as Malware, Phishing, and Command and Control traffic. These addresses can be found at https://docs.umbrella.com/deployment-umbrella/docs/block-page-ip-addresses.

Armed with this insight, there are two methods for correlating Umbrella DNS events with Firewall incidents. By filtering the Firewall connections for the destination IP address linked to Umbrella Malware blocks (146.112.61[.]107), we can pinpoint any connections attempted by the client following the resolution of the malicious domain. If these connections occur over HTTP or HTTPS, it is highly probable that the hostname can be identified in the HOST header or Server Name Indication (SNI) extension field, as the client presumes it is connecting to the intended malware server rather than Umbrella.

For non-web traffic, it is possible to align the timestamp from the Umbrella event with the IP connection in the firewall logs to confidently identify the specific internal client IP responsible for the malicious or suspicious DNS query. Geolocation data from the wireless infrastructure can then aid in locating devices and persons when the alert’s content justifies such actions.

Data Harvesting from Infrastructure Servers

Aditya Raghavan, Adam Kilgore

The narrative began when Adam detected a series of SSH sessions originating from an IP within the DC static host group range towards internal IP addresses on an unconventional port (TCP 830). Initially, all these connections were successful, suggesting legitimacy.

Further exploration using XDR Investigate revealed that another adjacent device from the Infrastructure Management host group was also engaged in similar traffic flows between the devices in the Infrastructure Management and DC Static host groups, triggering multiple Snort alerts on the firewall.