When you get to the level of analyzing packet data, the first obstacle is actually capturing the conversations that you may find relevant. While a capture filter can help on the acquisition, it is possible that an error in the filter could exclude relevant hosts, or ports, or protocols. In general wide open captures will then lead to very large capture files.
Wireshark performs poorly with opening such a large trace. Since Engineer's are costing well north of $150 per hour, having them twiddle their thumbs waiting is not economical or efficient.
Enabling a survey and filtering of the data as soon as possible from the pcap file will allow interactive examination. With Brim that identified data can be exported as a filtered, and thus much smaller file. This is similar to what tools such as Riverbed's Packet Analyzer do. PA is a .net front end to tshark type filtering and reporting. It ships with a number of useful filters to survey for common problems.[1]
Brim and Zeek appear to fill the need between Packet Analyzer on one hand and a regular Wireshark workflow on the other.
It might be more useful to think of Brim in terms of "a different zoom level" than "better than". Wireshark focuses on showing as much detail as it can about every packet that you see. If you have a lot of packets this will burn a corresponding amount of CPU time and memory. Brim, Packet Analyzer (mentioned in another comment), and other tools zoom out a bit and let you process a larger number of packets, but with less detail. If needed you can pass the interesting and relevant ones to Wireshark for more detailed inspection and analysis.
Disclosures: Wireshark pays for my mortgage. My employer makes Packet Analyzer. I'm friends with people at Brim.
Wireshark dissects protocols and gives debugging traces. Zeek (FKA "Bro") is an intrusion detection system that analyzes debugging traces (which it generates itself, not through Wireshark dissectors) to generate usefully semantic data, ranging from "which hosts are fetching this URL" to "where are there signs of this vulnerability being exploited".
Both Wireshark and Zeek use libpcap, which is the de facto standard for raw packet captures, so yes, the same data you'd be feeding to Wireshark to browse, you can also feed to Zeek to analyze.
Hadn't heard of zeek before - actually find this submarine article useful - it hints better at where and how, than the official zeek documentation. I still don't know what storage space I should budget for a gbps-month of zeek monitoring logs, though (I understand they're typically filtered... But also text?).
As to the where - I guess I need to find some port that can give me a mirror of traffic.... But not that easy with heterogeneous lan+cloud+SaaS setup to figure out where/how to source "all" network traffic :/
The long and short of it is Zeek(used to be call Bro) provides session based analysis whereas wireshark is primarily packet based. With a few exceptions, they both generally do the same thing but in different ways. Zeek is at the bones, an event-based engine with a scripting language (think node-js but from the 80s!) - the scripting language is actually what provides the logs generated by Zeek.
One key aspect of Zeek is that it can be deployed within a network to passively generate logs. As an incident response consultant, the few times I've worked with a client with Zeek logs, our ability to answer some critical questions in short order was increased dramatically! Back in my Sysadmin days, I used to run Zeek (when it was called Bro) to provide network logs for security review but also for general network analysis.
You can definitely run a pcap capture by wireshark through Zeek. you'd run `zeek -r <yourpcap>` and you'll end up with some lovely TSV separated logs in your current working directory!
Full transparency: I'm not part of the Zeek team, but I did author the original Zeek scripting guide for them back in 2012 or so. For my money, the Zeek team is building some of the better network appliances and detection/logging capabilities available.
It’s more of a Network Security Monitoring tool (NSM) than a Wireshark equivalent
It’s primary use case is to have a server(s) ingesting all traffic in a network and forwarding the traffic metadata to something like an Elastic Stack.
What makes it so powerful though, is that it actually has its own scripting language that allows you create automations based off what type of traffic is observed and these can be super intricate
Since everyone who are into pcap analyses are looking, I cannot help but drop references to two other open source DPI products that are worth considering for some use cases.
The first one I authored myself many years ago and has been dormant for a long time although I'd certainly enjoy working on it again if an opportunity arises:
https://github.com/rixed/junkie
The other one seems to stand in the same niche of highly specific, optimised programmable DPI, but I'm obviously much less familiar with it:
https://github.com/DanieleDeSensi/peafowl
I've been meaning to dig into bpf as a packet tracing/sniffing tool for the same reasons... pcap files can get enormous quickly enough to make space and parsing a pita. Cool work.
Agreed. using bpf can eliminate some of the requirements for packet captures. Computing the metrics on the host, also eliminates the network traffic of exporting logs or pcaps to for instance splunk. [1]
Brendan Gregg wrote a book that digs really deep into the bpf system. [2] I recommend you check it out if you need to do real work fast.
Current convention is to refer to eBPF as BPF, and where referring to the Original BPF as cBPF. This is why Brendan's book is "BPF Performance Tools"
"Then in 2013, Alexei Starovoitov completely reshaped it, started to add new functionalities and to improve the performances of BPF. This new version is designated as eBPF (for “extended BPF”), while the former becomes cBPF (“classic” BPF). New features such as maps and tail calls appeared. The JIT machines were rewritten. The new language is even closer to native machine language than cBPF was. And also, new attach points in the kernel have been created." [1]
Definitely prefer the Github link, the short intro video gives you a very quick understanding of a good part of the value proposition, what this tool does in general, etc.
Wireshark performs poorly with opening such a large trace. Since Engineer's are costing well north of $150 per hour, having them twiddle their thumbs waiting is not economical or efficient.
Enabling a survey and filtering of the data as soon as possible from the pcap file will allow interactive examination. With Brim that identified data can be exported as a filtered, and thus much smaller file. This is similar to what tools such as Riverbed's Packet Analyzer do. PA is a .net front end to tshark type filtering and reporting. It ships with a number of useful filters to survey for common problems.[1]
Brim and Zeek appear to fill the need between Packet Analyzer on one hand and a regular Wireshark workflow on the other.
[1]: https://www.riverbed.com/products/steelcentral/steelcentral-...