What is the best approach for working with very large collections of network traffic (500GB+)?
Specifically, I'd like to be able to filter packets which match various payload and protocol criteria (e.g. all TCP packets to port 1111 with the string "hello" in the payload). This is very hard to do since most of the tools I look at load the whole file into memory. I could script something in python but the best angle I can come up with would be to just loop through the whole file for every single search I want to do which would take forever at this scale.