Like you said.. "worry less"

Can a theoretical attacker intercept a port knocking sequence? maybe. Would a script kiddie running a new ssh 0day against the entire internet be able to do this? no.