"However, a 10x improvement would be very interesting to me."

I managed to optimize a similar problem as yours until it was I/O limited. 1/10th the CPU would not change anything in the overall system.