This was it for me. I don't have a CS background but with PhD in electronics/semiconductors I can usually brute force my way through things and get a good grasp.

I got half way through Andrew Ng's ML course and felt I fully understood the concepts but spent all of my time battling with matrices and the foundation mathematics rather than building on ML understanding. If I had a better grasp of the maths then I think I'd get on better with it.