The number of browser exploits requiring JS compared to the number of browser exploits not requiring browser JS disagree with your assertion.

The distinction is that JS is a complex and turing-complete language that browsers put a lot of effort into making fast and memory-efficient.

And as a result JS doesn't exactly have the best security record.

1. In the last year, numerous security holes have been found in the JavaScript implementations of every major browser.

2. Even if you assume JavaScript sandboxes are secure, the actual expected behavior of the language is that all code runs in a big global scope. A perfect sandbox would prevent sites from running arbitrary code on your machine, but any JavaScript running on a page effectively has access to all the data on the page. That means every ad, tracker, etc. on the page has access to all the data on the page, including passwords on forms, etc.