I've used something to this effect before, and you can use child selectors on e.g images to effectively exclude elements.
Note that the cost of this lazy method is very poor performance (css filter over the entire document), and lack of control over dark colours (inverted colours don't necessarily work best, shifting hue can help but you probably don't want the same as uninverted colours because they tend to be too dark and lack sufficient contrast).
If you use just invert(100%), no grayscale, on the body, you can add img { filter: invert(100%) } to reverse images back to their original color. However, if an image has any transparency or translucency, it may be harder to see with what the background color has become.
html.dark-mode { filter: grayscale(100%) invert(100%); }