>Take the naive case where you downscale a line of four pixels to two pixels - you can simply discard two of them so you go from `0,1,2,3` to `0,2`. It looks okay.

This is already wrong, unless the pixels are band-limited to Nyquist/4. Trivial example where this is not true:

  1 0 1 0

If such a signal is decimated by 2 you get

  1 1

Which is not correct.