its very straightforward. video comes in frames, what happens in between frames is that the same image is being displayed for that whole time. there is no way to end a scene half way through the frame - there is no visual cue or anything.

not to mention how you would go about defining the ends of scenes...

no standard format that i am aware of has variable framerate. it makes no sense.