You are correct. In fact old English didn’t have it as well. "Yes" comes from "gese" which means "So be it".

According to wiktionary entry on Yea.

From Middle English ye, ȝea, ya, ȝa, from Old English ġēa, iā (“yea, yes”), from Proto-Germanic ja (“yes, thus, so”), from Proto-Indo-European yē (“already”)

Wiktionary about yes says

From Middle English yes, yis, from Old English ġēse, ġīse, ġȳse, ġīese (“yes, of course, so be it”), equivalent to ġēa (“yes", "so”) + sī(e) (“may it be”), from Proto-Indo-European yē (“already”). Compare yea.

So "yes", seeming so simple, was once a compound word.

I was taught "ita vero" for yes and "minime vero" for no during Latin lessons at school.

Latin had a few ways to affirm or negate. Both "ita" (it is) and the more forceful "ita vero" (it is true) could be used for yes, but you could also repeat the verb from the question with modification to affirm. "Have you eaten?" -> "I have eaten."

For negation you could do the same verb-repeating, but add "non" to negate.