Sending drawing commands to an X server is no more difficult than sending escape code sequences to a terminal. Simpler, even, because it's actually designed for that. And yet it's also more flexible. If your GUI framework of choice is limiting you, you're welcome to write one, just like you're already writing a TUI framework.