Ah. I use RealESRGAN (or one of its descendants, rather) as a first pass upscaler before high-resolution diffusion. If you skip the diffusion step, of course it'll be faster.