I accidentally stumbled upon this and did not expect such a speedup. It seems anything less than cu118 does not properly support the RTX4090 (or H100).

Bumping to cuda12.2 with pytorch2.0.1+cu118 made my SDXL go 50% faster and ESRGAN 80% faster on the 4090.

You can also run PyTorch cu121 nightly builds,

These also allow `torch.compile` to function properly with dynamic input, which should net another 30%+ boost to SD.

Is there a trick to getting pytorch+cu121 and xformers to play nicely together? All the xformers packages I can find are torch==2.01+cu118.

Edit: After a bit more research it looks like scaled dot product attention in Pytorch 2 provides much the same benefit as xformers without the need for xformers proper. Nice.

xformers has to match the PyTorch build. For PyTorch nightly, you need to build from source.

xformers still has a tiny performance benefit (especially at higher resolutions IIRC), but yeah, PyTorch's SDP is good.

This comment brings a tear to my eye.

The underlying problem is the community's decision to make users manage this in the first place.

This is an example of a setup.py that correctly installs the accelerated PyTorch for your platform:

https://github.com/comfyanonymous/ComfyUI/blob/9aeaac4af5e19...

As you can see, never merged. For philosophical reasons I believe. The author wanted to merge it earlier and changed his mind.

Like why make end users deal with this at all? The ROI from a layperson choosing these details is very low.

Python has a packaging problem, this is well known. Fixing setuptools would be highest yield. Other package tooling can't install PyTorch, for example: https://github.com/python-poetry/poetry/issues/6409#issuecom....

PyTorch itself is wonkily packaged. But I'm sure they have a good reason for this. Anyway, it goes to show that you can put a huge amount of effort into fixing this particular problem that everyone touching this technology has, and the maintainers everywhere will go nowhere with it. And I don't think this is a "me" problem, because there is so much demand for packaging PyTorch correctly - all the easy UIs, etc.

> But I'm sure they have a good reason for this.

CUDA and ROCM make this an intractable problem. Basically there is no way to sanely package everything users need, and the absolutely enormous, cude/rocm versioned pytorch packages with missing libs are already a compromise.

TBH the whole ecosystem is not meant to be for end user inference anyway.

Sorry, no idea what you are talking about.

I am talking about dynamic shapes in torch.compile.

You seem to be talking about software packaging. You also make heavy use of the word "this" without it being clear what "this" is.

The two most popular stable diffusion UIs (automatic1111 and comfy) have longstanding issues with a few known but poorly documented bugs, like the ADA performance issue.

For instance, the torch.compile thing we are talking about is (last I checked) totally irrelevant for those UIs because they are still using the Stability AI implementation, not diffusers package that Huggingface checks for graph breaks. This may extend to SDXL.

Pretty interesting. Using nightly + cu121 im getting 8.18 it/s, another 5% improvement vs 7.78 that cu118 gave.

This was one of the reasons I skipped the 4090.

So few people have the technology that I knew I'd be spending significant time figuring out solutions to problems.

The other reason is that I'd wait a few years and get some 6090 with 4x the VRAM.

I doubt future generations will up the consumerarket cards past 24gb.

They know it's a bottleneck for LM training and inference, so they'll want to extract value by reserving it for the professional line cards

Good find!

I put Karlo diffusion by Kakao running as an inference endpoint on GPUX. Karlo Diffusion is more controllable, similar to Dalle-2.

Make some images here https://explorer.gpux.ai/inference/gpux/karlo

very cool how does this compare to NVIDIA nvfruc?

https://docs.nvidia.com/video-technologies/optical-flow-sdk/...

The target application is somewhat different. FILM is more focused on high quality animation generation while nvfruc is for real-time streaming.

nvfruc is extremely limited in its capability, basically small motion interpolation between two frames within a very short interval. FILM (and other deep learning based interpolation methods) is capable of interpolating two frames with large motion differences at a significantly larger computational cost. I don't understand why you're trying to imply nvfruc could be better at both cost and quality? Even Nvidia won't make such bold claims.

Thanks.

People still go to traditional-school?

what do you think?

Much needed thank you for doing gods work.

DDD is just microservices without head of line blocking not sure why everyone hates it. Instead of having 5 microservices you just build an umbrella app exposing the same 5 API. Calls into the umbrella are now at microsecond speed vs 10s of milliseconds in most (run of the mill publicly available) clouds.

Mouse lag needs to be fixed.

Yeah for sure, many more optimizations to be made. Thanks for the great feedback!

Would be cool to move realworld assets into virtual worlds, and vice versa.