I think this problem is mostly already solved with more capable models.

Here's the shoe-identification experiment with GPT-4 and it's not confused by a suit jacket:

https://imgur.com/a/cPP6pH7

It's also not confused by a tuxedo:

https://imgur.com/a/mWMxZbA

Also tried the same examples with Google Gemini 1.5 Pro and it doesn't hallucinate either.

https://imgur.com/a/3Yv7wuP