They were offered one mass-market opportunity on a silver platter, which they took: When Nintendo asked them to design the N64 GPU. It didn't seem to be very profitable for them.
It's very unclear in that era that there is a big market for 3D graphics at home. So their big customers would buy the cheap cards but in low volumes -> bankruptcy. And maybe there's either no big consumer market, or it grows too slowly to replace the loss of their main business.
What stopped SGI from offering such $199 add-on cards, but with their name on it?