I think the skill we need is holistic systems thinking, which aligns with what you said. Essentially a good system becomes the guardrails for the LLM. There can be dozens, hundreds of aspects of a business or product that need to be considered at once to create a good system. Generally speaking I find by the time you have gathered all those aspects, explained them sufficiently to the LLM, it no longer gives good results. So a good systems thinker can take all that input, break it down into logical components and then get the LLM to work on those components.

You still need to keep an eye on it, because the components necessarily lack context of 100% of the project, but you the human can actually keep it all in your head and can catch the nuances.