I wholeheartedly disagree. A much more important factor is how ready you are and what kind of support structure you have to start out with. If that is later in a career (and when you have a family) so be it. It's the right time.

Founded current company with two young kids and just had a third. Pulling together the right team way more of a factor than having a family.

> Why dont you start working on weekends/spare time on your idea

I suspect you don't have a family ;)

If you have a family it is better to start moving on your idea within normal business hours, where legally possible (some creativity may be required).