One risk I've heard of second hand. If you work on your personal-interests or hobbies ... then that might cease to be fun and just become "work." I've not had that experience (I've always found the projects in my career interesting even if they have non-fun overhead at times) but it is something my wife and others have reported.
Finding the perfect balance is the tricky part. I do I.T. consulting on the side, but I am not stringing myself out for a customer. Its one of the expectations when I offer my services.
No I will not be on call, no your computer turning on isn't a emergency and I will not drop everything to hit the power button.
If they don't like the terms, they're more than welcome to pay double my asking and pay a monthly retainer.
Surely it's a risk, I think with all things balance is critical. However, I've personally found the grind that can come and go with delivery dates and peculiar debugging are far more satisfying and easy to commit to when the medium is a personal passion/interest.