It would appear as though you have spent a good deal of time thinking about this. The irony is that even if 20 people posted their part time success stories, it would have no bearing on your success. It might give you a bit of confidence to go forward with it, but even that would be 'false'. If people instead post 20 part time failure stories that might make you hesitant, and that too would be 'false' as you might be walking away from a great experience.

Your trying to escape from a 9-5 job. You could do this by starting a company with a focus on making enough money part time (even freelancing) to quit, but you'd be surprised how quickly this too becomes a 'job' if you don't have a passion for it.

Your best option is to ignore most people's advice and build something you are passionate about with whatever free time you have available. Keep it as 'hobby' or 'lifestyle' as you can since that's what your seeking. If it becomes something you win. If it doesn't, you'd be surprised how many doors making something 'from scratch' opens. You still win.

Wondering if you can or should do it? That will take forever.