History is choc full of people that started something on the side and turned it into a profitable business. Absolutely pursue it, knowing that

a: no tree has even grown taller than the sky

b: Rome wasn't built in a day