You'll need four to six years just to earn a bachelor's degree, which is the case for many other professions. Then comes graduate school, which can take anywhere from five years for theorists to seven or eight for experimenters and observers. The good thing is that you can always be a backyard astronomer and have a great time, there just isn't any work in it. If you're interested in astronomy but don't want to pursue a career, I would absolutely recommend following astronomy as a hobby and joining some local groups.
In fact, a lot can be done with just a telescope and some dark skies. You can even discover asteroids relatively easily. By the time you become a full-fledged astrophysicist, you could have become a doctor. Not surprising, as astrophysicists must earn doctoral degrees and make two or three postdoctoral appointments, just as doctors must have residencies after medical school.
But then, you didn't want to be a doctor anyway and chances are you've always had your head in the stars and your eyes up to the sky. Astrophysicists are paid well above the national average salary, but getting there can seem overwhelming and the field is very competitive.