I keep "flying" with my Quicksilver Sprint ultralight. Costs gas and two stroke oil to operate if you don't count the hanger I bought. 3-4K buys you in (I spent more for a much better setup: carbon prop, gearbox, ailerons, radio, egt, cht, "yarn" angle of attack indicator). Made my own nose wheel brake and steering system. Strictly "no wind" flying for me. Best Kick-A** flying I've ever had and makes all other aircraft feel down right claustrophobic. But after a bit, Valdez performance and just boring holes in the sky loses it's novelty and only having a 20 mile range and did I mention "no wind", becomes boring. Not having any trim system wears out your arm and leg (yes I could fix most of that and should but I'm just building a better airplane and someone else can re rig it when they move it.), 35 mph cruise, stuff like that becomes annoying after the "thrill" is gone.
Anyone can afford to fly. It's a matter of "requirements" and if your interests are being met. Safe "Flying" costs less than a good junker car. Investment in your DIY skills, shop equipment and knowledge to use it correctly, training and the self discipline to not exceed limitations - costs the same in any kind of flying.