I've only stayed in the Garden Wing, but I've stayed in Bay Lake Tower and it's SO much better than the Garden Wing. The GW was very recently updated and beautiful, but didn't compare to BLT. And as far as distance, I felt like we walked more to get to our Garden Wing room (we were the last one at the end of the hall) than to get to our room in BLT. Obviously BLT is a little nicer than the Tower - it's DVC and newer, after all - but I've heard others say the same thing: Tower is better than GW. Staying in the Garden Wing first would be good so you're not potentially disappointed, but I would still want to branch out and try other hotels.
Poly is really nice - my parents stayed there when they went with us a few years ago. As far as location, it's usually mentioned as being the best because you have boat and monorail access to MK and monorail access to Epcot through the TTC. Depending where you end up, it can be a pretty long walk from the main building as well.
I've never heard of GF being stuffy - all resorts are family friendly and Disney doesn't want to alienate any families. It's the flagship hotel and very popular. I've never stayed there, but it's #1 on my list (DH and I will be taking a couples trip in two years around Christmas and that's where we will be staying because it's supposed to be amazing during the holiday season). It's a gorgeous hotel - we've been there a few times for the restaurants and I'm always in awe. People were always relaxing in the lobby and I didn't get a stuffy vibe at all. Grandeur, yes, but I was never uncomfortable.
Bay Lake Tower is great - we choose to stay there for the location and the rooms are wonderful, but comparatively (aside from the actual room layout/amenities) I've heard it's not much different from the Tower. I would actually be fine with staying in the Tower, but DH only gets one week of vacation a year and he chooses the hotel. He loves BLT, so that's our "home". If you're looking to save money, BLT is not the way to go unless you can rent points.
Personally, I would stay in the Tower and then move to an Epcot resort (or vice versa), but that's only because my husband and I loved Beach Club/Yacht Club when we took a more relaxed approach and walked around the boardwalk and stopped in to see what those hotels were like. Our kids are very pro-Magic Kingdom right now, but there will come a time when we switch to the Epcot hotels for a change of pace. Might be when it's just the two of us, but the three on the boardwalk looked like they would be so much fun to stay in!
Our only split-stay we did two MK hotels (because family was staying in Fort Wilderness). I hated the split-stay because we had a lot of issues with our tickets when we switched hotels and it wasted half a park day getting it resolved. I don't want to go through that again, but if I did, I would choose MK/EP hotels and get the best of both. Good luck!