There's really no right or wrong answer, just need to find the right fit for you!
I've stayed at both, and based on the description of your party, Im inclined to say keep the poly if money is not a huge issue for you.
POLY PROS:
-Location: perhaps the best on property. Monorail access to MK, plus you can walk or monorail to the TTC next door for the EPCOT monorail line. That is a huge bonus, esp for your first trip as first timers often dont allocate enough travel time. (You can also see MK fireworks from the resort grounds).
-Food options: Poly has some of my favorite. Your kids will love Ohana. I can never get enough of the breakfast at Kona Cafe. I love grabbing a quick morning coffee and bagel at Kona Island before the parks. And for an adults only trip when your kids are in bed perhaps, Trader Sams is the most fun you'll have a bar. Oh and did I mention there is Dole Whip on property at Aloha Isle? Capt Cooks also has great quick service options.
-Atmosphere: The Poly just FEELS like you're on vacation. I imagine this will be especially true for you with your cornfield background
-Pool: If themed pools are important for your kids, most families report that their young ones like this pool the best. Theres also a second quiet pool with a nice pool bar for frozen beverages.
POLY CONS:
-The Room: For a hotel that costs upwards of $700 a night....the rooms just don't cut it. Now, there's nothing bad about them, but there's definitely nothing special either. You can get a luxurious accommodation at a four seasons for what you pay for a completely basic hotel room at the poly. I felt very disappointed when I stepped in to my room there, and would not pay that amount for that room again (DVC rooms are super nice however). That said, with a large family, you may actually appreciate that there is lots of room to spread out. They are some of the largest hotel rooms in all of WDW, you can get 2 beds plus a pull out couch.
WL PROS:
-Atmosphere: not tropical like the Poly, but at WL you really feel like you've escaped somewhere. It transports you to the pacific northwest and you will feel a million miles away from the parks. Theres a stunning lobby, a babbling brook, beautiful grounds, and peaceful lake to look out on. It's just a wonderful place to come home to.
-I stayed in a Copper Creek room, and found this to be leagues better than the Poly in terms of how it was appointed. Just gorgeous and comfortable. But I think even the standard WL rooms feel more unique than the Poly rooms. However, all standard rooms at WL are 4 person occupancy and smaller than the poly. So your little one would need to be in a cot/crib, and that could be a deal breaker for your family.
-Pool: The slide isnt as intense as the Poly, but the theme is fun at both the active and quiet pools at WL. Plus your kids will have fun running over to watch the geyser shoot every hour.
-MK Boat: Such a fun and unique way to approach the Magic Kingdom. There's also a separate boat loop to Contemporary and Fort Wilderness campgrounds which increases your available dining options.
WL CONS:
-while Geyser Point is one of my favorite spots on property, Roaring Fork is not a great quick service place imo (I couldnt enjoy much besides the pulled pork). In addition, there is no coffee stand situation for your mornings like Poly has with Kona Island (if those things are important to you). So you must get morning breakfast/coffee in Roaring Fork which turns into more of a process than need be.
-The boat is fun for MK, but for every other park you will need to take a bus.
Though Wilderness Lodge would be my personal pick b/c of room quality and ambiance, my party (adults who have been many times) has different needs than your family. And I'm inclined to say that the ease of transportation plus the size of the Poly rooms will be the biggest benefit to you.
