I have stayed at both and prefer the Contemporary.
The Poly has such a devoted, vocal following, on our first trip I was originally leaning toward the CR, and then switched to the Poly because so many people feeling so strongly can't be wrong.
I just didn't agree with those people. I hated walking outside to get a cup of coffee or a snack. I hated the "Very Brady Bunch Hawaii Vacation" feel of the resort. I hated the dark walls/dark carpet/dark room they have going on there. (in Poly's defense, this was in Dec 2012 before the refurb, but the pics I've seen still don't suit me).
The CR on the other hand, is magic!! A monorail goes THROUGH the resort. That never got old. The bright, airy, sleek rooms are more my style. I'm a tower girl, MK view fireworks from my balcony in my pajamas! Yes, please. My coworker prefers the garden wing to the main tower, because you get the quiet AND the walk to the MK.
The walk. Oh, the blessings of not having to wait 35 minutes for a boat, or monorail to get back to your room from the MK. SWOON. When I'm tired, and the kids are tired, to be able to get back to the room on our own timetable was priceless than placating an overcooked child waiting in line for transportation.