No at all. We did something similar once. We had a package at ASMO, so our passes were good all through the day Friday(our last day). Due to flights, we were originally planning to leave fairly early that day and not do the parks at all. Then we ended up extending the trip by 1 day and we went to the Poly for that day. We checked out of ASMO early Fri, went and checked in at Poly. Then we had breakfast at Chef Mickeys and it was on to the MK around 10:00am. We stayed until the 3:00 parade was over and then went back to the Poly. We settled in our room then, went to the pool and even caught the torch lighting ceremony. After washing up, we went to dinner at the Kona, watched the EWP from the beach, then had a drink at the Tambu Lounge to end our vacation. We ended up getting to do more park time, and enjoying the Poly. It was a great day!
Im sure you can find just as much stuff to do at WL.