When we went in March a couple of years ago, we wore shorts and T-shirts during the day. At night, we wore jeans or shorts with a light jacket. There was one night at EPCOT that it was pretty chilly with the wind blowing.
But my DS did swim in the pool every afternoon. (It was a bit too chilly for me.)
So I would say, pack like 2 pair of jeans to change into at night and just wear them more than once. And take a light jacket.