Hi simbasmom, and welcome to the DIS!
You won't have any trouble filling up those 3 non-park days, trust me!! My kids just spent 3 days there with their grandparents and they only went into the parks for a short time one afternoon.
They had a great time at the volcano pool and visiting the arcade. My parents went to Cirque du Soleil one night and they got to go to the Neverland Club and they LOVED that. Every afternoon there are craft activities set up and they got to make all kinds of fun things. They also do hair wraps by the pool - a MUST if you have girls.
There's plenty to keep you busy at Downtown Disney, including a kiddie carousel and train, a playground, interactive water fountains and everybody's favorite - the
Lego Imagination Center.
If you head over to the Boardwalk (one of our favorite places) at night they offer surrey bike rentals, face painting, fortune telling, midway games, sword swallowers, magicians and the like. You can also stroll around the lagoon over to the Beach Club for some delicious burgers and shakes at Beaches and Cream, or walk over past the Swan to Fantasia Gardens for some miniature golf.
At night you can watch the fireworks and the Electrical Water Pageant from the beach and there's a nice path connecting the Polynesian with the Grand Floridian if you'd like to go for a stroll one evening.
Have fun!