It is a lot of work, but doable. I have 5 kids very close in age and my husband and I have gone 3 years in a row with them- in summer! This past summer oldest was 8 and youngest 1. The year before 7, 5 (turned 6 while we were there), 4, 2, 9 months. First year oldest 6 and pregnant with baby # 5. We each pushed a double stroller, packed lots of snacks, drinks and sunblock. We utilized rider switch and took breaks when we needed it. We expected kid tantrums, and adult tantrums. We had some stressful moments but many more amazing moments. It's so worth it! We were able to do very long days in the park or even leave and come back. We made the most of our time there.
Overall, Disney is designed for families- they make it as painless as possible. We went to a local amusement park a few years ago when I was pregnant with #5. This was after our first trip to Disney. My husband had to ride the flying whales (same ride as Aladdin's Carpets) 3 times in a row. They wouldn't let me ride pregnant. I had to wait with baby #4 while he rode with each of my older girls separately. It was a pretty small ride. But since my girls were not tall enough to ride alone (you had to be 48"), he couldn't stick the two older ones in one cart and ride with the younger. I asked the ride attendant if I could ride with my baby on my lap and sit with one kid, while he sat with the other two. Absolutely no pregnant people allowed on the ride. It took almost 20 minutes to get through all three rides, and my 18 month old knew she was missing out on something fun. She was sitting in the stroller crying that he couldn't manage a 4th ride with her (he was ready to lose his lunch after 3 rides). This happened a couple more times, but this would never happen in Disney. They really cater to young families (and I am sure their insurance policy is a lot bigger than the local theme park we were at lol).
My last piece of advice is to avoid table service. We have done just a couple of ADR with the kids and it was always a disaster- and our kids are generally well-behaved. It's just hard with that many small kids. Pack meals in a cooler or do quick service.
Have fun! I bet the last adult will cave and end up going with you guys.