My husband is sort of the same way. We went first time as a family when my kids were 2 and 4 (almost 5). (I planned 3 trips prior to kids, and he loved them, but had never been on a vacation before in his life until we went the year before we got married, stayed offsite, very inexpensively and he had a ton of fun). We had such a great time, and discovered how easy and relaxing it can be with two young kids (stayed 11 nights and had a BWV with the kitchen and washer/dryer which was so great).
He works long hours, and can't always take time, so I took the kids without him the first time when they were 4 and 6. We found a great airfare, great AP rate (we had just gone and bought AP's), and just went for 4 nights. The kids and I had such a nice time! I just needed to get away, and it's the easiest trip to take without another adult.
I'm also a SAHM, so am flexible. I've taken the kids several times without DH, or many times he has to leave early to get back home for work. Last year, my mom flew over from Naples for two nights and joined me and the kids (she lives only 3 or 4 hours drive, but didn't want to drive it, so she flew, and doesn't really "get" Disney but came along anyway, I think she enjoyed it).
One thing I do know about my DH: on vacation, he wants to go to the gym or a long run every morning, which to him is truly a vacation. He enjoys some time by the pool as well. So, if we go together, I make sure to take the kids myself out for the morning and we go to a park and take it at our pace (usually slow, DS has asthma so he just started not riding in a stroller very late, like at age 10, and uses a GAC to rest sometimes in lines). DH goes to get a good long workout, showers, and then meets us for lunch. I also plan special meals (we're not big eaters, but enjoy trying new things), like one year we had a dinner (before his flight home early) at Bistro de Paris (it was all I could snag while we were at the pool one day at noon, I just called dining). We had this super long meal with wine pairings, it was so elegant and wonderful (I won't tell you how much it cost, but he said it was so worth it!). One year we did an illuminations cruise on Breathless.
I guess I'm saying that I do try to make it a true vacation for him when he does join us. Last December (2008) was the last time he came, and I had gotten us a deal with a room discount plus free dining (yes, that was the offer, we stayed at the BWI) He only came down for 4 nights I think, and really enjoyed it. The dining plan is truly very generous!
I've been busy the last couple of weeks trying to figure out what to do this coming spring break. I figured if it's Disney (DD wants to do that) he probably won't come along. Well, I booked the BWV with the new discount (I can't ever justify $610 a night, even if I could afford it, just can't stomach that after we've used points several times for stays at the Swan and Dolphin for free!). The 45% off was reasonable at least. I figured we can cancel, but got that booked. Last night, he said that Disney sounded like the best option! Um, I'm thinking, "Who are you and what have you done with my husband?"! He may not stay the whole week, but he loves the Boardwalk area (the BCV isn't available for the discount, or we'd give that a try). So he's coming along, may have to come home early, but that is fine.
While we usually have one big family vacation every year all together, at least, he doesn't mind if I (frugally) take the kids again to Disney. I also take them to visit family in Missouri without him every year.
I'd at least consider taking them without him, or have him only come part of the time, or try to plan it around what your DH likes as well!
I hear from my DSis mostly, that she wants to take her kids so many other places for travel. I try to explain that there is a big difference between "travel" and seeing the world, and "vacation" where we just relax, unwind, enjoy some fun silly rides and things, and good food, and just spend time together. We go to Hawaii (used to live there, DS was born there) every other year now, and it's also just family time with familiar things.
Good luck, and sorry so long. My morning coffee makes me "chatty"!!!
His idea is a beach vacation and we do that every summer and our November trip is the girls trip. It really isn't as hard as some people might imagine. The kids love going and are as good as gold. My only suggestion is spending the extra for a double stroller, keeps everyone safe, legs fresh and noone gets lost. 
So far for this year we have booked Washington DC for 8 days, NC for 6 days and WDW for 8 days. I hope he doesn't catch on.
