DS was also small when he was two, and I used a hip carrier at a lot of places. It's not as restrictive as a younger infant carrier, or as cumbersome as a backpack, but it still helps distribute the weight, and frees up your hands to some extent. Mine was from a company called One Step Ahead, and folded down to practically nothing (like a fanny pack) when he wasn't using it. I don't see it on their website now, but maybe someone else makes one?
I kind of like the idea of parking the stroller at a central location, but that means you still have to deal with it on the busses, and I'm guessing not having to fold the stroller up on the busses is one of the benefits of carrying from your DH's perspective!
What would sway me towards a stroller in your case is the fact that you are going with three kids, and the stroller is not only a place for any of them to rest temporarily, but also a place to put all the stuff that you bring or accumulate throughout the day.
Since you're going to have the stroller anyway, I would agree to play each day by ear. See how much you actually rely on it on your long days. (He might just change his own mind.) If not, and it doesn't go well without it on the first short day, you can always bring it on the other one.