If something like this happens with me, I will find the time to stop and talk to someone in Transportation WHILE I am at Disney World, ideally that same day. (Also, the CMs at the Front Desk at Jambo House are all absolutely amazing, and they would have helped you with that situation for future reference.)
Because none of us have a way of knowing what was going on with that Driver, and why he didn't load y'all that morning, reporting that interaction sooner rather than later might have been very important.
Your recall of the incident will be fresher, and it gives greater incentive for the Transportation folks to have a discussion with that Driver the same day.
Not all Drivers are perfect, but the vast majority of them that I have met and worked with are hard-working, caring individuals, who want every Guest on their bus to have the best possible experience. From time to time we have ridden with a less-than-happy (or less-than-well-trained) Driver, but given the number of bus rides I have taken over the years at WDW, the number is startlingly small. Definitely less than 1% of the Drivers I have ridden with.
Transportation at WDW is not a perfect system; I don't know of anyone who would pretend it is. But overall, they do a good job of moving thousands of people around every day. The safety record is excellent, and everyone (eventually) gets where they are going. If I am taking Disney Transportation (of any kind or combination), I always allow at least 90 minutes to get where I am going on property - and the *only* exception to that is the trip from AKL to AK and back, and even then I will plan for 45 minutes, just in case the buses are full, and we have to wait a round or two.
Not everything at Disney World is "Magical", and not everything can be pixie-dusted, but I think that sometimes we all are guilty of forgetting that, myself included. It can feel extra hurtful to be disappointed at WDW; we often plan so long, and work so hard to get there, and then to have something that leaves a bad taste in your mouth happen just feels so wrong. The best thing you can do in the wake of an incident like that is to report it to the proper folks so that maybe another Guest doesn't have to go through what you did.