I would be annoyed but I have to ask, why is talking on speaker phone rude but 2 people sitting there talking to each other isn't? Is it because you can't hear the whole conversation and that bother you?
I guess my thought on it is, if it isn't a place where being quiet is expected- like church, or a library, or movie, etc then people are free to talk to whoever they want, even if that person is on the other end of their phone.
Personally, speakerphone just bothers me in general. I will use it at home when I'm by myself if I'm trying to multi-task, but hearing other people on speakerphone just grates on my nerves and also makes me feel anxious and uncomfortable. I think it's because the person on the other end often doesn't know they are on speakerphone in a public place, or even if they do they don't know the other person's surroundings.
A few other reasons--
1. At least 90% of the time, the volume is much higher than a normal conversation between two people. I know I talk way louder on speaker or car phone than I do if I'm holding it up to my ear or if I were in person.
2. It infringes on the other people around them. If someone is having a speakerphone conversation, others often feel the need to keep quiet so their voices aren't being picked up on the call. If you're in the bathroom, you might be uncomfortable with an unknown audience.
3. The person on the other end can't pick up on social cues. If you are having a conversation in person, you can see if you're speaking too loudly and bothering people sitting nearby, you can see if there's an interruption where you need to stop talking, you can see if the other person is distracted, you can see if there are small children and you should modify your language, etc. Speakerphone eliminates all those cues so there are more opportunities to be rude.
Sort of OT, but recently we were in a public restroom and thought a woman was on speakerphone with a man in the stall. When she came out to wash her hands, my daughters noticed that it was actually FaceTime. Busy bathroom with lots of toilets flushing and hand dryers and they just kept talking even louder so they could hear each other over the noise.