I totally get it. We can only travel in June, kids start activities the Monday after the 4th of July, they get out of school the first week in June, so we only have 5 weeks.
We went to Japan 6/17-7/5, my original plan was to return on 7/4 but someone got those tickets before I could book them on BA. We had an amazing trip and our timing was perfect. We hired a guide for our first day and he said the crowds had definitely decreased, he said May was crazy (not just Golden week, but most of the month). It hardly rained and the weather was in the 80s, by the end of our trip the weather was in the 90s but nothing crazy. At Disney and Universal we also got super lucky with low crowds at both theme parks.
We use AA miles to fly to Japan and BA Avios to fly back. The fees on BA are crazy but since we didn't pay for hotels (used Chase travel portal) it wasn't that big of a deal.