We ditched our dining room and building a tiki bar and in the other room we're building a lounge space - no TVs. Just spaces for us to relax and feel a bit of a break from the rest of the house/life. But we really don't have spaces that one of us goes to hang out for long periods, but we really enjoy each others company so that's not necessary but I do believe everyone needs a bit of "down" time too.
This sounds very similar to our current and previous home. There are little spots with chairs, small couch, bar, desks, etc that people can go to "retreat" (read a book, have a drink, etc), but there are not dedicated rooms for any one person (which is what I think of with terms like "man cave/she shed"). We also have lots of outdoor space (front/side and back porches) where one person can go to lie in a hammock, listen to music, etc.
In our previous home, we converted the "formal" dining room into a very cool pub room. In our current house, we have been purchasing vintage furniture to have a tiki lounge area. We have found these rooms/spaces get way more use on a daily basis than they would have in their original state.
I'm an only child and grew up with an entire upper floor to myself, except for the walk-in storage. I struggle without having a retreat to have silence, books and art supplies. My DH and 2 DC have been home since last March. I wish I had a dedicated space and have taken to hiding in the pantry on occasion.
Is the issue that your husband and kids don't understand that you need private/quiet time? Fortunately, I have a household of introverts so everyone respects everyone else's need to retreat to recharge. We spend a lot of quality time together, but then everyone scatters to their own corners for a bit. Maybe you should have a conversation with them to explain why this is important to you and come up with some ideas on how to implement.
You should be able to find something that works for you regardless of the size of your home. Our previous two homes were over 3000sqft, but our current is less than half that size. There are still probably at least 10 spots where someone could go for "alone" time and since there are only 5 of us, that's plenty.