It sounds like you just need to figure out what you want. First off, not having your husband's blessings...... What does that mean? Are you cheating or is he allowing this? That's the only part of this that would make you a "bad swinger."
The rest is up to you and your partners, you just need to be up front with that. I'm very sensual myself. I'm not a one and done type of guy myself. There are plenty of women who want regulars, friends with benefits, someone just to hang and go to events with, or even poly relationships, where you actually embrace your feelings and relationships, no matter how many there are....
In other words, there's all sorts of people and desires in this lifestyle. But if you two don't speak on these things beforehand, you're going to think it's one way, and he's going to think it's another. If you and this guy talked about things beforehand, you'd have not been shocked by his reactions and might not have even chose him at all.