We would just deny, ask who's spreading such a serious and controversial rumor like that and what would give them any reason to do it.
Recently while at work, someone I work with randomly blurted out, "I get the impression you guys are swingers, are you?" The reply was nope, sorry to disappoint you.
However thinking about it later, how does someone even think it appropriate to bring up randomly to an acquaintance and furthermore, what would make this person believe that the person asked would reveal such a thing to a bunch of other random acquaintances who happen to also be in the room? We would absolutely love to be able to figure people out. Baffling.