I try to be very conscientious about things like this due to my and my sister's food allergies. If I'm getting a shared salad for us at the grocery store salad bar, I'll avoid anything that's between the mushrooms and the front of the bar, and probably the things on either side too. People aren't very careful.
That being said, I've never actually seen anyone use the tongs for more than one thing, unless it was two or more trays of the same type of food. I can see how it would be okay to share tongs between those. But taking them back to the table? Wow, no.
I don't know what you'd be able to do in that situation, aside from staring in a very shocked manner, but I would desperately try to avoid buffet-style gatherings with that family in the future, unless there was someone on the other side of the buffet still serving the food. I've seen that before and it tends to work pretty well at some places.