Exactly, James. Vanier wrote about vulnerability and community. Cavey wrote about non-violence. Both their core teaching and ethics were violated by the abuses. That’s very different than in a field that is unrelated (as you said, for example, physics). I’m still not always sure, though, if there is ever a true “line”.
Your example of Wallace made me think of Orson Scott Card, who was very homophobic. His works somewhat reflected this, while also containing scenes that are arguably quite homoerotic. In cases like these, I think people should be made aware and able to make informed decisions.