Would you let your children get close to relatives that think mommy is a sinner?

My family thinks that I will one day regret my life and the way I am raising my kids because I don't believe in Christianity. My kids know my family, but I don't let them get close or spend to much time alone I don't want my kids around people that think that mommy is going to hell. Even if my family doesn't say anything... all their standards of right and wrong are everywhere and my kids pick up on them and realize that we do 'wrong'. I don't want my kids getting close. Am I wrong? Would you feel the same? If my kids choose that when they are older, then fine. But when they are young, I don't think I should be undermined as a parent, even subtlely.