Your husband is in no place to judge the state of your soul. Nor are you. Neither of you has a way of knowing how holy you would be if you were not receiving the sacraments. Nor do either of you know how much better he would be if he were receiving them. Certainly, there is no way to draw near to Christ and not be affected by him. We have the very words of Jesus on the matter: "He who eats my flesh and drinks my blood has eternal life, and I will raise him up at the last day" (John 6:54). Only Christ knows how holy you are. But we can certainly know that you are better off for receiving him than not.