"Skipping Mass affects no one but myself and God." And God? Since when are human persons more important than God?
There is an infinite difference between measuring God by our standards and measuring ourselves by his. We have to start with God when we look at everything. Since he came first, he comes first. He is not just a bigger version of us. He has commanded us to "Keep holy the Lord’s day," and he has a right to demand this of us. Anything of value in our lives (including human life) has value only because of his infinitely greater value. But Mass is not just a matter of acknowledging his sovereignty, which is what worship is. It is being present at the foot of the cross and giving thanks for the Passion and death that he endured for our benefit.
He loves us so much that he underwent all that suffering to demonstrate his love for us in a way we could somewhat understand. To skip Mass is one of many ways of turning our backs on that love.