Dr. Peter Kreeft, an eminently accessible Catholic philosopher, provides insight in his article “The First Cause Argument,” which he has posted on his personal website. (It’s an excerpt from his book Fundamentals of the Faith.) Kreeft presents four arguments of St. Thomas Aquinas on this matter, two of which are particularly helpful re: your question:
First, he argues that the chain of movers must have a first mover because nothing can move itself. (Moving here refers to any kind of change, not just change of place.) If the whole chain of moving things had no first mover, it could not now be moving, as it is. If there were an infinite regress of movers with no first mover, no motion could ever begin, and if it never began, it could not go on and exist now. But it does go on, it does exist now. Therefore it began, and therefore there is a first mover.
Second, he expands the proof from proving a cause of motion to proving a cause of existence, or efficient cause. He argues that if there were no first efficient cause, or cause of the universe’s coming into being, then there could be no second causes because second causes (i.e., caused causes) are dependent on (i.e., caused by) a first cause (i.e., an uncaused cause). But there are second causes all around us. Therefore there must be a first cause.
For Kreeft’s treatment of counterarguments to the “First Cause” argument, see again his aforementioned article.
One of Kreeft’s counterarguments is especially helpful in illustrating why we can have an infinite regress in numbers but not regarding real beings:
Fourth, it is often asked why there can’t be infinite regress, with no first being. Infinite regress is perfectly acceptable in mathematics: negative numbers go on to infinity just as positive numbers do. So why can’t time be like the number series, with no highest number either negatively (no first in the past) or positively (no last in the future)? The answer is that real beings are not like numbers: they need causes, for the chain of real beings moves in one direction only, from past to future, and the future is caused by the past. Positive numbers are not caused by negative numbers. There is, in fact, a parallel in the number series for a first cause: the number one. If there were no first positive integer, no unit one, there could be no subsequent addition of units. Two is two ones, three is three ones, and so on. If there were no first, there could be no second or third.