First, it is not unusual to become distracted during prayer. When this happens one may try a number of things—for example, praying for a shorter time but striving to be more focused as you pray. This might mean that instead of the whole rosary you say only one decade but will have fewer distractions while praying it. Then, little by little, lengthen the time. Be sure to set aside a special time and place for prayer so that you will not feel like you should be doing something else.
Second, it is not wrong to have a special time of meditation during the rosary, separate from saying the prayers. In fact, this seems to be envisioned as the normal way of doing it in Pope Paul VI’s 1974 apostolic exhortation on Marian devotion.
In it, he noted the various elements of the rosary and then observed that each of them
has its own particular character which, wisely understood and appreciated, should be reflected in the recitation in order that the rosary may express all its richness and variety. Thus the recitation will be grave and suppliant during the Lord’s Prayer, lyrical and full of praise during the tranquil succession of Hail Marys, contemplative in the recollected meditation on the mysteries, and full of adoration during the doxology. (Marialis Cultus, 50)
This seems to envision meditation on the mysteries as a separate element, alongside the Lord’s Prayer, the Hail Mary, and the doxology (the "Glory Be"). He characterizes the attitude displayed in the Hail Marys as being "lyrical and full of praise" but the attitude as "contemplative" in the meditation on the mysteries.
It should be noted that Paul VI was not trying to establish one and only one way of saying the rosary, but it seems that what you are doing is in line with what he envisioned.