The pastor is half right and half wrong. He's right about the fact that the Greek word for brother (adelphos; plural adelphoi) does mean sibling and about the fact that Greek has precise words for cousin, nephew, and other close relations. He's also correct in pointing out that adelphos is the word used whenever there's a mention of Jesus' "brothers" (Mt 12:46; 13:55-56; Mk 6:3; Jn 7:5; Acts 1:14; 1 Cor 9:5).
He's wrong, though, to infer from that bit of grammatical truth that Mary had children other than Jesus, mainly because he's wrong in claiming that adelphos can only mean sibling in the New Testament. In Matthew 13:55-56 four men are named as brothers (adelphoi) of the Lord: James, Joseph, Simon, and Jude. Your former pastor concludes wrongly that these are at least some of Mary's other children. The New Testament proves otherwise.
In John 19:25 we read, "Standing by the foot of the cross of Jesus were his mother and his mother's sister, Mary the wife of Cleophas, and Mary of Magdala." Cross reference this with Matthew 27:56: "Among them [at the cross] were Mary Magdalene and Mary the mother of James and Joseph, and the mother of the sons of Zebedee." We see that at least two of the men mentioned in Matthew 13 were definitely not siblings of Jesus (although they're called adelphoi); they were Jesus' cousins--sons of their mother's sister.
The Bible is simply silent on the exact relationship between Jesus and the other two men, Simon and Jude, mentioned in Matthew 13. This proves two important things. First, it proves that the Greek word for brother is sometimes used to mean something other than sibling, and it proves that Matthew 13:55-56 in no way demonstrates that Mary had other children.