Most people hold to the belief that the Romans used Gold rings to signify marriage on their third index finger.  The symbol of purity and an unending circle reflects the nature of marriage.

While it can be considered Pagan in origin, there is nothing inherently wrong with the tradition.  We could also say that no one should use the words "Thursday" or "September" as these are pagan in nature. (Thors Day - from Norse Mythology and Religion)  (Sept meaning Seven - Part of the Roman Calendar)

If it is something that bothers you, you do not have to legally wear a ring to be married.