It's probably all due to e-mail spoofing, the "From" address is easy to forge, and what address better than yours to pass the spam filtering. Spam filters that analyze the mail body in search for certain words will flag it as spam without problems, GMail's filter is a good example, which has blocked Viagra spam that apparently I send to myself (which is ridiculous, why would I have any need to remind myself to use something I don't need? :D)