I'm like you - numbers are just hard. (it gets worse across languages, too). My husband on the other hand can remember numbers real well
I had problems at school as long as most of it was memorisation based - i had trouble remembering vocabulary lists (made worse by the fact that i already knew the words, i think), math and science formulas, all these things that have no pattern or justification or them, that are arbitrary (math and science became easier once things were derived from principles, but i never could remember the constants beyond 1 or 2 digits). The same goes today for the details of programming languages/commands etc.
I on the other hand have an excellent memory for things that are not arbitrary - text of plays, music and songs, scientific derivations - and for larger picture elements. And stories (i cant tell you what the exact syntax is for something in python but i can tell you stories around python, zope, how we built what, problems we had and the time i just deleted the database and rebuilt it from the copy that was in memory)
And that is the problem with passwords, they have to be arbitrary to be secure. But arbitrary (for me) means forgotten unless there is a pattern of derivation that I can remember. A song, a rhyme, a proverb, a book title...