I would think the average casual collector has in the region of 20 different minerals. A serious collector will likely be in the three to four digit range.
Apparently the Smithsonian National Museum of Natural History has over 45,000 individiual specimens. And the Natural History Museum in London has approximately 185,000 mineral specimens, all organized by species and on a systematic basis.