Hi, it's going to depend on the specific minerals but if it's not water soluble, start with a soft nylon scrubbing brush and a bit of dish soap. You can clean them under running water, but using a small bucket to dip is often more effective for dust.
For the ingrained dirt on the surface Ultrasonic cleaners are effective at removing stubborn dirt and debris.
They can help dislodge particles from the surface of fibrous mineral, but be aware that Ultrasonic cleaning can affect the surface of some minerals, including fibrous ones. So assess the mineral’s condition before and while using this method.
Also note that fibrous minerals become hazardous when airborne. If damaged during cleaning, they can release smaller fibers that may be inhaled or ingested. Asbestos-containing minerals (especially amphiboles) should be handled with extreme care. Asbestos fibers are harmful when inhaled and can lead to conditions like asbestosis and an increased susceptibility to cancer.