I'm glad you mentioned solubility.

but a mineral like aluminum silicate, which is almost completely insoluble, is quite harmless for the human body.

AKA bentonite clay, which we use as an adsorbent to remove toxins during water treatment, and which has been used as a medicine traditionally ingested by some cultures, with varying effect.