I’ve found bamboo and hemp fibre to be excellent for clothes, especially shirts. I can’t wear synthetic fibres as tops. It’s not so much a problem for pants.

The dyes are a worry too.

There is a small designer in Byron Bay (Australia) called Kashi who make very good quality clothes with ethically-sourced materials and natural dyes, and they produce them in a small village in Asia (I think it may be Thailand), where all the profits return back to that specific village. I think the main designer also gets them to contribute to designs as well (but don’t quote me on that).