Great piece you've written here, and yes, men do need to hold one another accountable.
I think what someone is wearing has been used in the past as a convenient defense by men, but in most cases probably has nothing to do with what she's wearing, as women get raped and grabbed wearing sweatpants or overalls.
I once had a boss call me to meet for lunch. He led me to believe it was in regards to a promotion and so I went. I did ask my then boyfriend to go along just in case (because I thought it weird not to just meet at work) and the boss admitted he'd wanted to sleep with me?? When I asked him why he'd think I would he said I'd smiled so big at him walking into work. The boyfriend (now gone) accused me of leading him on?
This was a job in which I wore a uniform the same as everyone else--pants and a polo shirt.