It's also possible that the cases where you felt it bogged things down used the WRONG actions. It only works when the actions and descriptions accurately reflect the behavior of the characters in the situation. The fiddling has to be showing something about the characters. Are they nervous? Are they insecure? Are they lying? It has to fit the situation and not just be fiddling.