I have a sneaky tip for you. Every night - I think it is at 6pm but am not sure - there is a 'Ceremony of the Keys' - it is a sort of changing of the Guard but much more low key and held at the entrance to the dungeons , near the portcullis which goes out to The Thames ( all a bit sketchy in detail - it has been about 30 years) - anyway, if you ask a Beefeater the way and tell him you are supposed to be at the ceremony, he will direct you. Better still find a soldier - they can show you around though not into the Crown Jewels display rooms.
The Tower of London is extraordinary as you say. Michael Caine was stationed there when he did National Service and the Kray Twins (Reggie and Ronnie) were imprisoned there. David Bailey did a photo shoot of the Krays while they were in there! Trivia to add to the stories of The Tower.