Long story short: swastika is for Latvians an ancient symbol of luck and Soviets were worse than Nazis, because Soviets were doing mass deportation to gulags in Siberia and the Nazis after 1941 didn't do anything like that – they kinda "liberated" Latvians in their point of view. So the swastika remained a good symbol, not related to nazism at all.