It could be due to what you're eating. Swimming is an excellent form of cardiovascular exercise but I would recommend everyone to start lifting. Packing on muscle mass helps to burn additional energy. You mention about having a thin build, all the more you should get yourself in the gym, get on a training program and start gaining some size.