Hmm maybe try feeding it with some worm castings or some green manure (soaked comfrey) to speed up leafy growth? Also rosemary doesn't like to stay wet so making sure that it has plenty of drainage in the 5 gallon container, and keeping it in a full sun area. It may just need another season to get big enough to harvest though! You can propogate rosemary by taking 7 inch cuttings and sticking these cuttings in pots, and they will form into new plants, that way you'll have more plants to harvest from!