"Infantile Amnesia" is a misnomer. Amnesia implies that something has been forgotten. Experiential memories simply aren't recorded during infancy to have been forgotten.

For details of a specific experience to be strung together as a memory, one must have a medium for their description. The medium is "language." Early memories are fuzy because we have few words to describe our experiences. As we get older and learn more words, our memories become more and more vivid. The more words we learn - even better if multiple languages are learned - the clearer and more distinct our ability to recall information and, incidentally, the more vivid our dreams. Memories will always be faulty because it is impossible to capture the substance of time or experience using language; but words do their best to approximate meaning.

There is a kind of internal language that brains are hardwired with. It allows for the formation of concepts which persist even in the absence of language to describe thoughts. Babies respond to things that are familiar even though they cannot describe them internally or externally. That internal language exists to faciliate survival instincts and lays the foundation for eventual learning, but does not encode experiential memory.