Indeed, js-ipfs theorically enables true p2p over ipfs, without the need for the gateways. Gateways are convenient though, because they provide streaming (i.e video starts before fully downloaded, even if its a traditional file), and seeking (i,e if you go to the middle of the video it wont fetch the first half)

Also my tests with js-ipfs have been bad (unstable browser, tab crashing after a few minutes), so I dont think its production ready. But yes, that is the dream indeed, but we should give IPFS a bit of time to solve their problems first.