DTube Uploader, born to be alive!
Before I start talking about the uploader, I have to talk to you about how I joined the DTube team.
It goes back to 6 years ago! (Yes I know DTube to less than 1 year ^^).
It started with the meeting of @Heimindanger as an employee both in a company where have developed an eCommerce website.
We worked together a good year before he flies his own wings!
We kept a good distant relationship.
One day he contacted me and asked me if I wanted to participate in the project, he explains the project, I did not know Steem, IPFS, Meteor, crypto-currencies... I thought about it and I said yes!
The different versions
I started contributing to DTube on version 0.3 4 months ago with the possibility of translating DTube into several languages https://steemit.com/dtube/@heimindanger/dtube-0-3-this-update-contains-so-many-new-features-i-could-not-find-a-good-title-for-it.
Then, I hooked up with the creation of any part of the uploader for version 0.5 only 2 months ago https://steemit.com/dtube/@heimindanger/d-tube-0-5-big-optimizations.
Finally the possibility to have a preview of the video on the scrollbar, as well as the ability to send the Snap to resize and embed the new logo DTube, there is 1 month in v0.6 https://steemit.com/dtube/@heimindanger/d-tube-0-6-pushing-it-to-the-limit.
My first article!
If you followed, I'm quite new to Steem and it's also my first article on the internet, thanks in advance for your indulgence ^^
The subject of this article is about the uploader! I decided to do this article because many people are wondering about the operation, I thought it was a good reason to dive into the bath for this first article!
Why uploader?
To answer problems that users were facing and because it was not possible easily in decentralized mode!
The first need was to encode the source videos in a smaller format (480p) allowing more people to play videos with a low internet speed.
The 2nd was to have the possibility to have a progress bar when sending the video file.
We then took advantage (having on hand the source video!) to encode the video in a video and audio format allowing to be read by the largest number of browsers on the market (H264 for video and AAC for audio).
And this allowed us to take the opportunity to directly send the source video and 480p video directly into IPFS and thus lighten the customer of this long and unstable task at the time.
In version 0.5, we then added the creation of a video Sprite allowing to visualize thanks to the scroll bar of the video the preview of the moment of the video.
Let’s go!
Here is the block-by-block explanation of the operation of the uploader that allows you to send videos to DTube!
Upload video
b/ Sending the video to the server with progress bar
f/ Token recovery for encoding tracking, IPFS hash recovery, ...
The heart of the Uploader
For fast processing after the end of the upload, everything is treated as asynchronously and in parallel, there is only one token that is returned with prior requests for creation of Sprite (d), encoding (e) performed …
Encoding Daemon
Daemon encoding can handle 2 different tasks, creating Sprites files and 480p encoding of the source file.
How the Sprite is created to preview the video:
- Extraction of 100 images of the source video in 210 * 118 respecting the ratio
- Combining 100 images vertically => allows to see the first images if the image is slowly loading!
Note: if the video is less than 100 seconds, the extraction of images is done with 1 image per second, which is more than enough.
For 480p encoding operation, the source video is simply converted to 854 * 480 for 16: 9 video.
Video conversion to H264 is also performed as well as audio conversion to AAC. Thus this 480p video can be read by many people and by several browsers, unlike the source video.
Once each file is ready (Sprite or 480p file), it is placed in the IPFS queue.
The progress of each treatment is updated every second so that the user can follow it.
IPFS Daemon
The file is added to the decentralized IPFS network.
Once the user recovers all the hashes via repeated calls, the video article can be posted on the Steem network.
Finally on the home page
The video list is extracted from Steem's blockchain, and each video contains an url link from IPFS, hosting the Snap, video, Sprite...
Global scheme !
Extra: the Snap and the DTube logo inlay
Steps:
- Simultaneous scaling and cropping for a final image of 210 * 118
- Adding in IPFS this final image
- Adding IPFS to this image with the DTube logo overlay
Optimizations
If a user had to leave the upload page after sending the video, 480p video encoding and adding to IPFS would be useless and valuable resources would be lost.
That's why if the upload page is closed, the tasks associated with this source video on the video encoding, the creation of the Sprite, the addition in IPFS ... are canceled!
This point saved a lot of resources several weeks ago.
Limitations
The 480p encoding and the creation of the Sprite is for the moment limited to videos less than 20 minutes for cost reasons. Which is still the majority of videos sent on DTube.
The encoding of the source video is currently limited to 480p for cost reasons as well.
Note: These are things that will evolve and change in the future!
The source video
The source is kept as is, without any transformation and limitation in format, size, duration ... but we must pay attention to browsers and Internet speeds of users who may not be able to view the video and not vote at once!
A video with a smaller size, and an encoding in h264 for video and AAC for audio will allow all to see your source video, think about it!
Shortly
The uploader is evolving!
Since the last version v0.6 things have accelerated, the affluence helping and the technical challenges that grow with, and it's rather motivating!
The next article will explain these changes.
Conclusion
This is my very first article that I publish on the internet and also on the blockchain Steem, be indulgent!
Thanks to @heimindanger for his tips for writing articles!
Feel free to leave your comments. I’m french, so thanks to google translate!
Links
DTube: https://d.tube
Uploader: https://github.com/dtube/ipfs-uploader
I know how hard you guys go through with all these. I’ve been using Dtube since I learned about it there still a lot to improve here but i’ll be glad to see dtube be in a different level soon. Thanks for all your hard work to make dtube progress and ensure quality.
Yes, thank you
Why can I not search d.tube or post vids from my iPhone?
Yey!
Dtube is growing on the inside let's feed it with the original content
Thanks for sharing the inside on the improving of the Dtube's uploader. Tried it out today and had a very positive smooth experience. I believe that as much as Dtube needs to constantly upgrade its tech it should also be looking towards raising the quality standards of its platform, potentially looking into creating its own original content.
And I am extremely happy to start contributing content to my Dtube channel. I'm currently working on a documentary about the Crypto Utopia being built in Puerto Rico currently and am wondering would you guys (@dtube, @heimingdange, @superkoala and others) would like to discuss the potential of launching it as a Dtube original project.
This episode is a pilot for the web-documentary about the way Crypto changes our world. Shot in the lean format this project is able to catch the trends faster than the other traditional big media sources documenting the history in making.
Merci,
-K.
It's pretty amazing to see the behind the scenes of the video processing that DTube and the uploader do.
People don't realize how complicated this stuff really is, especially when dealing Peer To Peer. Thanks for all your hard work and I can't wait until we see even more improvements and quality adjustments!
Yes and it's a team work!
Welcome , you did pretty good for your first article and also thank you. If dtube is going to take over its going to need all the brain power it can get, much success.
Thank you
Just a thought. Maybe a Windows and Linux self installer and auto updater for those of us who wish to contribute processing power? Say I want to donate 20% or so of one of my processing cores and 10% of my GPU to process videos.
Additionally an option for redundancy for video files itself, not so much like Torrents but very similar like how Freenet Project handles websites/FreeSites for sake of example?
I'd willingly donate some processing power from my gaming rig and roughly 50 gigs of HDD.
Another approach is in progress to contribute.. a desktop version in windows linux and ios to make what the uploader do but on your computer!
You can also run a ipfs node to contribute for the ipfs network if you want
I'm aware I could contribute to the IPFS network voluntarily, what I'm getting at though is that there are likely people who would contribute in some size, shape or form, but when it comes to installing and setting up IPFS they get deer in headlights syndrome. Even some of the more technically savvy end up that way ;)
So all the same, having an installer that would essentially do the basic installation would be beneficial. Or does one exist now since the last time I checked and I'm just being an idiot? =\
Is there a tutorial on how to do this? Also if you host your own stuff on your own IPFS node can you achieve greater than 480p? Thanks for your awesome work!
Yes there is an tutorial, i found one for german:
Huh https://dist.ipfs.io/#go-ipfs
Wouldn't you know it =\ things have progressed a hell of a lot since the last time I peeked at this stuff.
You really landed with the right foot into the Blockchain. Great improvements DTube is having... as always, everything is a matter of time!
Thank you!
Thank you for your AMAZING work on Dtube it's a HUGE part of this community!
@superkoala Man, you guys are amazing!!! I have seen the progress from the beginning, and it is GREAT. Looking forward to big things, and I am recruiting to DTube every chance I get. :-)
If I may ask I'm very curious as to the availability of infrastructure that DTube has? As you may know, YouTube has 400 hours of video uploaded every minute. Granted, I don't anticipate you guys dealing with that load factor anytime soon, but I'm curious how you guys are set up.
I'm incredibly excited about the future of DTube, however I'm keeping a cautious outlook as I've seen platforms try and compete with YouTube, although I'm not suggesting that's what you guys are trying to do, I've seen platforms have to close shop because they just didn't have the capacity to handle all the traffic.
Thanks for providing this environment for us. My experience here has been phenomenal and I'm very excited about contributing and participating.
Thank you! You are right and we know that very very well 😀 it is our first priority since the start and we work hard on it. If you see now it is ok. We increase hardware when we reach a limit for cost reason and we have a lot of idea to manage more and more trafic. We have idea to increase trafic too but we delay it to keep the platform working 😉
When will dtube support videos on 720 or 1080 ?