4K 60FPS video is coming but where are we going to store it? The unexplored potential of optical discs
It was around 1994-1995 when the multimedia boom started in PCs... Up to the early 90's, PCs were kind of "mute". The typical PC didn't even have audio, beyond its funny-sounding PC speaker.
Video was limited to text and 2D representation - at most some complex images, but not the compressed type, because high analysis JPEGs could take seconds to render to the screen in a 286 or 386 CPU. Things were starting to get better in the multimedia aspect as PCs were increasingly able to support audio, images and videos around '93-94.
I remember my excitement, back in early '94 when I bought my CD-ROM drive that could read over 650MB of data from a single disc... The capacity was absolutely mindblowing...
The CD-ROM revolution
The CD-ROM was opening gates to a different kind of experience: Suddenly the PC became alive with a lot more music, images, animations and video. A lot of video. Even the games that we were playing at the time, were becoming ..."interactive movies".
Software developers could put so much video in the CD that adventure games could have multiple scenes branching out, depending the user choices while playing. Computer magazines were also distributing CDs, accompanying their printed edition, packed with shareware and freeware stuff downloaded from the Internet. With the slow modems we had back then, one would need days to download all of these (assuming they had somewhere to store all that stuff).
The data CD was, without exaggeration, the biggest thing after the World Wide Web. In fact, unlike the WWW which was unpopulated at the time, the multimedia CDs were packed with text, pictures, animations, videos, games and programs. I can say with certainty that the CDs were way more exciting to me the the near-empty WWW.
For comparison purposes, in 1994, my two year old hard drive was just 80mb and my new disk to upgrade that one was at 212MB. So the ratio of CD vs HDD, at that point, was pretty good for the CD. And compared to the older portable storage (1.44MB floppies) it seemed like a tremendous increase.
The main problem though was that it was Read Only (CD-R.O.Memory). At that point CD-R(ecordable) technology was very expensive, so only few people had CD-R drives.
Fortunately, a lot of computer shops were offering CD-writing as a service for something like 15-20$: You could get your hard disk to the shop, tell them what directory you wanted to write into a CD and then wait a couple hours to get your data written on that shiny, gold-looking CD, that looked so awesome :)
Fast forward to 1997-1998 and CD-R technology was still expensive but much more affordable (from 3000$ down to 400$ - for CD-R drive and SCSI controller), while the high end hard disks (IDE) were hitting capacities around the 4-8GB mark. But, by then, the DVD was out, again matching the top-end HDDs.
Granted, DVD-writing capacity came a few years later, but at that time, the optical disc was the undisputed king of carrying video content. We know what happened next: For the last 20 years, the DVD with sizes of around 4.5 - 9GB has been one of the most widespread mediums for storing standard definition videos and movies, commercial program and games, even user data as a storage medium. By the mid 2000's, buying 50 blank discs (4.7GB), would get you to near 1TB of storage but with just a fraction of the cost required for the equivalent 1TB storage from a hard drive. At that point something weird happened: The evolution of the optical disc practically stopped in its tracks. While there were constant news about new optical technologies that were going to take us to the moon (in terms of storage capacity), we "somehow" got stuck with the DVD for the desktop PC. Research and experimental implementations continued even after the Blu-Ray and up to this day: 2004: "Pioneer has technology to develop 500GB Blu-Ray discs" 2006: "How to burn a three terabyte CD" 2007: 1 to 5 TB of data in a CD/DVD-size disc 2010: TDK playing around with 1TB discs 2016: Tiny Quartz-based optical disc stores 360TB with practically infinite lifespan ...yet despite all these technological advances, we have seen none of it in the affordable consumer market. We still use CDs, DVDs and to a lesser extent Blu-Rays, which are all obsolete technologies (16-30+ year old techs). There was a protracted period when HD-DVD and Blu-Ray were battling it out for the "next-generation" DVD, around 2003-2008, which was ultimately won by Blu-Ray. Ever since, Blu-Ray adoption was very slow. Computers were virtually cut-off from the DVD's successor (due to costs) while Sony focused more on the game platform and movie market. By the time Blu-Ray reader/writers for PCs and laptops became affordable (along with affordable blank discs), they were next to useless - despite the need to replace the obsolete DVD-R. Right now Blu-Ray size is small, cost/GB not that good, and their read/time writes suck. Hard disks, USB flash sticks and data cards (a 1TB SD card was just released by SanDisk) have an advantage in terms of throughput, size and rewritable status. For the most part of the last 10 years, we are living in a distorted market in which PC users have been buying and stacking hard disks (!) to store their data, due to impractical alternatives (huh?) rather than actual technological constraints. In the age of 20+ megapixel photos and 1080p30 or p60 videos, that even our phone can record, the rate at which we would fill 2TB disks is unsettling. On the other hand, what can we do? Delete our photos and videos? What good is a video if we can't store it, or if its storage costs too much? The problem is only going to get even bigger with 4k60 video. Do you see any storage solutions in sight? We can continue to use hard disks, although since their $/GB ratio is not falling dramatically, and our video storage requirements are going up considerably, this means that the price that we'll be paying to store videos on our disks will be higher. And what about 8k video or even higher frame rates? All these would be very different if we could use, say, 2TB to 4TB optical discs (in line with the size of our modern hard disks - which is something that optical technology was always good at matching or exceeding). It's a real shame that optical discs, which were always a tremendously cost effective storage technology, have been disallowed to scale by the market monopolies. And even when they are allowed to grow, their growth is "locked" to the movie market instead of the desktop. While a Blu Ray upgrade to >1TB would allow the movie studios to market higher quality movies to the consumers, this in no way solves our problem of where to store the data we create, in an affordable manner. I think what we'll have, is a bit more of the 1080p case, with even bigger quality losses: We'll get a very toned-down 4K experience that is a shadow of what the technology can actually deliver. I'd go as far as to claim that the 4K that we'll get will be a marketing trick. And all this will be due to our lack of affordable storage capacity. Truth is that a video can be 720p, 1080p or 4k, but this doesn't automatically translate to very high quality. Bitrate, on the other hand, is a great indicator of quality: A 4k stream with 1 mbps is worse compared to a 1080p with more mbits per second. And both of them would be way more detailed at 100 or 200 mbps instead of being in the 1-2-mbps range. What ultimately matters, for the quality of a video stream, is that there is enough information on the stream to adequately reproduce high levels of detail. As we increase the bitrate (mbits per second), the quality is increased while compression artifacts and noise are reduced. (As a note, and since I got a bit technical, mbits are not megabytes, so 8 Mbits per second = 1 MByte per second.) Sites like YouTube have every reason to compress their streams and reduce quality because Internet connections are still slow (average connection speed is currently at 6.3 Mbps). With most connections being bandwidth-limited, a tradeoff in quality is considered acceptable because the alternative scenario is a choppy video that stops every 5 seconds to download the next video segment. For most viewers this is unacceptable and therefore quality concessions have to be made. I'm now checking a 1080p60 video on youtube, and I'm actually being able to stream it at around 5Mbps. But when I use full screen, it's clear that a lot of the detail that a 1080 video should have, is not even there. Whether it is due to the camera recorder or youtube secondary compression, the result is that the video doesn't have the expected quality.The inexplicable "halt" in progression
The format wars
So what's going to happen next?
A crippled 4K?
If this is happening with 1080/60 video you can expect this phenomenon to increase in 4K video. Most cameras (whether it's a phone camera, digital cameras or video recorders) will not capture high bitrates that exploit the 4K-definition to the higher-end of its potential because then the videos would be too large and users would have to delete them all the time. Our local storage devices (hard disks) would be getting filled like crazy and there would be no way to store all this material without seriously compromising its quality. And therefore the compromise in quality will become a "necessity".
Again, it is important to understand that all this will happen because there is a market failure in the optical drive industry. If people could shoot 4K / 60fps with a streaming rate close to 250 Mbps (31.25MB/sec), it would require around 112GB per hour. With hard disks costing 100$ per 3TB (33$ / TB), it would mean that just an hour of video (112GB) would cost around 3$ to store it - which is absurd. But if we could store 30-40 hours in an optical disc that cost us less than a dollar, that would be a different issue altogether.
So now that we have the camera technology, the processors, the graphic cards, the dirt-cheap 4K TVs, etc etc, we will not be able to enjoy it because a few companies have decided to choke the storage market.
We could, potentially, bypass this problem for streaming applications, but only if our Internet connections were in the 200MBps range, streaming very detailed 4K video through the net (or our Internet provider offering TV services over broadband). Yet we have to be realistic that this will not happen anytime soon for the average Internet user.
Conclusion
Optical storage has always been a technology that was not only "promising", but actually delivering -consistently- the best data storage that our money could buy. This has changed in the last 10 years. Quick progress was replaced by stagnation in developing new optical disc formats - despite the fact that technological innovations are actively pursued (even if they aren't marketed).
In the last 10+ years there was no serious progress in upgrading capacity, cost efficiency (cost/GB stored), or speed (read/write throughput). Yet, high analysis photography and video, are two applications that require an ever-increasing amount of storage. Due to the lack of affordable storage, mainly in the form of writable optical discs for the desktop, we will be getting a very toned-down experience of what 4K/60fps videos are able to actually deliver, both in terms of movies, but also in terms of what we are able to record and reproduce - at least until optical discs are upgraded significantly.
optical disks? come on. what speaks against those tiny sd card media in this day and age.
SDs and micro-SDs are nice and practical, but they are too expensive per GB for permanent storage. They are ok to use in the camera for writing the video though (and then unloading it on the PC).
One big problem with optical media is that they are vulnerable to scratches and the greater the data density, the greater the data damage.
In theory this can be fixed if a format performs its write operations deeper. It can also be bypassed through external protection, like a plastic cover that only opens at a certain spot during reading (3'5 floppy drives worked like that, as well as some optical backup drives).
Personally I've worked with thousands of CDs and DVDs and at most I scratched something like 5. But there are people that are mishandling them and are all scratches. I'm like "dude, did you use the steel brush on this?" ahahaha... These people, "coincidentally", also have a problem with hard disks "hitting" and developing read errors due to the G-shocks.
I used to have problems with homemade dvds. Probably because of the low quality of the media.
But plastic enclosures sounds like a good idea.
Storing ? What for ? ;-)
Streaming is the way to go.
It's not that easy. If you have content that is private, you don't want it on youtube for streaming. Plus it will get heavily downgraded in quality. I mean the concept of ultra high definition is to have the extra detail. Once you upload it to a streaming platform, you lose that extra detail.
Yeah, we'll be putting the uhd streams on cloud-based hard disks and on teeny little pieces of flash memory. Spinning disks are probably destined to only live on in datacenters, where they still get a lot of use as the low-speed, low-cost, low reliability alternative to flash memory, and can be made reliable by joining them together into insanely large (think: 1 million 10TB disks) arrays.
I think the problem will be "solved" for us, by simply doing a massive reduction in quality of the videos. The phones, the video cameras, etc, will simply save in a very "compressed" format, for "practical reasons", leaving most of the details out. Thus instead of having storage issues we'll have quality issues instead. But virtually nobody will be noticing them because 4K will be better than 1080 and seem like an "upgrade" that they are happy to pay.
Thanks for this remarkable article.
The University of Southampton project sounds phenomenal... I was digging for but haven't yet found information on write/read rates and so forth, but the raw capacity v storage size sounds phenomenal... I've been saying for 25 years that I want 100TB in my shirt pocket; this ought to do it!
Storage capacities like this would make 360 degree spherical life-logging practical. Have you ever watched a scene unfold and after the fact said "I wish I had a picture of that." ? ;)
There was an article the other day on the dilemma of living a moment and capturing a moment on the phone or camera. This is definitely something that can be solved technologically, meaning that if we had adequate capacity, there would be no problem having a small sensor on our eyeglasses or head, that simply takes very high quality photos, at something like 100-200fps. In this way you both live the moment, and be assured that everything is recorded in real time and you can replay it later.
This will need a few years to become feasible, but eventually it will be. It's just a matter of time.
In theory, high-capacity optical drives were also supposed to be a matter of time to manifest, but they didn't (for the mainstream). For data archival purposes, optical discs are alive and well, although only for professional environments because the premiums are extreme...
http://www.sony.gr/pro/product/archiving-storage-oda-stand-alone-drive/ods-d77u/features/#features
"The ODS-D77U has very high transfer speeds. Read is 1.1Gbps. Write-once is 440 Mbps and Rewritable is 160 Mbps in case of verify on. Performance is varied based on type of cartridges."
Yet the device costs something like 6000$+ and the 1.5TB discs around 150$.
Note that this is ~5yr old tech btw. They can do a lot of stuff, they just don't want to.
Wow, that's pretty pricey! :0 :(
I wonder if the University of Southampton tech will ever be productized... It sounds just like science fiction.
There are already some Kickstarter projects that offer "life cams" but I don't know what the quality and the capacity are. One way to deal with it is to make the storage selective - just keep a high-speed, high quality buffer loop going and have a "capture" button that lets you archive only the last minute or two once you notice something that you want to save.