Let's Fix File Uploading
Published:tl;dr: We are creating an open source project to provide a turnkey file uploading solution called tus. Today we are happy to show you our first resumable file uploading demo.
It’s 2013, and adding reliable file uploading to your app is still too damn hard. And if content is king, this poses a serious problem to those interested in acquiring it. In this post I’ll describe the current uploading landscape and how we are planning to fix it.
Resumable Uploads
Have you ever tried uploading a big file over an unreliable internet connection? Depending on the site you were using, the experience can be extremely frustrating. Except for a few bigger sites, network errors will force you to restart your upload from the beginning. Even worse, some sites will be unable to detect this problem, leaving it up to you to figure out why the progress bar is stuck.
The solution of course is to offer resumable file uploads, however, this is easier said than done. While there is no shortage of bits and pieces that can help you with building your own solution, there is no simple solution that you can get up and running in a few minutes.
The first thing you’ll need to figure out is a server side API. Google has
published several http protocols for resumable uploading (e.g.
YouTube,
Google Drive), but they
all rely on a non-standard http code 308 Resume Incomplete which is on
collision course with upcoming
standards. Amazon
S3 offers multipart
uploads
(not to be confused with multipart/form-data), but they are hard to use and
the minimum chunk size is 5MB which is far too large to be useful under bad
networking conditions.
And even if you figure out the server side, you’ll still need to take care of the client side. For HTML5, your best bet is to start out with something like jQuery File Upload which has some basic support for resumable uploads, but you’ll still have a lot of tweaking ahead of you in order to make it work with your server side API. If you are also running native iOS / Android apps, you will be entirely on your own to come up with a solution.
Fuck this. It’s 2013, and you shouldn’t have to waste several days for what will only be a rudimentary solution, hard to scale, and a pain to maintain. This is where we come in with tus. Tus is an open source project that aims to provide you with a turnkey solution to all your file uploading needs. We are designing a protocol, building a server, and are working on clients for HTML5, iOS and Android. Once these are solid, we will also provide a hosted service, making it even easier for people to get going, as well as allowing us to recoup our investment in this project.
Today is a great day to have a first look, as we have just completed our first resumable file uploading demo for HTML5. The demo is pointed at a tusd instance running on EC2, and uses jQuery File Upload under the hood. What’s exciting about it, is that it not only handles network errors, but also browser crashes and user errors like closing the page. No matter what happens, a previous upload will always continue where it left off.
So check it out, and give us your feedback. We are also very interested to hear what other issues you’d like to see solved. If you are not sure about the possibilities, just read on.
A world of possibilities
Resumable file uploads are just the tip of the iceberg, and there are many more problems ripe for a solution. However, we need your help in prioritizing them, so please let us know which of the things below you would find valuable.
- File upload acceleration: There are many situations where file uploads fail to achieve optimal performance due to latency and TCP limitations. We hope to tackle this from multiple angles: Hosting a reverse CDN to improve latency regardless of user location, utilizing multiple TCP connections to increase the effective tcp window size, and maybe in the far future implement a specialized UDP protocol.
- Huge files: With HD cameras on the rise, uploads become bigger and bigger. A good solution should be able to handle files > 2GB with ease.
- Streaming: A lot of files can be processed as a stream, so tus should make it easy to let you download and process an active upload.
- Service Integrations: It should be easy to let your users select files from their Dropbox, Facebook, Google Drive or other cloud services.
- Processing pipelines: Most files are media files and require additional processing before being consumed by other users. Tus should easily integrate with existing processing solutions such as Transloadit, Zencoder and others.
- Scalability: It should be easy to run a cluster of tus instances, allowing you to add or remove capacity as needed.
- Security: HTTPS is a no-brainer, but we are hoping to raise the bar by supporting file encryption on the fly for applications that can benefit from it.
- Protocols: HTTP is not the only protocol out there. We are interested in exploring uploads via FTP, E-Mail and other commonly used protocols.
If any of the above sounds exciting to you, please let us know. For now our priority is to release a solid server along with great client libraries for HTML5, iOS and Android, but at the end of the day it will be your feedback that will drive our roadmap.
-- Felix GeisendörferSubscribe to this blog via RSS or E-Mail or get small updates from me via Twitter.