fileUploader.js: Resumeable HTML5 Uploads

fileUploader.js

fileUploader.js is a javascript library designed to interact with nginx's upload module, allowing for resumeable uploads.The js library works as follows:

var uploader = fileUploader(file, segmentSize, sessionId)

where file is a File object from the HTML5 File API, segmentSize is the size in bytes of each segment of an upload, and sessionId is the sessionId used to represent this file upload (must be unique!). The methods of the uploader object return Bluebird Promises/A+; which may be interacted with using this API. It offers the following methods:

  • uploader.fetchStatus(): returns a promise for a status object, which has the following structure:

      {
        completed : [true|false],
        start : <The first byte uploaded. Should always be 0>
        end : <The last byte uploaded>
        total : <The total size of the file in bytes>
      }
    

    completed should be true if and only if end === total - 1.

  • uploader.uploadSegments(status, onSegmentComplete): given a status object, will upload the remaining segments of the file. onSegmentComplete(newStatus) will be called after each segment is uploaded, with a status argument reflecting the new state. It returns a promise with the completed status.

Thus, to upload a file, one may do:

  uploader.fetchStatus()
      .then(function(status) {
          uploader.uploadSegments(status, function(newStatus) {});
      })
      .then( function(status) { console.log("upload complete"); } );

Design

Resumable uploads in the upload module work in the following way:

  1. Uploaded segments of a single file should share a session ID to identify them as part of the same file.

  2. When a segment of a file is uploaded, it will return a status in the body, e.g. 0-5,9-15/24, indicating that the file is 24 bytes long, and bytes 0-5 and 9-15 have been uploaded. Partial uploads will get a 201 response.

  3. Segments may be reuploaded, as long as it is not in parallel, and the segments have the same data for that segment.

The fileUploader js library will attempt a small upload (the first 2 bytes of a file; it turns out the upload module has a bug wherein it will not handle repeated uploads of the single byte of a file) to get the status of an upload. From that starting status, the uploader will incrementally upload the file in 1MB segments.