Testing resumable uploads

The core idea behind resumable upload is straightforward if you are uploading a big file, then you are going to encounter users in the network conditions where they cannot upload the file in a single network session. The client-side code, to avoid restarting the file upload from the beginning, must figure out what portion of the file was uploaded and “resume” the upload of the rest.

How to do resumable upload

Before starting the upload, send a unique ID generated from the file contents to the server like MD-5 or SHA-256. The server decides and declares what the format of that unique ID is. Next, the server responds with an offset which indicates how many bytes server already has. The client uploads rest of the bytes with a Content-Range header.

How to test resumable upload

The right way to verify that this code works is to break the upload intentionally and randomly in the middle and check that the next upload session does not start from zero. Note that, it might not start from the exact byte offset where it was disconnected since the client network stack can have its buffer size to fill and it might discard the buffered bytes in case of an exception.  Therefore, the ratio of the number of bytes read to the file size should be close to one but might not be one.

The right way to verify that this code works is to break the upload intentionally and randomly in the middle and check that the next upload session does not start from zero.

Sample skeleton codes

// SHA-256 or MD-5 whatever the server decides
String uniqueId = generateUniqueId(File file);
// Number of already uploaded bytes (could be zero)
int alreadyUploaded = getAlreadyUploadedByteCount(uniqueId);
// Upload rest of the bytes
upload(file, uniqueId, alreadyUploaded);

boolean upload(File file, String uniqueId, int alreadyUploaded)
 throws IOException {
  InputStream in = getInputStream(file);
  uploadViaHttp(file, uniqueId, alreadyUploaded);
}

int readCount;

boolean upload(File file, String uniqueId, int alreadyUploaded) 
throws IOException {
  InputStream in = getInputStream(file);
  readCount = 0;  // Testing only
  uploadViaHttp(in, uniqueId, alreadyUploaded);
  double ratio = ((double)readCount)/file.size();  // Testing only
}


// Standard method
InputStream getInputStream(File file) throws IOException {
    return new FileInputStream(file);
}

// Enhanced method for testing
InputStream getInputStream(File file) throws IOException {
    return new FileInputStream(file) {
        int read(byte[] b, int off, int len) throws IOException {
            if (random.nextBoolean()) {
                throw new IOException("Intentionally broke the stream");
            }
            int tmp = super.read(b, 0, b.length());
            readCount += tmp;
            return tmp;
        }

        int read(byte[] b) throws IOException {
            return read(b, 0, b.length);
        }

        int read() throws IOException {
            readCount++;
            return super.read();
        }
    }
}