Low default chunk size slows GCS downloads

[davidn]

Steps to reproduce:

1. Create multi-gb file gs://your_bucket/obj
2. Test gsutil speed with "gsutil cp gs://your_bucket/obj /local/copy"
3. download via gcloud.storage.Blob.download_to_filename (see sample script below)

Expected result: similar timing

Actual result on my machine: gsutil downloads 4.6GB in 1min14sec, gcloud.storage takes 20min.

If I add blob.chunk_size=1<<30 prior to download_to_filename, gcloud.storage takes the same time as gsutil to the second.

Sample script:

#!/usr/bin/env python
import httplib2
httplib2.debuglevel=4
import gcloud.storage

bucket_id = 'BUCKET'
blob_id = 'OBJECT'
url = 'gs://%s/%s' % (bucket_id, blob_id)
project = 'dnewgas-test-1'
client = gcloud.storage.client.Client(project)
bucket = client.get_bucket(bucket_id)
blob = bucket.get_blob(blob_id)
//blob.chunk_size=1<<30
blob.download_to_filename('/tmp/down')

[dhermes]

@thobrla Is 2**30 a better chunk size than 2**20? Should we base it on the file size first?

[thobrla]

What download resume methods does gcloud-python support - in other words, how do you recover from transient service or network issues? If support is based on downloading entire chunks successfully, then chunk size is implicitly a tradeoff between performance vs. reliability.

But ideally you wouldn't have a chunk size at all - you would request the entire object in a single request, and in the presence of a transient network or service error, pick up where you left off based on how many bytes you had successfully downloaded.

[davidn]

I don't think the JSON API has a good resume mechanism. The XML API allows resuming with range headers. gsutil uses JSON API with chunked downloading like this and 100MB chunks I believe.

[thobrla]

The JSON API supports range headers as well.

gsutil does not use chunking for single object downloads via the JSON API. It does slice objects into multiple ranges so that it can download them in parallel, but whether that slicing occurs or not, a single range request for the entire range of each slice is set.

You can verify the simple (non-sliced) case by running:
gsutil -D -o 'GSUtil:sliced_object_download_threshold=0' cp gs://your-bucket/large-object local-file