feat: add blob.open() for file-like I/O

[andrewsg]

Fixes #29

[andrewsg]

Docstrings are WIP and system tests should be expanded before merging.

[frankyn]

Added some feedback that I want to talk through in implementation. Otherwise this is looking good. I'll review the design document as well.

[frankyn]

Do you know if partial success occurs does transmit_next_chunk handle the retry from the remote offset?

[andrewsg]

Define "partial success"?

[frankyn]

GCS may receive a subset of bytes from client upload, leaving the client and server next byte offset unaligned. A request to get current GCS offset can be made per Step Checking the status of a resumable upload.

If the client doesn't attempt to recover from this state, the inconsistent byte alignment will cause a 400 error which is not recoverable.

[andrewsg]

Got it. With the changes in this recent update, sliding buffer implements seek() so this feature is as supported as it's going to be in python-storage. If we want further support we need to flesh it out in resumable media.

[frankyn]

Let's do that for recovering in a mismatch (address it in resumable media repo). Could you file a tracking issue in that repo?

[andrewsg]

Added docstrings, seek functionality in the sliding buffer, and other features. PTAL. Remaining questions: can we set up a system test to test end-to-end the error-on-chunk-upload-and-retry behavior? And, even though blob.download methods do not populate the blob generation, can we fetch that generation information from download metadata somehow in order to implement generation lock? (alternative: force blob.reload() before d/l if generation is not populated - race condition possible).

[frankyn]

Remaining questions:

1. Can we set up a system test to test end-to-end the error-on-chunk-upload-and-retry behavior?

• Not without GCS emulator. You'd be better off mocking out the API request at this time.

2. Even though blob.download methods do not populate the blob generation, can we fetch that generation information from download metadata somehow in order to implement generation lock? (alternative: force blob.reload() before d/l if generation is not populated - race condition possible).

• You can get generation information from response headers. A similar issue came up in Node.js (for checksums) and here's example of responses specifically the header is named X-goog-generation:.

[frankyn]

Over LGTM, I have a few nits.

Also retrying on mismatched offset may be better handled / addressed in resumable media than in this wrapper so we could table it for now and open tracking issues in resumable media repo. WDYT?

[frankyn]

LGTM! 🚢