Cannot download multiple OneDrive files concurrently since v.1.13.0-beta

[tipa]

Expected behavior

Until and including v1.12.0 I was able to download many files from OneDrive concurrently (see code snippet below) without any issues (and fast!).

Actual behavior

Since v1.13.0-beta (and incl. the now available v1.14.0) the download of multiple OneDrive files is slow from the beginning and is getting even slower over time. The app can only download up to ~15 files before the process eventually stops completely and the following exception is raised (no custom timeout set):

Code: generalExceptionMessage: An error occurred sending the request. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Exception: The text associated with this error code could not be found.The operation timed out at System.Net.Http.HttpHandlerToFilter.SendAsync(HttpRequestMessage request, CancellationToken cancel) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.Graph.AuthenticationHandler.SendAsync(HttpRequestMessage httpRequestMessage, CancellationToken cancellationToken) at Microsoft.Graph.RetryHandler.SendAsync(HttpRequestMessage httpRequest, CancellationToken cancellationToken) at Microsoft.Graph.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Microsoft.Graph.HttpProvider.SendRequestAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken) --- End of inner exception stack trace --- at Microsoft.Graph.HttpProvider.SendRequestAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken) at Microsoft.Graph.HttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken) at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption) at Microsoft.Graph.BaseRequest.SendStreamRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption) at Diarium.OneDriveHelper.DownloadFile(GraphServiceClient graphClient, String fileName)

Steps to reproduce the behavior

Microsoft.Graph v1.14.0
UWP app (but also happens in Xamarin)

var client = new GraphServiceClient("https://graph.microsoft.com/beta", new DelegateAuthenticationProvider(reqMsg =>
{
    reqMsg.Headers.Authorization = new AuthenticationHeaderValue("bearer", authResult.AccessToken);
    return Task.CompletedTask;
}));

SemaphoreSlim semaphore = new SemaphoreSlim(10);
if (filesToDownload.Count > 0)
{
    var downloadTasks = new List<Task>();
    var downloaded = 0; var error = false;
    foreach (var file in filesToDownload)
    {
        await semaphore.WaitAsync();
        if (error) { semaphore.Release(); break; }
        downloadTasks.Add(Task.Run(async () =>
        {
            try { file.Data = await DownloadFile(graphClient, file.Name); }
            catch { error = true; throw; }
            finally { semaphore.Release(); }
        }));
    }
    await Task.WhenAll(downloadTasks);
}


static async Task<byte[]> DownloadFile(GraphServiceClient graphClient, string fileName)
{
    using (var response = await graphClient.Drive.Special.AppRoot.ItemWithPath(fileName).Content.Request().GetAsync())
    {
        return Decrypt(response);
    }
}

[MIchaelMainer]

Thank you for reporting this issue and providing repro steps.. We will investigate this.

[peombwa]

@tipa, We have released a fix for this issue as part of release 1.14.1. Give it a try and let us know if it addresses your scenario.

[tipa]

I am struggling including the project into my solution. Would it be possible to release it on Nuget.org as a preview package or make the NuGet file available to me as a direct download?

[peombwa]

@tipa Microsoft.Graph.Core 1.14.1 package is available on nuget.org. Just update the core package to 1.14.1.

[tipa]

I installed both packages via Nuget.org now:

Unfortunately, the issue was not resolved and the same error occurs. It looks like the library is receiving the download location (public.by.files.1drv.com....), but then doesnt send the GET request to that URL:

[darrelmiller]

My reading of that trace is that request 53-59 + 62-63 are the initial requests that redirect with a 302 and then 66,67,70,71,etc are the corresponding GETs to the redirect location with response payload and .200.

Is there a reason you believe this is not the case?

[tipa]

No, this is absolutely correct. As I wrote in the first post, the first ~10-15 files are downloaded correctly. But you then see the requests in line 86 ff: They redirect to a different URL, but the library never sends a GET requests to them

[darrelmiller]

@tipa My apologies, I missed that detail in your original post. I will continue to try to repro your issue. Could you try performing the multiple downloads by simply accumulating the tasks into an array and doing WaitAll and see if you still experience the problem. I'm assuming the semaphone/Task.Run/try/catch is a way of aborting all the tasks if one fails. If so you might be interested in this approach https://gist.github.com/svick/9992598 which uses a cancellation token to cancel all other tasks if one fails.

[darrelmiller]

@tipa When just loading tasks into an array and doing WaitAll I am able to download 25 files without any issue or slowdown.

[darrelmiller]

@tipa Do you have AppInsights enabled on your app? In the stacktrace I see a reference to DiagnosticsHandler and I suspect that might be coming from AppInsights.

[tipa]

Could you try performing the multiple downloads by simply accumulating the tasks into an array and doing WaitAll and see if you still experience the problem.

Tried that (I simply commented out the await semaphore.WaitAsync()) but then not even a single file (out of 748 in my test case) could be downloaded.
It might be worth noting that:

• I am using ServicePointManager.DefaultConnectionLimit = 20; to increase the amount of max. concurrent HTTP calls. However I played around with multiple values and also commented out the line, but the problem persisted
• All the files I am downloading are around 40-100KB in size
• When I initialize the semaphore with 10, sometimes the app does download more than 25 files, up to maybe 40 - but then ultimately stops.
• When I initialize the semaphore with 5, I was able to download way more files, but at a very slow pace. I stopped after 6 minutes with 160 files downloaded
• When I initialize the semaphore with 1 (so no parallel downloads) it works better and I can download more files. However, even then the downloads slow down over time. After after 50 files it takes around 1.5 seconds for each file
• In all the HTTP responses from public.by.files.1drv.com I see this header:
  WWW-Authenticate: Bearer realm="OneDriveAPI", error="invalid_token", error_description="Invalid auth token"

I'm assuming the semaphone/Task.Run/try/catch is a way of aborting all the tasks if one fails.

The main reason for using the sema is to only download 10 files at a time. Sometimes I need to download multiple hundreds or even thousands of files which would lead to timeout errors very quickly if I requested them all at once. The try-catch is for the case a task fails and to not make any additional requests. I don't think the semaphore is the cause of the issue here because it worked fine with v1.1.12 and with other cloud storage libraries I implemented (Google Drive, Dropbox, ...)

Do you have AppInsights enabled on your app

I am using AppCenter in my app