Firestore Realtime listener drops a connection and does not reconnect

[saminiemi]

• Operating System version: Nodejs (Alpine based) docker container, node:8.7.0-alpine
• Firebase SDK version: 5.4.2
• Library version: @google-cloud/firestore 0.8.2
• Firebase Product: Firestore

We have a simple Firestore Realtime listener running on Docker container at Google Container Engine. Few times, listener for some reason have lost a connection and does not reconnect. Updates (sets, updates and deletes) does work but we don't get notifications of updates, until we restart a container.

Error message we have got couple of times during last week:

Error: Error: Endpoint read failed
    at sendError (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:254:15)
    at maybeReopenStream (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:268:9)
    at BunWrapper.currentStream.on.err (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:301:13)
    at emitOne (events.js:120:20)
    at BunWrapper.emit (events.js:210:7)
    at StreamProxy.<anonymous> (/usr/src/app/node_modules/bun/lib/bun.js:31:21)
    at emitOne (events.js:120:20)
    at StreamProxy.emit (events.js:210:7)
    at ClientDuplexStream.<anonymous> (/usr/src/app/node_modules/google-gax/lib/streaming.js:130:17)
    at emitOne (events.js:115:13)

Once (first time) we have got this:

Error: Error: Transport closed
    at sendError (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:254:15)
    at maybeReopenStream (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:268:9)
    at BunWrapper.currentStream.on.err (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:301:13)
    at emitOne (events.js:120:20)
    at BunWrapper.emit (events.js:210:7)
    at StreamProxy.<anonymous> (/usr/src/app/node_modules/bun/lib/bun.js:31:21)
    at emitOne (events.js:120:20)
    at StreamProxy.emit (events.js:210:7)
    at ClientDuplexStream.<anonymous> (/usr/src/app/node_modules/google-gax/lib/streaming.js:130:17)
    at emitOne (events.js:115:13)

Our code is basically like this:

 var admin = require("firebase-admin");
  var serviceAccount = require("./credentials.json");
  
  admin.initializeApp({
    credential: admin.credential.cert(serviceAccount),
    databaseURL: "https://xxxxxxxx.firebaseio.com"
  });
  var db = admin.firestore();
  
  db.collection("demodata").onSnapshot(querySnapshot => {
    querySnapshot.forEach(doc => {
      //.. doing stuff with data
      console.log(doc.data());
    });
  });

Is listener planned to survive on these situations, or should be build some kind retry system by ourselves?

Thanks!

[google-oss-bot]

Hey there! I couldn't figure out what this issue is about, so I've labeled it for a human to triage. Hang tight.

[google-oss-bot]

Hmmm this issue does not seem to follow the issue template. Make sure you provide all the required information.

[hiranya911]

This should probably be reported at https://github.com/googleapis/nodejs-firestore

@schmidt-sebastian can you take a look?

[schmidt-sebastian]

@saminiemi We have retry logic in place to handle this specific situation. If you can reproduce this failure regularly, do you mind turning on debug logging and sending us the logs from just before this occurred?

You can do this as follows:

admin.firestore.setLogFunction(console.log);

[saminiemi]

Thanks @schmidt-sebastian . We did like suggested and here is the result:

Firestore (0.8.2) 2017-10-27T15:32:09.055Z [Watch.onSnapshot]: Processing target change
Firestore (0.8.2) 2017-10-27T15:33:07.944Z [Firestore._initializeStream]: Received stream error: { Error: Deadline Exceeded
    at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/node/src/client.js:255:19)
    at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/node/src/client.js:233:8)
    at /usr/src/app/node_modules/grpc/src/node/src/client.js:757:12 code: 4, metadata: Metadata { _internal_repr: {} } }
Firestore (0.8.2) 2017-10-27T15:33:07.944Z [Watch.onSnapshot]: Stream ended, re-opening
Firestore (0.8.2) 2017-10-27T15:33:07.944Z [Watch.onSnapshot]: Opening new stream
Firestore (0.8.2) 2017-10-27T15:33:07.945Z [Firestore.readWriteStream]: Opening stream
Firestore (0.8.2) 2017-10-27T15:33:07.946Z [Firestore._initializeStream]: Sending request: {"database":"projects/xxxxxxxxxxxxx/databases/(default)","addTarget":{"query":{"parent":"projects/xxxxxxxxxxxxx/databases/(default)","structuredQuery":{"from":[{"collectionId":"sites"}]}},"targetId":240,"resumeToken":{"type":"Buffer","data":[10,9,8,158,159,178,239,143,145,215,2]}}}
Firestore (0.8.2) 2017-10-27T15:33:07.947Z [Firestore.readWriteStream]: Streaming request: {"database":"projects/xxxxxxxxxxxxx/databases/(default)","addTarget":{"query":{"parent":"projects/xxxxxxxxxxxxx/databases/(default)","structuredQuery":{"from":[{"collectionId":"sites"}]}},"targetId":240,"resumeToken":{"type":"Buffer","data":[10,9,8,158,159,178,239,143,145,215,2]}}}
Firestore (0.8.2) 2017-10-27T15:33:07.948Z [Firestore._initializeStream]: Marking stream as healthy
Firestore (0.8.2) 2017-10-27T15:33:07.948Z [Firestore._initializeStream]: Releasing stream
Firestore (0.8.2) 2017-10-27T15:33:07.948Z [Watch.onSnapshot]: Opened new stream
Firestore (0.8.2) 2017-10-27T15:33:08.638Z [Firestore._initializeStream]: Received stream error: { Error: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
    at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/node/src/client.js:255:19)
    at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/node/src/client.js:233:8)
    at /usr/src/app/node_modules/grpc/src/node/src/client.js:757:12
  code: 16,
  metadata: Metadata { _internal_repr: { 'www-authenticate': [Array] } } }
Firestore (0.8.2) 2017-10-27T15:33:08.638Z [Watch.onSnapshot]: Stream ended, sending error:  { Error: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
    at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/node/src/client.js:255:19)
    at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/node/src/client.js:233:8)
    at /usr/src/app/node_modules/grpc/src/node/src/client.js:757:12
  code: 16,
  metadata: Metadata { _internal_repr: { 'www-authenticate': [Array] } } }
Firestore (0.8.2) 2017-10-27T15:33:08.639Z [Watch.onSnapshot]: Invoking onError with: Error: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
Error: Error: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
    at sendError (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:254:15)
    at maybeReopenStream (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:268:9)
    at BunWrapper.currentStream.on.err (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:301:13)
    at emitOne (events.js:120:20)
    at BunWrapper.emit (events.js:210:7)
    at StreamProxy.<anonymous> (/usr/src/app/node_modules/bun/lib/bun.js:31:21)
    at emitOne (events.js:120:20)
    at StreamProxy.emit (events.js:210:7)
    at ClientDuplexStream.<anonymous> (/usr/src/app/node_modules/google-gax/lib/streaming.js:130:17)
    at emitOne (events.js:115:13)
Firestore (0.8.2) 2017-10-27T15:33:08.639Z [Watch.onSnapshot]: Processing stream end

Seems to be something to do with authentication. Hope this helps and if there is any information you would like me to provide, please let me know.

[saminiemi]

I can also confirm that most of the time listener survived just fine:

Firestore (0.8.2) 2017-10-28T08:49:12.884Z [Firestore._initializeStream]: Received stream error: { Error: Deadline Exceeded
    at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/node/src/client.js:255:19)
    at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/node/src/client.js:233:8)
    at /usr/src/app/node_modules/grpc/src/node/src/client.js:757:12 code: 4, metadata: Metadata { _internal_repr: {} } }
Firestore (0.8.2) 2017-10-28T08:49:12.884Z [Watch.onSnapshot]: Stream ended, re-opening
Firestore (0.8.2) 2017-10-28T08:49:12.884Z [Watch.onSnapshot]: Opening new stream
Firestore (0.8.2) 2017-10-28T08:49:12.885Z [Firestore.readWriteStream]: Opening stream
Firestore (0.8.2) 2017-10-28T08:49:12.885Z [Firestore._initializeStream]: Sending request: {"database":"projects/xxxxxxxxxxxx/databases/(default)","addTarget":{"query":{"parent":"projects/xxxxxxxxxxxx/databases/(default)","structuredQuery":{"from":[{"collectionId":"sites"}]}},"targetId":240,"resumeToken":{"type":"Buffer","data":[10,9,8,130,131,139,187,247,146,215,2]}}}
Firestore (0.8.2) 2017-10-28T08:49:12.886Z [Firestore.readWriteStream]: Streaming request: {"database":"projects/xxxxxxxxxxxx/databases/(default)","addTarget":{"query":{"parent":"projects/xxxxxxxxxxxx/databases/(default)","structuredQuery":{"from":[{"collectionId":"sites"}]}},"targetId":240,"resumeToken":{"type":"Buffer","data":[10,9,8,130,131,139,187,247,146,215,2]}}}
Firestore (0.8.2) 2017-10-28T08:49:12.886Z [Firestore._initializeStream]: Marking stream as healthy
Firestore (0.8.2) 2017-10-28T08:49:12.886Z [Firestore._initializeStream]: Releasing stream
Firestore (0.8.2) 2017-10-28T08:49:12.886Z [Watch.onSnapshot]: Opened new stream

[saminiemi]

We also have caught following error:

Firestore (0.8.2) 2017-10-29T20:18:09.700Z [Firestore._initializeStream]: Received stream error: { Error: Deadline Exceeded
    at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/node/src/client.js:255:19)
    at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/node/src/client.js:233:8)
    at /usr/src/app/node_modules/grpc/src/node/src/client.js:757:12 code: 4, metadata: Metadata { _internal_repr: {} } }
Firestore (0.8.2) 2017-10-29T20:18:09.700Z [Watch.onSnapshot]: Stream ended, re-opening
Firestore (0.8.2) 2017-10-29T20:18:09.700Z [Watch.onSnapshot]: Opening new stream
Firestore (0.8.2) 2017-10-29T20:18:09.700Z [Firestore.readWriteStream]: Opening stream
Firestore (0.8.2) 2017-10-29T20:18:09.701Z [Firestore._initializeStream]: Sending request: {"database":"projects/xxxxxxxxxxxxx/databases/(default)","addTarget":{"query":{"parent":"projects/xxxxxxxxxxxxx/databases/(default)","structuredQuery":{"from":[{"collectionId":"sites"}]}},"targetId":240,"resumeToken":{"type":"Buffer","data":[10,9,8,141,191,180,169,211,150,215,2]}}}
Firestore (0.8.2) 2017-10-29T20:18:09.701Z [Firestore.readWriteStream]: Streaming request: {"database":"projects/xxxxxxxxxxxxx/databases/(default)","addTarget":{"query":{"parent":"projects/xxxxxxxxxxxxx/databases/(default)","structuredQuery":{"from":[{"collectionId":"sites"}]}},"targetId":240,"resumeToken":{"type":"Buffer","data":[10,9,8,141,191,180,169,211,150,215,2]}}}
Firestore (0.8.2) 2017-10-29T20:18:09.702Z [Firestore._initializeStream]: Marking stream as healthy
Firestore (0.8.2) 2017-10-29T20:18:09.702Z [Firestore._initializeStream]: Releasing stream
Firestore (0.8.2) 2017-10-29T20:18:09.702Z [Watch.onSnapshot]: Opened new stream
Firestore (0.8.2) 2017-10-29T20:18:09.703Z [Firestore._initializeStream]: Received stream error: { Error: Endpoint read failed
    at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/node/src/client.js:255:19)
    at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/node/src/client.js:233:8)
    at /usr/src/app/node_modules/grpc/src/node/src/client.js:757:12 code: 14, metadata: Metadata { _internal_repr: {} } }
Firestore (0.8.2) 2017-10-29T20:18:09.703Z [Watch.onSnapshot]: Stream ended, sending error:  { Error: Endpoint read failed
    at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/node/src/client.js:255:19)
    at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/node/src/client.js:233:8)
    at /usr/src/app/node_modules/grpc/src/node/src/client.js:757:12 code: 14, metadata: Metadata { _internal_repr: {} } }
Firestore (0.8.2) 2017-10-29T20:18:09.739Z [Watch.onSnapshot]: Invoking onError with: Error: Endpoint read failed
Error: Error: Endpoint read failed
    at sendError (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:254:15)
    at maybeReopenStream (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:268:9)
    at BunWrapper.currentStream.on.err (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:301:13)
    at emitOne (events.js:120:20)
    at BunWrapper.emit (events.js:210:7)
    at StreamProxy.<anonymous> (/usr/src/app/node_modules/bun/lib/bun.js:31:21)
    at emitOne (events.js:120:20)
    at StreamProxy.emit (events.js:210:7)
    at ClientDuplexStream.<anonymous> (/usr/src/app/node_modules/google-gax/lib/streaming.js:130:17)
    at emitOne (events.js:115:13)
Firestore (0.8.2) 2017-10-29T20:18:09.741Z [Watch.onSnapshot]: Processing stream end

And this. We have paid plan, so I wonder which quota this might be.

Firestore (0.8.2) 2017-10-29T15:06:12.415Z [Firestore.readWriteStream]: Received response: {"targetChange":{"targetIds":[240],"targetChangeType":"REMOVE","cause":{"details":[],"code":8,"message":"Quota exceeded."},"resumeToken":[],"readTime":null},"responseType":"targetChange"}
Firestore (0.8.2) 2017-10-29T15:06:12.415Z [Watch.onSnapshot]: Processing target change
Firestore (0.8.2) 2017-10-29T15:06:12.415Z [Watch.onSnapshot]: Invoking onError with: Error 8: Quota exceeded.
Firestore (0.8.2) 2017-10-29T15:06:12.416Z [Watch.onSnapshot]: Processing stream end
Error: Error 8: Quota exceeded.
    at sendError (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:254:15)
    at DestroyableTransform.stream.on.proto (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:532:13)
    at emitOne (events.js:115:13)
    at DestroyableTransform.emit (events.js:210:7)
    at addChunk (/usr/src/app/node_modules/readable-stream/lib/_stream_readable.js:284:12)
    at readableAddChunk (/usr/src/app/node_modules/readable-stream/lib/_stream_readable.js:271:11)
    at DestroyableTransform.Readable.push (/usr/src/app/node_modules/readable-stream/lib/_stream_readable.js:238:10)
    at DestroyableTransform.Transform.push (/usr/src/app/node_modules/readable-stream/lib/_stream_transform.js:146:32)
    at afterTransform (/usr/src/app/node_modules/readable-stream/lib/_stream_transform.js:102:51)
    at TransformState.afterTransform (/usr/src/app/node_modules/readable-stream/lib/_stream_transform.js:79:12)
Firestore (0.8.2) 2017-10-29T15:06:12.523Z [Firestore._initializeStream]: Received stream end

[jkildea]

Hi, is there an update on this issue? Was the solution found?

[saminiemi]

Hello!

@schmidt-sebastian any update on this one? We are still getting this regularly, at least once in a day. Tried it also on Debian-based docker container, with same result.

Most of the time error log looks like this:

Firestore (0.8.2) 2017-11-07T19:25:51.263Z [Watch.onSnapshot]: Opened new stream
Firestore (0.8.2) 2017-11-07T19:25:51.264Z [Firestore._initializeStream]: Received stream error: { Error: Endpoint read failed
    at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/client.js:255:19)
    at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/client.js:233:8)
    at /usr/src/app/node_modules/grpc/src/client.js:757:12 code: 14, metadata: Metadata { _internal_repr: {} } }
Firestore (0.8.2) 2017-11-07T19:25:51.264Z [Watch.onSnapshot]: Stream ended, sending error:  { Error: Endpoint read failed
    at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/client.js:255:19)
    at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/client.js:233:8)
    at /usr/src/app/node_modules/grpc/src/client.js:757:12 code: 14, metadata: Metadata { _internal_repr: {} } }
Firestore (0.8.2) 2017-11-07T19:25:51.264Z [Watch.onSnapshot]: Invoking onError with: Error: Endpoint read failed
Error: Error: Endpoint read failed
    at sendError (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:254:15)
    at maybeReopenStream (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:268:9)
    at BunWrapper.currentStream.on.err (/usr/src/app/node_modules/@google-cloud/firestore/src/watch.js:301:13)
    at emitOne (events.js:120:20)
    at BunWrapper.emit (events.js:210:7)
    at StreamProxy.<anonymous> (/usr/src/app/node_modules/bun/lib/bun.js:31:21)
    at emitOne (events.js:120:20)
    at StreamProxy.emit (events.js:210:7)
    at ClientDuplexStream.<anonymous> (/usr/src/app/node_modules/google-gax/lib/streaming.js:130:17)
    at emitOne (events.js:115:13)
Firestore (0.8.2) 2017-11-07T19:25:51.265Z [Watch.onSnapshot]: Processing stream end

[schmidt-sebastian]

These log statements are very helpful. Our retry logic considers these streams as healthy once it is able to send out a network package ("Marking stream as healthy"). Unfortunately, just because the TCP layer accepts our packages, doesn't necessarily indicate that the outbound network link is active. We may have to retry more aggressively. I will kick off an internal discussion.

[sqra]

Hi. I have the same problem. With onSnapshot method.

    admin.itemsRef.onSnapshot(function (querySnapshot){
        console.log("Received query snapshot of size "+querySnapshot.size);
            mqtt.sendPushboxUpdate('Update available !');
    }, function(err) {
        console.log("Encountered error: "+err);
    });

Firebase console:

2017-11-09T20:06:42.280530+00:00 app[web.1]: Firestore (0.8.2) 2017-11-09T20:06:42.280Z [Firestore._initializeStream]: Marking stream as healthy
2017-11-09T20:06:42.280577+00:00 app[web.1]: Firestore (0.8.2) 2017-11-09T20:06:42.280Z [Firestore._initializeStream]: Releasing stream
2017-11-09T20:06:42.280699+00:00 app[web.1]: Firestore (0.8.2) 2017-11-09T20:06:42.280Z [Watch.onSnapshot]: Opened new stream
2017-11-09T20:06:42.351243+00:00 app[web.1]: Firestore (0.8.2) 2017-11-09T20:06:42.351Z [Firestore._initializeStream]: Received stream error: { Error: Transport closed
2017-11-09T20:06:42.351246+00:00 app[web.1]:     at ClientDuplexStream._emitStatusIfDone (/app/node_modules/firebase-admin/node_modules/grpc/src/node/src/client.js:255:19)
2017-11-09T20:06:42.351246+00:00 app[web.1]:     at ClientDuplexStream._receiveStatus (/app/node_modules/firebase-admin/node_modules/grpc/src/node/src/client.js:233:8)
2017-11-09T20:06:42.351248+00:00 app[web.1]:     at /app/node_modules/firebase-admin/node_modules/grpc/src/node/src/client.js:757:12 code: 14, metadata: Metadata { _internal_repr: {} } }
2017-11-09T20:06:42.351544+00:00 app[web.1]: Firestore (0.8.2) 2017-11-09T20:06:42.351Z [Watch.onSnapshot]: Stream ended, sending error:  { Error: Transport closed
2017-11-09T20:06:42.351545+00:00 app[web.1]:     at ClientDuplexStream._emitStatusIfDone (/app/node_modules/firebase-admin/node_modules/grpc/src/node/src/client.js:255:19)
2017-11-09T20:06:42.351546+00:00 app[web.1]:     at ClientDuplexStream._receiveStatus (/app/node_modules/firebase-admin/node_modules/grpc/src/node/src/client.js:233:8)
2017-11-09T20:06:42.351547+00:00 app[web.1]:     at /app/node_modules/firebase-admin/node_modules/grpc/src/node/src/client.js:757:12 code: 14, metadata: Metadata { _internal_repr: {} } }
2017-11-09T20:06:42.351903+00:00 app[web.1]: Firestore (0.8.2) 2017-11-09T20:06:42.351Z [Watch.onSnapshot]: Invoking onError with: Error: Transport closed
2017-11-09T20:06:42.352141+00:00 app[web.1]: Encountered error: Error: Error: Transport closed
2017-11-09T20:06:42.352380+00:00 app[web.1]: Firestore (0.8.2) 2017-11-09T20:06:42.352Z [Watch.onSnapshot]: Processing stream end

And earlier today:

2017-11-09T13:16:25.184731+00:00 app[web.1]: Error: Error: Transport closed
2017-11-09T13:16:25.184749+00:00 app[web.1]:     at sendError (/app/node_modules/firebase-admin/node_modules/@google-cloud/firestore/src/watch.js:254:15)
2017-11-09T13:16:25.184751+00:00 app[web.1]:     at maybeReopenStream (/app/node_modules/firebase-admin/node_modules/@google-cloud/firestore/src/watch.js:268:9)
2017-11-09T13:16:25.184752+00:00 app[web.1]:     at BunWrapper.currentStream.on.err (/app/node_modules/firebase-admin/node_modules/@google-cloud/firestore/src/watch.js:301:13)
2017-11-09T13:16:25.184754+00:00 app[web.1]:     at emitOne (events.js:101:20)
2017-11-09T13:16:25.184754+00:00 app[web.1]:     at BunWrapper.emit (events.js:188:7)
2017-11-09T13:16:25.184755+00:00 app[web.1]:     at StreamProxy.<anonymous> (/app/node_modules/firebase-admin/node_modules/bun/lib/bun.js:31:21)
2017-11-09T13:16:25.184756+00:00 app[web.1]:     at emitOne (events.js:101:20)
2017-11-09T13:16:25.184756+00:00 app[web.1]:     at StreamProxy.emit (events.js:188:7)
2017-11-09T13:16:25.184757+00:00 app[web.1]:     at ClientDuplexStream.<anonymous> (/app/node_modules/firebase-admin/node_modules/google-gax/lib/streaming.js:130:17)
2017-11-09T13:16:25.184757+00:00 app[web.1]:     at emitOne (events.js:96:13)
2017-11-09T13:16:25.184759+00:00 app[web.1]:     at ClientDuplexStream.emit (events.js:188:7)
2017-11-09T13:16:25.184760+00:00 app[web.1]:     at ClientDuplexStream._emitStatusIfDone (/app/node_modules/firebase-admin/node_modules/grpc/src/node/src/client.js:258:12)
2017-11-09T13:16:25.184760+00:00 app[web.1]:     at ClientDuplexStream._receiveStatus (/app/node_modules/firebase-admin/node_modules/grpc/src/node/src/client.js:233:8)
2017-11-09T13:16:25.184762+00:00 app[web.1]:     at /app/node_modules/firebase-admin/node_modules/grpc/src/node/src/client.js:757:12

Please solve this issue. I making app which I have to give to the client next week ! All last days I spent over looking for bugs in my application but nothing found.

[schmidt-sebastian]

We are aiming to push a release that addresses these network issues within the next two weeks.