🐛 The user running cloudflared process has a GID (group ID) that is not within ping_group_range

[FezVrasta]

Describe the bug

I'm seeing the following error while running cloudflared through the official Docker image.

WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 65532 is not between ping group 1 to 0"

After it, I also see:

WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 65532 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.

To Reproduce
Steps to reproduce the behavior:

1. Start cloudflared on a Kubernetes GKE cluster on Google Cloud

Expected behavior
These errors seem harmless, everything seem to be working fine, but I would like to understand why I'm seeing them and if I can do something to stop them.

Environment and versions

• OS: [e.g. MacOS]
• Architecture: [e.g. AMD, ARM]
• Version: [e.g. 2022.02.0]

Logs and errors
If applicable, add logs or errors to help explain your problem.

Additional context
Add any other context about the problem here.

[cyanidium]

I found this while searching for the same problem, but there is already a fix in Issue #1109 (comment here)

For the UDP buffer I'm running on baremetal and was able to update the sysctls as described in the link from the logs.

[FezVrasta]

I'm unable to set the sysctls calls on GKE autopilot 😞

 unknown field "spec.securityContext"

[qinyuhang]

I got the same error here
2024-10-28T15:16:48Z INF Starting tunnel tunnelID=deleted_here
2024-10-28T15:16:48Z INF Version 2024.10.1 (Checksum b32e729d43adb66d22abf6539e287b436b1c312742c2488514ef6ea0a2d37adf)
2024-10-28T15:16:48Z INF GOOS: linux, GOVersion: go1.22.2-devel-cf, GoArch: amd64
2024-10-28T15:16:48Z INF Settings: map[no-autoupdate:true token:*****]
2024-10-28T15:16:48Z INF Generated Connector ID: deleted_here
2024-10-28T15:16:48Z INF Initial protocol quic
2024-10-28T15:16:48Z INF ICMP proxy will use deleted_here as source for IPv4
2024-10-28T15:16:48Z INF ICMP proxy will use :: as source for IPv6
2024-10-28T15:16:48Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 1 is not between ping group 1 to 0"
2024-10-28T15:16:48Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 1 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2024-10-28T15:16:48Z INF Starting metrics server on 127.0.0.1:42361/metrics
2024-10-28T15:16:48Z INF You requested 4 HA connections but I can give you at most 0.
2024-10-28T15:16:48Z INF Tunnel server stopped
2024-10-28T15:16:48Z ERR Initiating shutdown error="there are no free edge addresses left to resolve to"
2024-10-28T15:16:49Z INF Metrics server stopped
there are no free edge addresses left to resolve to

[M4st3rITA]

+1 I've the same problem....now my tunnel is DOWN from CluoudFlare admin panel

I also tried to turn ON the ICMP feature, but nothing happens

2024-11-01T18:19:10Z INF Starting tunnel tunnelID=9fb1bb25-8e7b-458f-87f5-f8ec91251ac1
2024-11-01T18:19:10Z INF Version 2024.10.1 (Checksum b32e729d43adb66d22abf6539e287b436b1c312742c2488514ef6ea0a2d37adf)
2024-11-01T18:19:10Z INF GOOS: linux, GOVersion: go1.22.2-devel-cf, GoArch: amd64
2024-11-01T18:19:10Z INF Settings: map[no-autoupdate:true token:*****]
2024-11-01T18:19:10Z INF Generated Connector ID: 235d312f-0763-44f8-9086-b53ecc3b61b0
2024-11-01T18:19:10Z INF Initial protocol quic
2024-11-01T18:19:10Z INF ICMP proxy will use 192.168.1.100 as source for IPv4
2024-11-01T18:19:10Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
2024-11-01T18:19:10Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 65532 is not between ping group 1 to 0"
2024-11-01T18:19:10Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 65532 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2024-11-01T18:19:10Z INF Starting metrics server on 127.0.0.1:33565/metrics
2024-11-01T18:19:10Z ERR Failed to serve quic connection error="context canceled" connIndex=0 event=0 ip=198.41.192.107
2024-11-01T18:19:10Z INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.192.107
2024-11-01T18:19:10Z INF Tunnel server stopped
2024-11-01T18:19:10Z ERR Initiating shutdown error="context canceled"
2024-11-01T18:19:11Z INF Metrics server stopped
context canceled

[se4203]

I found this while searching for the same problem, but there is already a fix in Issue #1109 (comment here)

For the UDP buffer I'm running on baremetal and was able to update the sysctls as described in the link from the logs.

@cyanidium , please provide more details how to use this information. I'm running cloudflared as a Docker container in Unraid.
If the snippets you point to resolve the issue, then perhaps they should be implemented in the Docker image (by the maintainer).

[cyanidium]

I'm not familiar with Unraid. If you're running it in Docker on linux, then on the machine that is running docker you need to run as root (or with sudo):

sysctl -w net.core.rmem_max=7500000
sysctl -w net.core.wmem_max=7500000

This resets after each boot. If you want to persist the changes, and you use systemd, then you can create a file /etc/sysctl.d/fix-buffer.conf which contains:

net.core.rmem_max=7500000
net.core.wmem_max=7500000

That first set of commands is what you find at the link in the error message. If you're not running linux, there are alternate instructions at that link. This isn't a fatal error, though, so really what you want to fix is the ping_group_range issue.

The error in the OP says Group ID 65532 is not between ping group 1 to 0. This means cloudflared is running as GID 65532. You need to ensure that whatever GID cloudflared is running as is between the values in net.ipv4.ping_group_range. You can check what the range is using sysctl net.ipv4.ping_group_range, but it's also in the error message (1 to 0 means greater than 1 and less than 0, so no groups can ping because there's no valid/allowed range). You can set the value using sysctl -w "net.ipv4.ping_group_range=0 2000000" as root (or with sudo). Ensure that the lower (first) number <= the GID for cloudflared <= upper (second) number. Your cloudflared is likely not running with GID 65532, so make sure you check your error message. Again, this will reset each boot, so you'll want to create a file in /etc/sysctl.d with:

net.ipv4.ping_group_range=0 2000000

Again, this applies to linux, and I'm not sure what the equivalent commands are for other platforms.

If you use Kubernetes, you can (should?) set the ping_group_range value in the pod securityContext as described in the link I gave. I'm not sure if you need to do anything special with Docker or if setting it on the host machine is sufficient.

If the snippets you point to resolve the issue, then perhaps they should be implemented in the Docker image (by the maintainer).

The settings are security constraints on the host machine that the container is running on. Some cloud environments don't seem to allow you to change these settings at all, most container environments frown upon anything needing root privileges. The settings also apply to all processes on the host machine, so if you want the range to be "1000 1000" and someone else wants it to be "2000 2000" then who wins? I don't work for cloudflare, but this is not something that will get fixed in the container image.

Does that help?

[cyanidium]

I'm unable to set the sysctls calls on GKE autopilot 😞

 unknown field "spec.securityContext"

I can't find anything in the GKE Autopilot docs to suggest that this sysctl is blocked, in fact the docs indicate that it should be allowed given it's considered a "safe sysctl". That error message makes it seem like you've put the securityContext section in the wrong part of the YAML file. If you're creating a Deployment, then it should be in spec.template.spec.securityContext.

[149segolte]

The error in the OP says Group ID 65532 is not between ping group 1 to 0. This means cloudflared is running as GID 65532. You need to ensure that whatever GID cloudflared is running as is between the values in net.ipv4.ping_group_range. You can check what the range is using sysctl net.ipv4.ping_group_range, but it's also in the error message (1 to 0 means greater than 1 and less than 0, so no groups can ping because there's no valid/allowed range). You can set the value using sysctl -w "net.ipv4.ping_group_range=0 2000000" as root (or with sudo). Ensure that the lower (first) number <= the GID for cloudflared <= upper (second) number. Your cloudflared is likely not running with GID 65532, so make sure you check your error message. Again, this will reset each boot, so you'll want to create a file in /etc/sysctl.d with:

net.ipv4.ping_group_range=0 2000000

Even after updating the ping_group_range and rebooting, the change is not reflected inside the container. Does the container contain sysctl? Do we have to mount the .conf files inside the container as bind mounts?

[one49segolte@coreos-proxmox-1 ~]$ sysctl net.ipv4.ping_group_range
net.ipv4.ping_group_range = 0   2147483647

[one49segolte@coreos-proxmox-1 ~]$ podman logs cloudflared
2025-01-24T14:30:53Z INF Starting tunnel tunnelID=f35fcb1a-7278-...
2025-01-24T14:30:53Z INF Version 2025.1.0 (Checksum d4cf352fbf29a3f5ae7726f6f0a13d00da8934661f0caa3c633f907812940968)
2025-01-24T14:30:53Z INF GOOS: linux, GOVersion: go1.22.5-devel-cf, GoArch: amd64
2025-01-24T14:30:53Z INF Settings: map[no-autoupdate:true token:*****]
2025-01-24T14:30:53Z INF Generated Connector ID: f4ae5173-b0ed-...
2025-01-24T14:30:53Z INF Initial protocol quic
2025-01-24T14:30:53Z INF ICMP proxy will use 10.89... as source for IPv4
2025-01-24T14:30:53Z INF ICMP proxy will use fe80::5cf6:... in zone eth0 as source for IPv6
2025-01-24T14:30:53Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 65532 is not between ping group 0 to 0"
2025-01-24T14:30:53Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 65532 is not between ping group 0 to 0 nor ICMPv6 proxy: socket: permission denied"
2025-01-24T14:30:53Z INF ICMP proxy will use 10.89.1.213 as source for IPv4
2025-01-24T14:30:53Z INF ICMP proxy will use fe80::5cf6:... in zone eth0 as source for IPv6
2025-01-24T14:30:53Z INF Starting metrics server on [::]:20241/metrics
2025/01/24 14:30:53 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2025-01-24T14:30:53Z INF Registered tunnel connection connIndex=0 connection=45cc7945-6119-... event=0 ip=198.41... location=atl11 protocol=quic
2025-01-24T14:30:53Z INF Registered tunnel connection connIndex=1 connection=29a21285-f265-... event=0 ip=198.41... location=atl06 protocol=quic
2025-01-24T14:30:54Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"*...dev\",\"originRequest\":{\"access\":{\"audTag\":[],\"required\":false}},\"service\":\"http://traefik\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=15
2025-01-24T14:30:55Z INF Registered tunnel connection connIndex=2 connection=931015b1-130a-... event=0 ip=198.41... location=atl01 protocol=quic
2025-01-24T14:30:56Z INF Registered tunnel connection connIndex=3 connection=dace2f89-1eeb-... event=0 ip=198.41... location=atl10 protocol=quic

[one49segolte@coreos-proxmox-1 ~]$

[cyanidium]

Even after updating the ping_group_range and rebooting, the change is not reflected inside the container. Does the container contain sysctl? Do we have to mount the .conf files inside the container as bind mounts?

@149segolte have you tried setting the sysctl value when you podman run?

[149segolte]

Yes. Just tried removing the container, running sysctl -w "net.ipv4.ping_group_range=0 2000000", then running,

podman run --net systemd-traefik --name cloudflared --restart unless-stopped -d docker.io/cloudflare/cloudflared:latest tunnel --no-autoupdate run --token <token>

It still gives the same logs.

[cyanidium]

Yes. Just tried removing the container, running sysctl -w "net.ipv4.ping_group_range=0 2000000", then running,

podman run --net systemd-traefik --name cloudflared --restart unless-stopped -d docker.io/cloudflare/cloudflared:latest tunnel --no-autoupdate run --token <token>

It still gives the same logs.

The link I gave is for a podman flag; I meant that you should try something like:

podman run --net systemd-traefik --name cloudflared --sysctl=net.ipv4.ping_group_range="0 2000000" --restart unless-stopped -d docker.io/cloudflare/cloudflared:latest tunnel --no-autoupdate run --token <token>

Note the --sysctl part in the middle

[149segolte]

Thank you for the help! @cyanidium

Yes, the --sysctl flag works. But with a slight modification. Using --sysctl=net.ipv4.ping_group_range="0 2000000" gives the error,

Error: OCI runtime error: crun: write to `/proc/sys/net/ipv4/ping_group_range`: Invalid argument

But then I found the thread: podman#21031 so I tried,

--sysctl=net.ipv4.ping_group_range="65532 65532"

This works.

[pokwir]

Not sure if this solves anything, what worked for me was to revert to the last known version that worked

docker pull cloudflare/cloudflared:2024.4.1