File descriptor / socket leak when timeout occurs during initialization

[rpeng]

Making a corresponding issue for the related fix PR: #762

Timeouts that occur during connection initialization can cause the socket to never be closed properly, leading to a resource leak. This can manifest in a lot of dangling CLOSED_WAIT or ESTABLISHED sockets that are kept open by the application.

Steps to repro:

In the client:

# irb or rails c
client = HTTP.timeout(5)

client.get("https://www.example.org").body.to_s # or use an unallocated ip like 240.0.0.0
client.get("https://www.example.org").body.to_s
client.get("https://www.example.org").body.to_s

Simultaneously:

1. Set up a mitm to intercept example.org, and add network conditioning to make the request very slow. Or use a custom server and endpoint where you can control the connect delays.
2. Check the sockets opened by the ruby process:

watch -n 1 "lsof -i -P | grep PROCESS_PID" # replace PROCESS_PID with the rails process pid

You'll notice that sockets will stay in ESTABLISHED or CLOSE_WAIT as you make more and more requests that time out.

ruby    28497 rpeng   21u  IPv4 43444771      0t0  TCP f6eb5410737c:57558->93.184.216.34:443 (CLOSE_WAIT)
ruby    28497 rpeng   23u  IPv4 43449275      0t0  TCP f6eb5410737c:56876->93.184.216.34:443 (CLOSE_WAIT)
ruby    28497 rpeng   24u  IPv4 43452649      0t0  TCP f6eb5410737c:34242->93.184.216.34:443 (ESTABLISHED)
ruby    28497 rpeng   25u  IPv4 43458906      0t0  TCP f6eb5410737c:55366->93.184.216.34:443 (ESTABLISHED)
ruby    28497 rpeng   26u  IPv4 43461133      0t0  TCP f6eb5410737c:37510->93.184.216.34:443 (ESTABLISHED)