A fast-track process for security updates?

[chorrell]

Hi there,

With the recent security updates for the Node.js image on my mind (#1239) I wanted to open up a discussion to see if there are ways to fast-track adding Docker images that have security updates. The current process is fine for typical updates, but for security updates it would be great if we could release something sooner, perhaps in an automated fashion.

Aside from the turnaround time, we also hit a recurring issue of having to explain to users of the docker-node image how the update process works, and why a given update hasn't landed yet. Granted, we probably need an FAQ of some sort to explain the process, but I think this also speaks to the turn around time:

nodejs/docker-node#75
nodejs/docker-node#68
nodejs/docker-node#61
nodejs/docker-node#60
nodejs/docker-node#27

For typical updates, I wouldn't characterize this as being all that bad, but for security related releases it doesn't look so good for Docker or Node.js.

I'll also admit that I don't have an ideal turnaround time in mind. A few hours? For security updates I guess that seems reasonable, but then it depends on the severity. Typically anything longer than 8-12 hours seems to make users anxious.

Anyway, just wanted to kick this off to see what options might be available.

Thanks!

[tianon]

I would love to work together to figure out a way to fast-track security updates better! 😄 👍

I've started a build test of #1239 and will be merging it as soon as that finishes, but I would point out that reading the text of #1239, the only indication that it's a security update is the word "security" in the URL at the bottom (which I normally would've ignored -- it looks just like a normal "bump version" request with a link to the upstream announcement), so I think an easy first step would be to come up with a simple way to more clearly mark a security bump as such.

As an additional datapoint, that specific PR was filed basically right as I was climbing into bed, so the timing wasn't really great either and I'm not sure whether there's a good solution to that; even if we had something that automated the merge, it's not going to automate the official build/push. Any particular reason the Node project has been making a habit of pushing the security updates at strange times (such as EOD in the US, or right at EOD before a US holiday weekend as I believe the previous one was)?

For reference, we handle OpenSSL CVEs by starting a tracking bug (such as #1235) as soon as we see the upstream announcement to figure out what level of update they're going to require, but they require a lot more moving parts to align in order to actually make an update, and then usually a full rebuild of our whole library, where Node is much simpler (there's a CVE, there's updated Node, we build updated Node).

[chorrell]

Hi!

Yeah, it occurred to me after I filed this that we need some agreed upon way to flag security updates to make them obvious. What's the best approach there? Have something in the title and add a suitable label?

And admittedly, the timing kind of sucked. I'm east-coast and happened to already be working on something related when the recent releases landed. Typically things don't always work in our favour like that.

I'm down stream of these things, so I can't really speak to the release timing; that's probably worth raising with the build group (https://github.com/nodejs/build). Personally, I'd prefer releases to land closer to the middle of the week to give us more time to put things together. I think that's worth raising as a concern since the timing of things surely affects other downstream projects too.

[yosifkit]

I think that if there is a security fix on its way then an early warning would be the easiest way to get our attention. Maybe @wblankenship being part of the node build and docker teams can help give us the advanced notice of security releases so that we can get a faster turn around on the official images PRs. Filing an issue is fine, but if you'd rather keep them less public before release an email would be fine with me (and I am pretty sure @tianon wouldn't mind either).

Tags on the PR are nice, but I usually process the queue of PRs and issues through email so I don't see tags until I visit it. Maybe a [Security] label in the title could also help?

[ncopa]

There is a mailing list for early notifications for distros. http://oss-security.openwall.org/wiki/mailing-lists/distros Might be docker could get subscription there?

[chorrell]

I think a heads-up email from the build team would be ideal.

For highlighting security releases, I'd like to adopt a security label and also have [Security] in the PR name/description.

[yosifkit]

Sounds good to me. What do you think @tianon?

[tianon]

I'm definitely +1 to a [security] prefix in the PR title, and additionally to an email heads-up when possible that something's coming (the more in-advance the notice, the more likely we can actually arrange to be able to do something about the bump when we get it). As we've discussed here, the track record for the day of week / time of day that the Node security updates in particular have dropped in the past has been less than ideal, but workable if we'd had some advance notice that something was coming. 😄

[tianon]

@ncopa if we could manage to get ourselves onto http://oss-security.openwall.org/wiki/mailing-lists/distros, that'd definitely be helpful, but I'm not sure what the process would be; it appears they prefer personal members to groups since they encrypt every message individually to the intended recipient, so using my tianon@dockerproject.org email address might be our best chance at being able to get on that list reasonably -- if there's a more formal HREF for what the process of getting on the list is, I'd love to get that ball rolling 😄

[tianon]

@chorrell IMO, this [security] prefix has been working reasonably well -- your thoughts? 😄

[chorrell]

Yep, agreed!