Guide on reusing docker host SSH server

[IlyaSemenov]

This PR adds explanation on how to use the SSH daemon on the Docker host to connect to Gitlab, and adds the option to set more relaxed permissions if needed (see #725). This needs proofreading at the very least.

[sameersbn]

I'd prefer documenting this procedure in a supporting document and linking to it from the main README.

[IlyaSemenov]

@sameersbn I did as you asked, please check the updated PR.

[apertureless]

Any update on this?

[IlyaSemenov]

@apertureless I rebased the PR against 8.10.4
I also published a Docker image with the applied patch as ilyasemenov/gitlab:8.10.4 (see https://hub.docker.com/r/ilyasemenov/gitlab/tags/)

[sameersbn]

Sorry I completely missed tracking this PR. Will review and merge soon. Thanks.

[solidnerd]

I will throw my 2 cents in before we start to merge this. I think it's not a good idea to introduce SSH_AUTHORIZED_KEYS_WORLD_READABLE in my opinion it's a hack and this will result in a security vulnerability. I think we could do this only with the correct setting of USERMAP_UID and USERMAP_GID like you described. I know that you post something about it why you want to make it readable.

Just in case, I am aware about USERMAP_UID/USERMAP_GID but in some circumstances it's not convenient (e.g. it breaks linking to sameersbn/redmine container which expects them to be 1000)

I think it's a separate problem of sameersbn/redmine or generally share the keys. You could propose a way there to use another UID and GID. If you want i can help you to do this.

[IlyaSemenov]

This is surely a hack, but for now it gets its job done. Revealing the list of public keys is not much of a vulnerability in most circumstances. In a sense, this is not much different from revealing /etc/passwd which is already world-readable.

In the end, yes, it would suffice to provide a way for Redmine to use repositories with non-default UID/GID, although that would be hacky as well. To support arbitrary GID, one would end up with something like:

# In Redmine or other container that needs to access Gitlab data directory
group=$(getent group $GID | cut -d: -f1)
if [ -z "$group" ]; then
    group=group_$gid
    groupadd -g $gid $group
fi
usermod -aG $group www-data

Not very elegant at best, and in theory prone to vulnerabilities as well (if GID matches with some non-secure service inside the container).

[IlyaSemenov]

Anyway, I removed SSH_AUTHORIZED_KEYS_WORLD_READABLE and now this PR is purely about adding the guide. The permissions problem when keeping default uid/gid is now advised to be solved with incron. The guide specifically includes the warning for that case:

PLEASE NOTE: keeping .authorized_keys world-readable brings a security risk. Use this method with discretion.

[ostueker]

PLEASE NOTE: keeping .authorized_keys world-readable brings a security risk. Use this method with discretion.

Is it? Your public SSH key is already public on GitHub: https://github.com/IlyaSemenov.keys
(and so are mine and everyone else's).

[lenovouser]

@ostueker yep, it is because every user on the system could write in it

[MeinAccount]

@lenovouser world-readable is not world-writeable
Or am I missing something?

[splattael]

It should read GitLab.

[IlyaSemenov]

Thanks, fixed this one.

[IlyaSemenov]

@ostueker:

1. I am personally not advocating this paranoia, but it is indeed a common belief that authorized_keys shoud not be world-readable. In particular, openssh with StrictModes yes specifically checks that authorized_keys permissions are set to 600.

2. In case of GitLab, I believe there is a certain additional security risk of exposing the list of key names (thus, in away, revealing users who use this particular GitLab installation).

3) In any case, I've updated the incron script to use group-readable authorized_keys rather than world-readable, so it's not a problem anymore. And it's only used for certain scenarios when you want different uid/gid for git on the host and on the container, which is not the default and suggested way.

[IlyaSemenov]

ATTENTION Seems like the incrond script that I've included suffers from the dead loop and raises thousands of zombies, please don't use it directly.

Edit: resolved by the commit below.

[averissimo]

great guide and work @IlyaSemenov _o_

[MichaelEischer]

I think this method of forwarding the git ssh access allows for a full container host compromise from inside the gitlab container!
The security of this approach relies on the fact that users logging in via ssh can't execute arbitrary commands. However, this only holds as long as Gitlab is creating a proper authorized_keys file that includes a command attribute for each key. This might not be the case if Gitlab had a security issue allowing for arbitrary file creation.
Once the command attribute is missing, its possible to login as the git user and run arbitrary docker commands, which is equivalent to root access on the container host!

My current workaround for this is to chain two ssh connections. The git user on the container host uses the authorized_keys file from gitlab, but also has its own key-pair that is used to forward the connection to the gitlab container's ssh. The container then uses a suitable sshd_config to force use of a handcrafted authorized_keys file that allows the host git user to connect. In case there's interest in my solution, I can write a detailed tutorial for it.

[mikebentley15]

I'd be interested in such a tutorial of how to chain the two ssh servers. Do you have it automated so that when I use the git user, it transparently connects to the docker container?

[IlyaSemenov]

@MichaelEischer I agree that you bring valid security concern, and that the double SSH trick will avoid it. What I'm thinking however is that perhaps it's easier to remove the host git user from docker group and instead add a custom sudo rule on the host which allows user git to run sudo docker exec -i -u git gitlab <anything> specifically (I don't think /etc/sudoers allows wildcard arguments, so it will rather need one more root-owned proxy script that will pull environment variables and actually run docker exec).

[MichaelEischer]

sudo is in fact able to handle wildcard arguments. This might look like the following /etc/sudoers entry:
example ALL = (root) NOPASSWD: /usr/bin/docker image *

This will allow the example user to run commands like sudo docker image ls or sudo docker image ls --quiet, while ensuring that image and at least one parameter is present. Passing data via environment variables should also be possible, but requires whitelisting of the used environment variables (sudo sanitizes the environment to prevent privileged applications from getting confused).

[IlyaSemenov]

I updated the document correspondingly. Thanks for bringing this matter up, @MichaelEischer.

[MichaelEischer]

@IlyaSemenov Thanks for the update. Btw, the recent gitlab security vulnerability (https://about.gitlab.com/2017/08/10/gitlab-9-dot-4-dot-4-released/) should be enough to allow an attacker to insert arbitrary authorized_key commands. With the git user belonging to the docker group this could then be used for a root privilege escalation on the host.
@mikebentley15 I've finally got around to writing a short description for using gitlab with chained ssh. However, the description got much longer than I'd anticipated even though it could use some additional details. https://gist.github.com/MichaelEischer/fec2ddf1267e3caca796813340044552
I'm currently not sure whether my solution with chained ssh is better/more secure than just using docker exec.

[estan]

I think it's meant to be "if you rely on the default uid and gid" here? (note "gid" instead of "git").

[estan]

Also, capital "F" at the beginning of the sentence.

[IlyaSemenov]

Fixed both typos. Thanks.

[estan]

I'm trying to follow this guide, and I'm unsure at this step. Looking inside my omnibus Docker container, I can only find /var/opt/gitlab/gitlab-shell, not /home/git/gitlab-shell. Does this mean I should create /var/opt/gitlab/gitlab-shell/bin (and intermediate directories) on the host?

[MichaelEischer]

The guide is written with this alternate gitlab docker container in mind. That said, the primary different to the omnibus docker container is probably just different paths. The path on the host must match the path to gitlab-shell inside the container, thus using /var/opt/gitlab/gitlab-shell/bin should work.
I'm, however, not sure which uid/gid for the git user is used by the omnibus docker container. As the host machine should ideally use the same uid/gid for its git user, this may cause further problems.

[estan]

Right, I finally realized that this guide was not for the official Docker images :) Sorry for the noise.

I noticed it when I realized the USERMAP_UID and USERMAP_GID variables were not being respected.

I'll have to figure out if I can do something similar with the offical images, because this "hack" to be able to use port 22 for git is really neat (thanks!).

[IlyaSemenov]

I personally use incron permissions fixer on two different GitLab installs and it works fine. It's far from being elegant for sure, but it gets the job done.

[estan]

@IlyaSemenov: Ah, I wasn't asking about permissions though, but about the directory/file paths in the guide, which I couldn't reconcile with what I was seeing locally. But it was just me being confused and thinking this guide was for the official omnibus image.

(The permission problem I solved by matching the git uid/gid on the host with those hard-coded in the official container, instead of chmod:ing, since we don't want to run OpenSSH with StrictModes off.)

[reberinformatik]

@estan You're mentioning that you solved the permission issue without turning off StrictModes. Could you elaborate this a bit more? Probably in the context of the manual from @IlyaSemenov?
Which commands must be changed to achieve this?

[estan]

@reberinformatik Well, we were running the official omnibus Docker image from GitLab (I didn't realize that this guide was about the alternate image when I first posted here).

We solved the problem by simply making sure that the host system has a git/git user/group with the same UID and GID as the one used inside the official omnibus Docker container (I think that's 998 / 998).

Then, where this guide says

Create the shell proxy script at `/home/git/gitlab-shell/bin/gitlab-shell`

we instead created the shell proxy script at /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell, since this is the path where the offical omnibus Docker container expects to find it.

We did not use any of the steps under ## Advanced: Use default UID/GID in the container in this guide. Since we made sure the host system UID/GID for git/git matches that of the Docker container, there was no need to. The authorized_keys file exposed by the container will have the correct permissions in the eyes of the host SSH daemon (even with StrictModes on).

[estan]

Note that it can be a little awkward to make sure you have a git/git user/group with UID/GID 998/998, since you may already have some other user/group with that UID/GID. We had that problem, but I went through the trouble of changing the UID/GID for that user to make 998/998 available. I thought it was worth the trouble to get GitLab up and running.

[reberinformatik]

@estan Thanks for taking the time to explain this. We also use the official omnibus Docker container. First I just skipped Advanced part in the guide. But as you mentioned, our git-user on the docker-host has a 999 id. Luckily there is no user with 998 yet. So I guess we have to try changing the UID/GID. Thanks!

[solidnerd]

Hello,

I think this pull request is outdate because we already implemented a way how to reuse ssh.

For this see #1731.

If it's not enough please reopen this PR we and start a discussion again.

[IlyaSemenov]

I see. Yes, since you added support for the proxy key to the core this method is easier and more reliable as it doesn't need to monitor permissions. My PR was working with unmodified docker-gitlab. I believe it's fully superseded now, indeed.

What I don't like in your new guide is that it tells to run the large bash script on the host without any explanation on what it's going to do and which distros it's compatible with. But, who cares...