Refactor Repository (Massive update incl Documentation website)

[gjrtimmer]

@solidnerd @sameersbn Massive update of repository.

Cannot be auto merged, requires replacement of master branch.

Main features are:

• documentation is now website auto generation with Hugo.
  • Documentation is still in Markdown so that contributions are very easy.
  • See https://docker.timmertech.nl/gitlab/ for the documentation end result.
  • Final todo is to add the deployment of the documentation website to circleci.
• Entire repository configuration with a global config.yml which means that if you update the container to a new version of gitlab all the version numbers of the documentation will change with it too.
• Documentation uses a config.template file more variables can be added to make the documentation even more automatic. When you run make docs it will generate the config.yml for the documentation and generate it in docs/public see the .gitlab-ci.yml pages job. We still need to add a job / script to circleci which will publish the documentation to the gh-pages branch. This is possible found some examples on github on how to do this.
• After you replaced the master branch with this refactoring you will have to update the config.yml to reflect the config of the sameersnm/docker-gitlab repository.
• Also note that while it looks like contrib files are missing they are actually moved to their respective documentation page as attachment see the following links as examples
  • https://docker.timmertech.nl/gitlab/configuration/examples/
  • https://docker.timmertech.nl/gitlab/guides/kubernetes/
• After circleci job for deployment to branch gh-pages has been added the README.md must be updated to use this repo gh-pages instead of mine.
• Circle CI will now also push to Dockerhub, value of repository image in config.yml (Global) and Circle-CI config has to be changed. Also two variables has to be added to Circle-CI build environment
  • DOCKERHUB_USERNAME
  • DOCKERHUB_PASSWORD
• Upgrade Gitlab: 11.5.4 [Tagged]
• Upgrade Gitlab: 11.5.5 [Tagged]
• Upgrade Gitlab: 11.6.0 [Current]
• Upgrade Gitlab Components

Repository Refactor

• Update build to multi-stage docker build
  • Layer compression due to multistage build
  • TODO: Strip Gitlab of unnecessary files
  • TODO: When COPY --chown option supports env / arg variables, update Dockerfile
• Updated build to use a base image
  • Allow nightly scheduled base image to speed up builds
  • Allow nightly scheduled builder image to speed up build
• Allow user to override base image for faster builds
• Allow user to override builder image for faster builds
• Implemented global configuration (config.yml)
• Implemented new setup for quickstart/testing of image
• Repository refactoring
• Rewrite documentation for Pages (Github & Gitlab)
  • Automatic documentation generation with Hugo
  • Documentation uses the global configuration (config.yml)
• Update format of CHANGELOG for pages generation
• Upgrade Gitlab: 11.6
• Upgrade Components
  • Workhorse: 8.0.0
  • Gitaly: 1.12.0
  • Shell: 8.4.4
• Renamed Changelog.md -> CHANGELOG.md for Gitlab compliancy
• Moved exec_as_git from function to script
• Moved exec_as_git => /sbin because of symlinks / layer compression
• Formatted exec_as_git to 2 spaces ident
• CI
  • Gitlab-CI
    • Update Gitlab-CI to build Pages
    • Update Gitlab-CI to allow user arguments
  • Circle-CI
    • TODO: Update Circle-CI to build Pages and deploy to github pages
    • Switched to docker_layer_caching: true instead of manually saving layers
    • Update image name
    • Added support for build tag
    • Add deploy to DockerHub
      • Add CI-ENV: DOCKERHUB_USERNAME
      • Add CI-ENV: DOCKERHUB_PASSWORD
    • Deploy to Dockerhub is now optional based on presence of DOCKERHUB_* variables
    • Only deploy master and tags to DockerHub
• Add CRON=1 to backup cron command
• Do not send documentation to docker daemon
• Update build scripts suppress
  • Add variable DEBUG
  • Suppress DEBUG logging by default
  • Allow override of DEBUG variable from command line DEBUG=true make docker
  • Suppress GIT Detached HEAD advice
  • Add forced docker fail on failed command
    • Added || exit 1 to non build commands
• Fix Makefile target logs
• Issues / Merge Requests
  • Fixes Explain [OPTIONS] placeholder in docker run commands #690 (Explain [Options])
  • Fixes add env vars for LDAP Attributes #699 (LDAP Attributes)
  • Fixes Guide on reusing docker host SSH server #737 (Add guide on how to use docker host ssh daemon)
  • Fixes Added Docker Labels from label-schema.org #738 (Implemented Labels from org.label-schema)
  • Fixes Create version bump script #1468 (Auto build script)
  • Fixes Fixed readme to match new backup file naming #1472 (Backup restore documentation)
  • Fixes Add support for building the enterprise edition #1477 (Add support for building enterprise edition)
  • Fixes Suggestion: Add pre/post-config hook support to entrypoint.sh #1569 (Implemented pre / post config hook)
  • Fixes Large LFS Transfers Fail #1636 (LFS Uploads)
  • Fixes Fix for LFS Large File upload #1637 (LFS Uploads)
  • Fixes Improvements to readme.md : database restore, mysql, pg_trgm, & app:rake #1695 (Update documentation)
  • Fixes Fix LDAP username issue #1746 (LDAP Attributes)
  • Fixes Fix container timezone #1768 (Container Timezone)
  • Fixes Fix: Gitlab Pages Custom Domain Setup #1791 (Gitlab Pages Custom Domain Setup)
  • Fixes fix-backup-subdirectory #1792 (Backup Subdirectory)
  • Fixes Gitlab Pages is too old / Gitlab Pages Access requires additional arguments #1793 (Gitlab Pages Access Control)
  • Fixes Fix Gitlab Pages Access Control #1794 (Gitlab Pages Access Control)
  • Fixes Versions in Dockerfile don't match gitlab-ce repository #1797 (Versions in Dockerfile don't match gitlab-ce repository)
  • Fixes Add support for custom LDAP name value in config #1814 (LDAP Attributes)
  • Fixes Update to Gitlab Version 11.5.4 #1816 (Update to Gitlab Version 11.5.4)

[gjrtimmer]

@solidnerd @sameersbn

• Prefer not to squash the commit for these will close several open issues and PR's.
• Documentation still requires some work of course but I think this is a good start one of the todo's is to replace all the links for references.
• What is very nice about this refactoring is that everything is now managed from the global config.yml

Note: Documentation must be generated with the make docs commands. If you want to run it on your local machine, install hugo and run make docs-server

[luv2code]

What is your impression about the new documentation ? Is it in the right direction ?

I like it, and I think having an organised document site is a great idea. There is only one thing that I would change. When you click on a top level topic, I would expect it to bring up a page with all the content for the topic on it. Then in the navigation, clicking on a 2nd level topic would take you to anchors within the top level content. That way I can scan the whole top level topic without having to click very much; but if I know the subtopic I'm looking for, I can still quickly jump to it.

maybe do this conditionally; because it doesn't make sense for certain large topics like guides. But for Configuration and Maintenance, I think it makes sense to combine the bottom level subtopics into one page and navigate via anchors.

Thank you.

[CrystalMethod]

Yes, I have looked at the docs briefly and I like them too. But what delights me the most is that your PR includes some very old PRs that I've missed for a long time. (mainly LDAP) So I can only hope that this PR will pass through and serve as a "new basis" for the project. We have been using this container for quite some time and are very happy with it.

[gjrtimmer]

@luv2code Would be a great addition, currently don't know if it is possible maybe just as simple as adding it to the _index.md files.

To give you a quick summary how it works.

• Its built with Hugo
• Theme DocDock
• Theme Demo with all its functionality
• All the documentation content is in docs/content

Check out the demo of the theme to see what an insane amount of features it has.

[gjrtimmer]

@CrystalMethod I've updated the Circle-CI config of this new repository, and it is able to push the latest and tags to DockerHub. Also this PR now includes update of subcomponents and upgrade of Gitlab-CI to 11.5.4. It is available on my DockerHub account. So if you cannot wait until this PR is merged and want to play with the new image check here on DockerHub

Have fun playing.

P.S.
I've you decide to upgrade, as always make full backup of your data, so you can roll back.

*Note: This repository update also fixes the local testing. So if you clone my fork you only have to give make all and afterwards make test it will spin up a full testing environment.
docker-compose located in docker/quickstart.yml exposes localhost port 10080.

Have fun playing

@CrystalMethod I've you start playing with the new 11.5.4 build from this massive update, please let me know in this thread what you think. Any feedback or improvement is welcome.

[solidnerd]

Hey @gjrtimmer,

thank you for your Pull Request. I seems that there is a lot of work in it. But it has also a lot of breaking changes for this repository. Also there are some new features that at the moment out of scope. To clarify we should start a roadmap for the project and future changes. (This will come from my side)

First of all I’ll say that I won’t replace the current master with your master branch, because of the breaking changes and that I can’t guarantee anyone from my side that this will work backwards and with no migration steps. (This is the current state of the Pull Request) .

Now let’s dive inside your Components:

Upgrade Components
Workhorse: 8.0.0
Gitaly: 1.12.0
Shell: 8.4.4

These versions are the latest from master and for the release. This could have breaking changes and breaks the whole releases or unintended side effects.

Implemented global configuration (config.yml)

I’m not a really fan of an global configuration. This requires also a parser and structure for the configuration. I know you implemented it in bash but I won’t increase the readability. I think it increases only complexity, because not everyone has the use case to build different gitlab images.

I think we should cover some features here in the upstream but for this I need sometime to think about it how we can handle this in a good manner with a stable process without breaking changes until 12.X .
For this we need smaller commits and probably more pull requests to implement them separately.

Regarding the merged issues:

• Fixes add env vars for LDAP Attributes #699 (LDAP Attributes)
  • For this routine we need some tests of how to integrate LDAP and see that it’s working.
• Features: GitLab Pages && GitLab Pages Access Control
  • This should be included in the next release. I need only to verify that it works
• LFS Uploads
  • This should be included in the next release. I need only to verify that it works

I hope you can understand my concerns regarding this pull request.
It’s okay for me that you have your own fork with different changes.

[solidnerd]

I'll close this issue cause we can't integrate it directly and we have different directions. But I'm open for discussion so let's move this to an issue.