init

doc/development/gitlab-ctl-commands.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Adding new `gitlab-ctl` commands
+
+New `gitlab-ctl` commands can be added to two directory : `files/gitlab-ctl-commands` and `files/gitlab-ctl-commands-ee`. Chef stores `gitlab-ctl` commands in `/opt/gitlab/embedded/service/omnibus-ctl`.
+
+There are two main functions you will use to add commands (<https://github.com/chef/omnibus-ctl>):
+
+## `add_command_under_category(string, string, string, int, ruby_block)`
+
+This method will add a new command to your `gitlab-ctl` under a category, useful for grouping similar commands together logically in help output.
+
+Input arguments:
+
+1. Name of the command.
+1. Category of the command. It should be string consisting of only characters and "-". If the category does not exist, it will be added. Default categories are "general" and "service-management" (if the latter is enabled).
+1. Description. This will be outputted below the command name when the help command is run.
+1. Arity. Always set to `2`, due to [a bug in `omnibus-ctl`](https://github.com/chef/omnibus-ctl#add_command_under_categorystring-string-string-int-ruby_block).
+1. Ruby block. Ruby code to be executed when your command is run (arguments to that command will be passed into the block).
+
+## `add_command(string, string, int, ruby_block)`
+
+This method will add a new command to your `gitlab-ctl` without a category. It will be displayed above all categories when the help command is called.
+
+Input arguments are the same as `add_command_under_category` except 2 doesn't exist.

doc/development/index.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Contribute to Omnibus GitLab development
+
+## Getting started
+
+1. [Understanding the Omnibus GitLab Architecture](../architecture/index.md)
+1. [Setting up a local development environment](setup.md)
+
+## How everyone can contribute to Omnibus GitLab
+
+- [How to make common enhancements](contribute-to-omnibus-gitlab.md#common-enhancement-tasks)
+- [How to perform common maintenance tasks](contribute-to-omnibus-gitlab.md#common-maintenance-tasks)
+- [Building and testing enhancements](contribute-to-omnibus-gitlab.md#build-and-test-your-enhancement)
+- [Submitting an enhancement merge request for review](contribute-to-omnibus-gitlab.md#submit-your-enhancement-for-review)
+- [Development for s390x architecture](s390x.md)
+
+## Omnibus GitLab continuous integration
+
+- [CI variables](ci-variables.md)
+- [Repositories, branches and CI pipelines](pipelines.md)
+- [Release process](../release/index.md)
+- [Version format of packages, Docker images and other artifacts](package_version_format.md)
+- [Handling broken master pipelines](broken_master.md)
+- [Handling vulnerabilities](vulnerabilities.md)
+
+## Technical notes
+
+- [AWS AMis and Marketplace Listing](aws_amis_and_marketplace_listings.md)
+- [Behind the scenes of `gitlab-ctl reconfigure`](reconfigure_in_detail.md)

doc/development/maintainership.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Maintainership
+
+## Scoped Maintainership
+
+`omnibus-gitlab` is a comparatively complex project that handles both software
+builds and deployment.
+
+The build aspect covers the build toolchain, CI infrastructure, and dependency
+management. The deployment aspect covers the installation and configuration of
+GitLab at the user site along with upgrade tasks.
+
+Because the two aspects are very different both in nature and the technical
+stack involved, trainee-maintainers must spend a large amount of time to gain
+both sets of competencies. Scoped maintainership decreases the time required
+to onboard a new maintainer through separation of these core responsibilities.
+
+### Build scope
+
+The build scope covers all the parts of the codebase that build the artifacts
+used to ship Omnibus GitLab.
+
+This includes:
+
+1. Omnibus project configurations and component software definitions.
+1. Patches used in software definitions.
+1. Libraries and Rake tasks used for build, release, and other maintenance
+   activities.
+1. CI configuration used for build, release, and other maintenance activities.
+1. Infrastructure management required for the above.
+
+Ideally, a build-scoped maintainer of `omnibus-gitlab` should be
+well versed in all the above-mentioned topics. Several of these areas
+get infrequent updates, therefore it is not fair to expect trainee-maintainers
+to work on all of them. The following checklist provides a guideline to evaluate
+the progress of a trainee-maintainer in the build scope.
+
+NOTE:
+We do not differentiate between the trainee-maintainer as the author
+or reviewer in this list because `omnibus-gitlab` is a relatively stable
+and mature project. The majority of merge requests follow established
+patterns and only need to pass a set of well known tests that prove
+the changes work as expected.
+
+1. Author or review merge requests which update any 3 components from the list below that
+   support high availability. The trainee-maintainer should gain familiarity with complex
+   deployment scenarios and how to test them.
+
+     1. PostgreSQL
+     1. Patroni
+     1. PgBouncer
+     1. Consul
+     1. Redis
+     1. Sentinel
+
+1. Author or review a merge request which updates any of our "runtime" environments, preferably
+   Ruby/Go. This ensures familiarity with the
+   [Omnibus Builder](https://gitlab.com/gitlab-org/gitlab-omnibus-builder)
+   project and how it relates to `omnibus-gitlab`.
+
+     1. Ruby
+     1. Go
+     1. Python
+
+1. Author or review a Mattermost version update merge request. Ensures familiarity with its update
+   process and Distribution team's communication process with the Mattermost team.
+
+1. Author or review merge requests which update 5 other components.
+
+1. Author or review 3 merge requests which modify CI configuration.
+
+1. Author or review 3 merge requests which refactor build related code.
+
+1. **OPTIONAL**: Author or review 1 merge request which modifies the `omnibus` project. This
+   is an optional requirement , because updates to `omnibus` itself are
+   comparatively rare. It is highly recommended that the Maintainers look through
+   the commits we have [added on top of the upstream tag](https://gitlab.com/gitlab-org/omnibus/-/compare/9.0.19...9.0.19-stable).
+
+### Deploy scope
+
+TBA
+
+### Architectural changes
+
+Architectural changes to `omnibus-gitlab` project, as well as the `omnibus`
+project require sign-off from current full maintainers of the project.

doc/development/managing-postgresql-versions.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Managing PostgreSQL versions
+
+The PostgreSQL Global Development Group typically releases [one major version of PostgreSQL each year](https://www.postgresql.org/support/versioning/), usually in the third quarter. Our goal is to balance supporting and adopting new PostgreSQL features with the downtime and administrative costs of having our users upgrade. GitLab aims to support two versions of PostgreSQL at any given time. This means that prior to adding a new version of PostgreSQL, we will remove the oldest version of PostgreSQL that we support, and bump the minimally required PostgreSQL version up by one major version. PostgreSQL removals are only done in major GitLab releases.
+
+## Software definitions
+
+The software definitions are in:
+
+- `config/software/postgresql_old.rb`
+- `config/software/postgresql.rb`
+- `config/software/postgresql_new.rb`
+
+## Default version
+
+The version that should be installed by default is controlled by using the 'link bin files' step. The software definition with this step will be used on a new installation.
+
+## Upgrading
+
+The [`gitlab-ctl pg-upgrade` command](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-ctl-commands/pg-upgrade.rb) is used to upgrade from `postgresql_old` or `postgresql` to `postgresql_new`. See our [documentation for how to use it](../settings/database.md#upgrade-packaged-postgresql-server)
+
+### Automatic upgrades
+
+The [`gitlab-ctl pg-upgrade` command](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-ctl-commands/pg-upgrade.rb) is executed during every upgrade. To change the default PostgreSQL version used during the upgrade and revert processes, you must adjust the configuration to specify the desired PostgreSQL version that will be automatically upgraded during installation.
+
+## Removing an older version
+
+When it is time to remove an older version, create an epic with issues to track the following:
+
+1. Remove the old version from Omnibus.
+1. Remove the old version from the Helm install.
+1. To minimize CI costs, remove the older version from test suites (before doing this, check that .com is not using the old version still).
+1. Remove references to that version of PostgreSQL in the GitLab user documentation.
+1. Start printing deprecation notices in the release post three GitLab versions before a major PostgreSQL version is removed. If the PostgreSQL version is a version that will be removed within three releases, print deprecation notices in the Admin UI and during the GitLab upgrade process, regardless of whether it is an Omnibus-managed PostgreSQL database or an external database.
+
+For removals, perform the following steps:
+
+1. Run `git rm config/software/postgresql_old.rb`
+1. Run `git mv config/software/postgresql{,_old}.rb`
+1. Edit `config/software/postgresql_old.rb` and change name from `postgresql` to `postgresql_old`
+1. Run `git mv config/software/postgresql{_new,}.rb`
+1. Edit `config/software/postgresql.rb` and change name from `postgresql_new` to `postgresql`
+
+## Adding a new version
+
+1. Run `git cp config/software/postgresql{,_new}.rb`
+1. Edit `config/software/postgresql_new.rb`. Update:
+
+   1. `name` to `postgresql_new`
+   1. `default_version` to the new version
+   1. `version` to have the new version, and the `sha256`
+   1. `major_version` if necessary
+
+1. Add the new PostgreSQL version to the full test suite.
+1. Run a nightly test of the `gitlab-org/gitlab` repo against the new version of PostgreSQL.
+1. Ensure that the package build includes both versions of PostgreSQL.
+1. For Helm installs, update the default PostgreSQL chart version if the default is changing.
+1. Update user documentation.
+
+### Testing
+
+1. GitLab runs on the new version of PostgreSQL.
+1. Test running GitLab on the new PostgreSQL version at the 10k reference architecture scale and check for performance regressions.
+
+Test upgrades and fresh installs for the following environments:
+
+1. Single node.
+1. Install with a separate database node managed by Omnibus.
+1. HA database cluster with 3 or more database nodes in the cluster.
+1. Geo installations with a single node primary and single node secondary (`postgresql` and `geo-postgresql` on the same secondary node).
+1. Geo installations with a HA database cluster on the primary.
+1. Geo installations with a separate database and a separate tracking database on the secondary.
+1. Helm installs.
+1. After testing that upgrades to the newest version work, confirm that `revert-pg-upgrade` successfully downgrades to the previously used version, including on a Geo secondary standalone tracking database.
+1. If the default PostgreSQL version changes, test GitLab upgrades with external PostgreSQL databases.
+1. Back up and restore.
+If the default PostgreSQL version is changing:
+
+1. Auto upgrades on a single node install, separate database node, HA cluster.
+1. Auto upgrades where an external PostgreSQL database is being used.
+1. Geo installs are not auto upgraded.
+
+If the minimally required version is changing:
+
+1. GitLab upgrade errors out if an old version of Omnibus-managed PostgreSQL is still installed.
+
+If the above tests are manual, we risk missing a breaking change that is introduced after the manual tests have been performed. We should automate as many of these tests as possible.
+
+1. The package build includes both versions of PostgreSQL
+1. Running `gitlab-ctl pg-upgrade` works
+
+### The case of `libpq`
+
+Some modules, including `pyscopg2`, depend on PostgreSQL client library, i.e. `libpq`. It should be always linked to the
+latest bundled version. By using the latest version we rely on backward compatibility of `libpq`.
+
+## Known issues
+
+Geo uses streaming replication, which requires that the entire secondary database be resynced after a major PostgreSQL upgrade. This can cause hours or days of downtime, and as such, we do not recommend auto upgrades for Geo customers. Starting in 12.10, automatic PostgreSQL upgrades are disabled if Geo is detected.

doc/development/new-services.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Adding a new Service to Omnibus GitLab
+
+In order to add a new service to GitLab, you should follow these steps:
+
+1. [Fetch and compile the software during build](#fetch-and-compile-the-software-during-build)
+1. [Add a top-level configuration object for the service](#add-a-top-level-configuration-object-for-the-service)
+1. [Include the service in the services list](#include-the-service-in-the-services-list)
+1. [Create enable and disable recipes for the service](#create-enable-and-disable-recipes-for-the-service)
+1. [Determine and document how log rotation will be handled](#determine-and-document-how-log-rotation-will-be-handled)
+
+Optionally another common task is to add [additional configuration parsing](#additional-configuration-parsing-for-the-service)
+for the service.
+
+## Fetch and compile the software during build
+
+You need to add a [new Software Definition](new-software-definition.md) for your
+service if it is not already included in the project.
+
+## Add a top-level configuration object for the service
+
+The cookbooks and recipes located in `files/gitlab-cookbooks` are what get run during
+`gitlab-ctl reconfigure` in instances where the Omnibus GitLab package has been
+installed. This is where we will need to add the settings for a new service.
+
+### Define the default attributes
+
+Pick one of the existing cookbooks to configure your service within, or create a
+new cookbook if your service warrants its own.
+
+Within the cookbook there should be an `attributes/default.rb` file. This is where
+you want to define the [Default Attributes](../architecture/index.md#default-attributes)
+for your service. For a service you should define an `enable` option by default.
+
+```ruby
+default['gitlab']['best_service']['enable'] = false
+default['gitlab']['best_service']['dir'] = '/var/opt/gitlab/best-service'
+default['gitlab']['best_service']['log_directory'] = '/var/log/gitlab/best-service'
+```
+
+- `default` is how you define basic cookbook attributes.
+- `['gitlab']` contains the cookbook name.
+- `['best_service']` is the name of your service.
+- `enable`, `dir`, and `log_directory` are our configuration settings.
+- `/var/opt/gitlab` is where the working directory and configuration files for the services are placed.
+- `/var/log/gitlab` is where logs are written to for the GitLab package.
+
+Define all your settings that you want configurable in the package here. Default
+them to `nil` if you need to calculate their defaults based on other settings for
+now.
+
+#### Naming convention
+
+A service is referred to mainly in three scenarios:
+
+1. Accessing the Chef attributes corresponding to the service
+1. Referencing items such as users, groups, and paths corresponding to the
+   service
+1. Passing the service name to methods which look up on service properties
+   similar to the following examples:
+   - "Is the service enabled?"
+   - "Get the log ownership details corresponding to this service"
+   - "Generate runit configuration for this service"
+
+For the first case mentioned above, we use underscores to differentiate words in
+the service name. For the other two cases, we use hyphens to differentiate words
+in the service name. Since the configuration is mainly used as a Ruby object,
+using underscores instead of hyphens is more flexible (for example, underscores
+make it cleaner to use symbols in configuration hashes).
+
+For example, if we take GitLab Pages, the attributes are available as
+`Gitlab['gitlab_pages']` and `node['gitlab_pages']` while the default
+directories and paths might look like `/var/log/gitlab/gitlab-pages` and
+`/var/opt/gitlab/gitlab-pages`. Similarly, method calls will look like
+`service_enabled?("gitlab-pages")`.
+
+### Create a configuration Mash for your service
+
+In order for user to be able to configure your service from `/etc/gitlab/gitlab.rb`
+you will need to add a top level Mash for the service.
+
+In `files/gitlab-cookbooks/package/libraries/config/gitlab.rb` you will find the list of
+`attribute` methods.
+
+If your service exists within the attributes for the GitLab cookbook, you should
+add it as an attribute within the `attribute_block('gitlab')` block. Otherwise,
+if your service has its own cookbook, add it above.
+
+```ruby
+attribute('best_service')
+```
+
+For an EE only attribute, use `ee_attribute` instead.
+
+```ruby
+ee_attribute('best_service')
+```
+
+### Add service configuration to the settings template
+
+We maintain a [global configuration template](../architecture/index.md#global-gitlab-configuration-template)
+where examples of how to configure the services are available, commented out.
+
+This file becomes the `/etc/gitlab/gitlab.rb` on fresh installs of the package.
+
+Once you want to expose your service's configuration to users for them to change, add it
+to this file. `files/gitlab-config-template/gitlab.rb.template`
+
+```ruby
+### Best Service configuration
+# best_service['enable'] = true
+# best_service['dir'] = '/var/opt/gitlab/best-service'
+# best_service['log_directory'] = '/var/log/gitlab/best-service'
+```
+
+The values provided are not meant to reflect the defaults, but are to make it
+easier to uncomment to use the service. If that isn't possible you can use
+values clearly meant to be replaced like `YOURSECRET` etc. Or use the default
+when it makes the most sense.
+
+## Include the service in the services list
+
+In order to allow the service to be easily enable/disabled within the recipes, it
+should be added to the [services list](../architecture/index.md#services)
+and given appropriate groups.
+
+In the `files/gitlab-cookbooks/package/libraries/config/services.rb` file, add the
+service to the appropriate Config class, Base or EE depending on whether the
+service is only for GitLab EE.
+
+```ruby
+service 'best_service', groups: ['bestest']
+```
+
+Specifying groups makes it easier to disable/enable multiple related services as
+once.
+
+If none of the existing groups match with what your service does, and you don't
+currently need to enable/disable the service using a group. Don't bother adding
+at this time.
+
+Some examples of existing groups you may want to use:
+
+- If the service is enabled in omnibus be default, it should add the `DEFAULT_GROUP` group.
+- If the service should really not be disabled in almost any scenario, add the `SYSTEM_GROUP`.
+- If the service relies on GitLab Rails having been configured, add the `rails` group.
+- If the service is a new Prometheus exporter, add the `prometheus` group.
+
+## Create enable and disable recipes for the service
+
+### Enable recipe
+
+The enable recipe should be created as `files/gitlab-cookbooks/<cookbook-name>/recipes/<service-name>.rb`
+if it being added to an existing cookbook. If the service has its own cookbook,
+the enable recipe can be created as `files/gitlab-cookbooks/<cookbook-name>/recipes/enable.rb`.
+
+In the recipe you will want to create the working directory in `/var/opt/gitlab`
+for your service. You will want to ensure the system user that runs your service
+is created. Render any configuration files needed for your service into your working
+directory.
+
+Near the end of the recipe you will want to make a call to the runit service definition
+to define your recipe. In order for this work you will need to have created
+a run file in the cookbooks `templates/default` directory. These file names start
+with `sv-` followed by the service name, followed by the runit action name.
+
+A service typically needs a `run`, `log-run`, and `log-config`.
+
+`sv-best-service-log-config.erb`:
+
+```ruby
+<%= "s#@svlogd_size" if @svlogd_size %>
+<%= "n#@svlogd_num" if @svlogd_num %>
+<%= "t#@svlogd_timeout" if @svlogd_timeout %>
+<%= "!#@svlogd_filter" if @svlogd_filter %>
+<%= "u#@svlogd_udp" if @svlogd_udp %>
+<%= "p#@svlogd_prefix" if @svlogd_prefix %>
+```
+
+`sv-best-service-log-run.erb`:
+
+```ruby
+#!/bin/sh
+exec chpst -P \
+  -U root:<%= @options[:log_group] || 'root' %> \
+  -u root:<%= @options[:log_group] || 'root' %> \
+  svlogd -tt <%= @options[:log_directory] %>
+```
+
+`sv-best-service-run.erb`:
+
+```ruby
+#!/bin/sh
+exec 2>&1
+<%= render("mount_point_check.erb") %>
+cd <%= node['gitlab']['best-service']['dir'] %>
+exec chpst -P /opt/gitlab/embedded/bin/best-service -config-flags -etc
+```
+
+Depending on what you are running, and which user should run it, your run file
+should be configured differently. Look in our other `-run.erb` for examples.
+
+Within your recipe, the runit service should be called and started:
+
+```ruby
+runit_service "best-service" do
+  options({
+    configItem: 'value',
+    [...]
+    log_directory: logging_settings[:log_directory],
+    log_user: logging_settings[:runit_owner],
+    log_group: logging_settings[:runit_group],
+  }.merge(params))
+  log_options logging_settings[:options]
+end
+
+if node['gitlab']['bootstrap']['enable']
+  execute "/opt/gitlab/bin/gitlab-ctl start best-service" do
+    retries 20
+  end
+end
+```
+
+#### Log Directory
+
+The example settings referenced above that include `logging_settings` make use of
+the [`LogfilesHelper`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab/libraries/logfiles_helper.rb)
+class in order to provide a consistent reference to the configuration settings
+for the service log directory, the log group assigned to the log directory, and
+the group used for svlogd execution.
+
+To make use of these settings, please include the `LogfilesHelper` class in your
+`enable.rb` for your service, for example:
+
+```ruby
+[...]
+logfiles_helper = LogfilesHelper.new(node)
+logging_settings = logfiles_helper.logging_settings('best-service')
+[...]
+```
+
+Please add `best-service` to the list of services in the `default_logdir_ownership`
+class method with the default user/group that should be used for the log directory
+user/group. If you don't have a specific user/group need - default to
+`{ username: gitlab_user, group: gitlab_group }`
+
+### Disable recipe
+
+The disable recipe should be created as `files/gitlab-cookbooks/<cookbook-name>/recipes/<service-name>_disable.rb`
+if it being added to an existing cookbook. If the service has its own cookbook,
+the disable recipe can be created as `files/gitlab-cookbooks/<cookbook-name>/recipes/disable.rb`.
+
+The recipe needs to container any cleanup you want to do when you service is disabled,
+and have a call to disable the runit service.
+
+```ruby
+runit_service "best-service" do
+  action :disable
+end
+```
+
+## Determine and document how log rotation will be handled
+
+In Omnibus, [log rotation](https://docs.gitlab.com/ee/administration/logs/index.html#log-rotation) for
+a given service can be handled by `logrotate`, `svlogd`, both or neither. The new service should
+be included in the [log rotation](https://docs.gitlab.com/ee/administration/logs/index.html#log-rotation)
+table with an indication about what is responsible for managing and rotating the logs for that
+service. When adding a service to Omnibus GitLab, you should:
+
+- Ensure that log rotation is in place for the new service.
+- Open a merge request to have the new service added to
+  [the log rotation table](https://docs.gitlab.com/ee/administration/logs/index.html#log-rotation).
+
+If a new log is added that is not using `runit` (`svlogd`), the log must be manually
+added to the logrotate configuration. The
+[Improve logrotate handling](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6182) issue
+has more information.
+
+## Additional configuration parsing for the service
+
+If you want to populate certain configuration options based on other options being set
+by the user, we add a library for your service to parse variables.
+
+The library should be added as `files/gitlab-cookbooks/<cookbook name>/libraries/<service-name>.rb`
+
+The library should be a module named after your service that has a `parse_variables` method.
+
+```ruby
+module BestService
+  class << self
+    def parse_variables
+      # setup some additional configuration based on the values of the user provided configuration
+    end
+  end
+end
+```
+
+We then need to have the GitLab configuration call your parse_variables method.
+
+Go into `files/gitlab-cookbooks/package/libraries/config/gitlab.rb` and update
+your attribute to use the library.
+
+```ruby
+attribute('best_service').use { BestService }
+```
+
+Note that sequence for parsing variables matters. So if your library expects to
+be parsed after another service's library, you need to update your attribute with
+a `priority` value that comes later. (The default `priority` value is `20`)
+
+```ruby
+attribute('expected_service').use { ExpectedService }
+attribute('best_service', sequence: 25).use { BestService }
+```

doc/development/new-software-definition.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Adding a new Software Definition to Omnibus GitLab
+
+In order to add a new component to GitLab, you should follow these steps:
+
+1. [Fetch and compile the software during build](#fetch-and-compile-the-software-during-build)
+1. [Add a dependency for the software definition to another component](#add-a-dependency-for-the-software-definition-to-another-component)
+
+## Fetch and compile the software during build
+
+[Software Definitions](../architecture/index.md#software-definitions), which
+can be found in `/config/software`, specify where omnibus should fetch the
+software, how to compile it and install it to the required folder. This part of
+the project is run when we build the Omnibus package for GitLab.
+
+When adding a component that should be fetched from Git the clone address of the
+repositories of the local mirror and upstream should be added to
+`/.custom_sources.yml`.
+
+The local mirror should be created in the [omnibus-mirror project](omnibus-mirror.md) by a member of the Distribution team.
+
+See other Software services in the directory for examples on how to include your
+software service.
+
+## Handling Licenses
+
+Most software repositories include a license file. Add the license using a patch
+file if it is not explicitly included. Software installed using a package manager
+such as `gem` or `pip` should also use this method.
+
+[Create patches](creating-patches.md) for licenses added manually and store them
+at a directory path with the naming convention
+`config/patches/SOFTWARE_NAME/license/VERSION_NUMBER/add-license-file.patch`.
+
+> Licenses can and do change over the lifetime of a project. This method
+> intentionally causes builds to fail reminding contributors to verify manually
+> installed licenses. If the license has not changed then `git mv` the `VERSION_NUMBER`
+> directory containing the patch file to the new `VERSION_NUMBER`.
+
+## Add a dependency for the software definition to another component
+
+Add a `dependency` statement to the definition of the GitLab project found in
+`/config/projects/gitlab.rb`, unless there is a more specific component it makes
+sense to be a dependency of (eg `config/software/gitlab-rails.rb` for a
+component only needed by `gitlab-rails`)
+
+Omnibus will build dependency components first, and then other ones in the order
+of their presence in `/config/projects/gitlab.rb`. So, when a software component
+A is marked as a dependency of another software B, A will be built towards the
+beginning of the process. In cases where A is a component that changes frequently, cache gets invalidated often causing every subsequent component to be
+rebuilt, increasing overall build time. A workaround is to ensure A gets built
+immediately before B, avoiding cache invalidation.
+
+1. Add the software A to `/config/projects/gitlab.rb` immediately before
+   software B
+
+   Since A and B are now top-level dependencies of the project, omnibus will
+   build them in the order of their presence in `/config/projects/gitlab.rb`.
+
+1. In the software definition of B, add a line similar to the following
+
+   ```ruby
+   dependency '<name of software A>' unless project.dependencies.include?('<name of software A>')
+   ```
+
+   Above ensures that whenever the project is not built from
+   `/config/projects/gitlab.rb`, A is marked as a dependency of B and is built
+   before B. There will be no effect on builds from `/config/projects/gitlab.rb`
+   however.
+
+## Validating changes to a single software dependency
+
+It can be useful to only build one piece of software, rather than rebuild the whole package each time. For instance,
+when adding a new software definition. Using this method, you can quickly rebuild an omnibus package containing only
+the software and its dependencies. Once you've confirmed the software builds on its own, you can add it to the Omnibus GitLab
+build and confirm it there. To use this:
+
+1. [Setup your development environment](setup.md)
+1. Copy the [simple.rb](examples/simple.rb) file into your projects
+
+   ```shell
+   cp doc/development/examples/simple.rb config/projects/
+   ```
+
+1. Change the `dependency` in `config/projects/simple.rb` to match the software you are testing
+1. Build the simple project by running
+
+   ```shell
+   bundle exec omnibus build simple
+   ```

doc/development/omnibus-mirror.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# omnibus-mirror
+
+In order to minimize the dependency on external resources during our build and release process, we maintain a mirror of various software dependencies within GitLab resources.
+
+The system consists of two groups, one on GitLab.com, and the other on `dev.gitlab.org`
+
+1. [GitLab.com projects](https://gitlab.com/gitlab-org/build/omnibus-mirror)
+
+   1. Pull mirrors of the upstream source, and push mirrors to the corresponding `dev.gitlab.org` project.
+   1. Publically available. Should be available to the `trigger-package` pipeline, as well as community developers for building custom `omnibus-gitlab` packages
+
+1. `dev.gitlab.org` projects: `https://dev.gitlab.org/omnibus-mirror`
+
+   1. Pipelines on `dev.gitlab.org` use these projects to build `omnibus-gitlab` projects. This includes the package releases, as well as builds used by GitLab Team members to build custom `omnibus-gitlab` packages for all supported platforms.
+
+## Adding a project
+
+1. Log into GitLab.com as the `gitlab-omnibus-mirror-bot`. Login details are in the Build vault in 1Password
+1. Click on **New Project** to create a project
+   1. **Project name**: usually should match the upstream name
+   1. **Project URL**: Select `gitlab-org/build/omnibus-mirror` from the **Groups** sub-group in the drop-down
+   1. **Visibility Level**: Set this to **Public**
+   1. Disable any options that would create files, such as initializing with a `README.md`
+   1. Leave the remaining options as their default, and click on **Create project**
+1. Set up the pull mirror
+   1. Click **Settings -> Repository** in the left hand menu
+   1. Click **Expand** next to the **Mirroring repositories** option
+   1. Set **Git repository URL** to the upstream URL we'll be mirroring
+   1. Change **Mirror direction** to **Pull**
+   1. Select the **Overwrite diverged branches** option
+   1. Click **Mirror repository** to add the pull mirror. Depending on the size of the repository, this can take a few minutes to run
+   1. Leave this page open as you will need it later
+1. Log into `dev.gitlab.org` as the `build_mirror_bot`. Login details are in the Build vault in 1Password
+1. Click on **New Project** to create a project
+   1. **Project name**: usually should match the upstream name
+   1. **Project URL**: Select `omnibus-mirror` from the **Groups** sub-group in the drop-down
+   1. **Visibility Level**: Set this to **Private**
+   1. Leave the remaining options as their default and click on **Create project**
+1. Enable the `omnibus-builder` deploy key
+   1. Navigate to **Settings -> Repository**
+   1. Click on **Expand** next to the **Deploy Keys** section
+   1. Click the **Privately accessible deploy keys** tab
+   1. Find the `omnibus-builder deploy key` and click on the **Enable** button next to it
+1. Setup the push mirror
+   1. Return to the **Mirroring repositories** page for the GitLab.com project
+   1. In **Git repository URL** enter the path to the `dev.gitlab.org` with `build_mirror_bot@` inserted between `https://` and `dev.gitlab.org`
+   1. In **Password** enter the `PAT` from the **Build Team Mirror Bot** entry in the **Build** vault in 1 Password
+   1. Disable any options that would create files, such as initializing with a `README.md`
+   1. Leave the remaining options as their default, and click on **Mirror repository**
+   1. Click the **Update Now** button to trigger initial mirroring

doc/development/package_version_format.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Version format for the packages and Docker images
+
+## Packages
+
+Packages built by the `omnibus-gitlab` continuous integration pipelines have
+a version string in the format `<build_version>-<build_iteration>`. The
+pipelines generally produce three types of packages:
+
+- feature branch builds
+- nightly builds
+- tagged release builds
+
+NOTE:
+The `build_iteration` portion of versions strings conveys a specific meaning
+and contributes to how `build_version` is computed. The next sections are
+written with that in mind as `build_iteration` must be understood first even
+though it comes last in the version string.
+
+### Build Iteration
+
+Version strings use `build_iteration` when packaging related logic changes
+that do not contain changes to bundled components.
+
+|Release Type|Example Pipelines|`build_iteration` String|
+|-|-|-|
+|non-tagged|feature branches, nightly builds|`0`|
+|tagged|releases|`ce.<OMNIBUS_RELEASE>` or `ee.<OMNIBUS_RELEASE>` |
+
+The edition component, **ce** or **ee**, instructs package managers such as
+`apt` or `yum` to treat Enteprise Edition packages as an upgrade from
+Community Edition when the package version is the same because **ee** is
+sorted lexographically after **ce**.
+
+The `OMNIBUS_RELEASE` component is deprecated and always set to `0`.
+Historically, `OMNIBUS_RELEASE` indicated quick bug fixes targeting
+`omnibus-gitlab` with no changes in GitLab Rails or other bundled
+components. The updates frequently required Semantic Version updates because
+of user impact. For this reason, `OMNIBUS_RELEASE` was not useful in
+practice and is no longer relevant.
+
+### Build version
+
+The build version component of the version string changes based on whether
+the package is a [a feature branch](#regular-feature-branch-builds),
+[a nightly build](#nightly-builds), or a [tagged release](#tagged-release-builds).
+
+#### Regular feature branch builds
+
+For regular feature branch builds, the version format is
+`<latest stable git tag>+rfbranch.<pipeline id>.<omnibus-gitlab SHA>-<build iteration>`.
+
+As noted above, `build iteration` for regular feature branch builds is set to
+`0`. An example version string of this type is
+`13.1.1+rfbranch.159743.eb538eaf-0`.
+
+The `+rfbranch` string denotes the package as built from a regular feature
+branch build. It also lexographically places it after a stable branch
+causing package managers such as `apt` or `yum` to view it as an upgrade
+from a stable release package.
+
+#### Nightly builds
+
+For nightly packages , the version format is
+`<latest stable git tag>+rnightly.<pipeline id>.<omnibus-gitlab SHA>-<build iteration>`.
+
+As noted above, `build iteration` for nightly builds is set to `0`. An example
+version string of this type is `13.1.1+rnightly.159756.b2b5f05e-0`.
+
+The `+rnightly` denotes the package as the output from a nightly build. When
+compared alphabetically by package managers, `+rnightly` is considered
+greater than both the latest stable and the `+rfbranch` packages. Package
+managers will always treat a nightly package as a package upgrade.
+
+#### Tagged release builds
+
+For tagged release builds, while the Git tags are of the format
+`<SemVer version>+<build iteration>`, the version strings follow the format of
+`<SemVer version>-<build iteration>`.
+
+For example, if the tags are `13.1.0+rc42.ce.0`, `13.1.0+ce.0`, and
+`13.1.0+ee.0`, the version strings will be `13.1.0-rc42.ee.0`, `13.1.0-ce.0`,
+and `13.1.0-ee.0` respectively.
+
+As noted above, unlike feature branch and nightly builds, the `build iteration`
+component of these releases are of the format `(ce|ee).0`
+
+## Docker images
+
+Docker images created by `omnibus-gitlab` CI pipelines are based on the Ubuntu
+package built in the previous stage. Hence, the Docker image tag also reflects
+the same information given by the package version string. Because `+` symbol
+used in package versions is not a supported character for image tags, it is
+replaced with a `-` to get a slug.
+
+As a general rule, Docker images will use a slug of the package version as tags.
+Also, all Docker images will be pushed to Docker container registries
+corresponding to the host where they are being built.
+
+The entire image reference will be of the form
+
+```plaintext
+dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-(ce|ee):<slug of package version>
+```
+
+For example,
+`dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ce:13.2.1-rfbranch.163015.32ed1c58-0`
+will be the image tag for a Docker built from a GitLab CE package with version
+`13.2.1+rfbranch.163015.32ed1c58-0`
+
+The special cases that deviate from this general rule is listed below.
+
+### Triggered builds in [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror)
+
+Docker images created as part of triggered pipeline (for Package and QA run) has
+their tag decided depending on where the trigger originated. If the trigger
+originated from an `omnibus-gitlab` pipeline, the image tag is the slug of the
+package version. But, if the trigger originated from a GitLab or GitLab-FOSS
+pipeline, the image tag will be set to the SHA of the commit corresponding to
+that pipeline.
+
+These jobs are not published to Dockerhub repository.
+
+### Nightly builds
+
+In addition to the general naming scheme, Docker images built by scheduled
+nightly pipelines gets tagged with the `nightly` tag and both tags are pushed to
+the Dockerhub repository also.
+
+`gitlab/gitlab-ee:nightly` and `gitlab/gitlab-ce:nightly` denote image
+references for the two editions available.
+
+## Tagged release builds
+
+In addition to the general naming scheme, Docker images built by tagged release
+pipelines also gets tagged as `latest` and both tags are pushed to the Dockerhub
+repository also.

doc/development/pipelines.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Repositories, branches, and CI pipelines
+
+`omnibus-gitlab` CI pipeline is a moderately complex one with pipelines split
+between all the mirrors of the project:
+
+1. [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) for normal feature development.
+1. [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) for building release artifacts.
+1. [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) for security development.
+1. [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) for developers to run a package build and QA for development
+   purposes.
+
+## Branch and tag protection
+
+### Protected branches
+
+NOTE:
+Unless specified otherwise, the listed users/groups have permission to both merge, and push to
+protected branches.
+
+1. Development repository
+    1. `master`: Maintainers, [Delivery team](https://gitlab.com/gitlab-org/delivery)
+    1. `*-stable` : [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
+    1. `*-stable-ee` : [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
+    1. `*-auto-deploy-*` : Maintainers, [`delivery` group](https://gitlab.com/gitlab-org/delivery), [`managers` group](https://gitlab.com/gitlab-org/release/managers)
+
+1. Release mirror:
+    1. `master`: Maintainers
+    1. `*-stable` : Maintainers
+    1. `*-stable-ee` : Maintainers
+    1. `*-auto-deploy-*` : Maintainers
+
+1. Security mirror:
+    1. `master`: [GitLab Release Tools Bot](https://gitlab.com/gitlab-release-tools-bot), [GitLab Bot](https://gitlab.com/gitlab-bot), [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
+    1. `*-stable`: [GitLab Release Tools Bot](https://gitlab.com/gitlab-release-tools-bot), [GitLab Bot](https://gitlab.com/gitlab-bot), [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
+    1. `*-stable-ee`: [GitLab Release Tools Bot](https://gitlab.com/gitlab-release-tools-bot), [GitLab Bot](https://gitlab.com/gitlab-bot), [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
+    1. `*-auto-deploy-*`: [GitLab Release Tools Bot](https://gitlab.com/gitlab-release-tools-bot), [GitLab Bot](https://gitlab.com/gitlab-bot), [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
+
+1. QA mirror:
+    1. `master`: Developers (Merge only), Maintainers
+
+NOTE:
+Developers get access to `master` branch in QA mirror because
+that's required to run a triggered pipeline against the branch. There is an
+[open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24585) to change this
+situation.
+
+### Protected Tags
+
+NOTE:
+Unless specified otherwise, the listed users/groups have permission to both merge, and push to
+protected tags.
+
+1. Development repository:
+    1. `*` : Maintainers, [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
+
+1. Release mirror:
+    1. `*`: Maintainers
+
+1. Security mirror: Nil
+
+1. QA mirror: Nil
+
+## Mirroring between repositories
+
+Most of the development happens in the [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab)
+and security related changes go to [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab).
+These changes then gets mirrored to [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab)
+and [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror). The
+following diagram describes what all gets mirrored between each of these
+repositories.
+
+```mermaid
+graph TD
+  A[Development repository] -->|Protected branches| B[Security mirror]
+  A[Development repository] -->|Protected branches| C[Release mirror]
+  A[Development repository] -->|All branches| D[QA mirror]
+  B[Security mirror] -->|Protected branches| C[Release mirror]
+  B[Security mirror] -->|All branches| D[QA mirror]
+```
+
+## Types of pipelines
+
+Since the jobs in `omnibus-gitlab` are spread across multiple mirrors, the CI
+config becomes relatively complex when using `rules` keyword for specifying
+where/when to run jobs. Hence, a different approach is used for
+`omnibus-gitlab`, which marks each necessary pipeline type with a specific
+variable named `PIPELINE_TYPE`, and attach jobs to various pipeline types as
+needed. The different pipeline types are documented in the below table:
+
+| Pipeline type                    | Mirror(s) where the pipeline runs | Remarks                                                                                                                |
+| -------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
+| `DEPENDENCY_SCANNING_PIPELINE`   | Canonical                         | Check for security vulnerabilities in dependencies. Requires `DEPENDENCY_SCANNING` variable to be set to `true`.       |
+| `DEPS_IO_VERSION_BUMP_PIPELINE`  | Canonical                         | On branch pushes done by `deps.io`. Requires branches to have `deps-` prefix.                                          |
+| `DEPS_IO_VERSION_CHECK_PIPELINE` | Canonical                         | Run `deps` to detect updates. Requires `DEPS_PIPELINE` variable to be set to `true`.                                   |
+| `LICENSE_PAGE_UPDATE_PIPELINE`   | Canonical                         | Update license pages. Requires `PAGES_UPDATE` variable to be set to `true`.                                            |
+| `CACHE_UPDATE_PIPELINE`          | Canonical, QA                     | Update the gem cache and package build cache. Requires `CACHE_UPDATE` variable to be set to `true`.                    |
+| `DURATION_PLOTTER_PIPELINE`      | QA                                | Package build to plot duration of builds. Requires `DURATION_PLOTTER` variable to be set to `true`.                    |
+| `DOCS_PIPELINE`                  | Canonical, Security               | Requires branch names to either have `docs-` as prefix or `-docs` as suffix.                                           |
+| `PROTECTED_TEST_PIPELINE`        | Canonical, Security               | On protected branches and tags. Does not include trigger jobs and other unwanted jobs like danger-review.              |
+| `GITLAB_BRANCH_TEST_PIPELINE`    | Canonical, Security               | On non-protected branch pushes. Does not include unwanted jobs like danger-review. Includes trigger jobs.              |
+| `GITLAB_MR_PIPELINE`             | Canonical, Security               | On MRs with branches in Canonical/Security (i.e. MRs from GitLab and team members). Has trigger jobs.                  |
+| `AUTO_DEPLOY_BUILD_PIPELINE`     | Release                           | Builds when auto-deploy tags are pushed.                                                                               |
+| `CE_BRANCH_BUILD_PIPELINE`       | Release                           | Builds when regular branches are pushed.                                                                               |
+| `CE_NIGHTLY_BUILD_PIPELINE`      | Release                           | Nightly CE builds.                                                                                                     |
+| `CE_RC_BUILD_PIPELINE`           | Release                           | Builds when CE RC tags are pushed.                                                                                     |
+| `CE_TAG_BUILD_PIPELINE`          | Release                           | Builds when stable CE tags are pushed.                                                                                 |
+| `EE_BRANCH_BUILD_PIPELINE`       | Release                           | Builds when regular branches are pushed, but forced to be an EE pipeline.                                              |
+| `EE_NIGHTLY_BUILD_PIPELINE`      | Release                           | Nightly EE builds. Requires pipeline to be forced to be an EE one.                                                     |
+| `EE_RC_BUILD_PIPELINE`           | Release                           | Builds when EE RC tags are pushed.                                                                                     |
+| `EE_TAG_BUILD_PIPELINE`          | Release                           | Builds when stable EE tags are pushed.                                                                                 |
+| `TRIGGER_CACHE_UPDATE_PIPELINE`  | QA                                | Updates build cache of triggered QA pipelines. Requires `CACHE_UPDATE` variable to be set to `true`.                   |
+| `TRIGGERED_CE_PIPELINE`          | QA                                | Triggered e2e:package-and-test build with CE packages and images.                                                            |
+| `TRIGGERED_EE_PIPELINE`          | QA                                | Triggered e2e:package-and-test build with EE packages and images.                                                            |
+| `FORK_BRANCH_TEST_PIPELINE`      | Forks                             | Test suite when run on forks of the project. Does not include trigger jobs and other unwanted jobs like danger-review. |
+| `FORK_MR_PIPELINE`               | Forks                             | On MRs from forks of the project. Does not include trigger jobs.                                                       |
+
+### Branch pipelines
+
+`omnibus-gitlab` doesn't make use of detached merge request pipelines yet. So,
+pipelines caused by branch pushes are the common pipelines encountered in this
+project. These pipelines are used in [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab), [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab),
+and [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab).
+
+### Tag pipelines
+
+Tag pushes to [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) starts a pipeline with the
+[release jobs](#release-jobs). Tag pushes to [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and
+[Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) behaves as regular branch pushes (except that they don't have an option
+to start a e2e:package-and-test pipeline) and run basic stylistic checks and unit
+tests.
+
+### Scheduled pipelines
+
+In [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab), there are two scheduled pipelines.
+
+1. `Dependencies Update` - Pipeline to check for outdated dependencies using the
+   [`dependency_update`](#dependency_update) job.
+1. `Generate license pages` - Pipeline to populate [License collection webpage](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html)
+   with license information from the S3 bucket.
+
+In [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab), there are two scheduled pipelines
+
+1. `CE nightly` - Pipeline to build nightly packages and Docker images for
+   GitLab CE
+1. `EE nightly` - Pipeline to build nightly packages and Docker images for
+   GitLab EE
+
+Other mirrors doesn't have any scheduled pipelines
+
+### Auto-deploy pipelines
+
+GitLab uses [Auto deploy](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/auto-deploy.md) branches and tags for its release process. These
+branches are named `<MAJOR>-<MINOR>-auto-deploy-<TIMESTAMP>` and tags have the
+format `<MAJOR>.<MINOR>.<TIMESTAMP>+<gitlab sha>.<omnibus-gitlab sha>`.
+
+Only certain jobs - that are needed to deploy the changes to GitLab
+environments - will be part of this pipeline.
+
+### Triggered pipelines
+
+We use triggered pipelines to run a "e2e:package-and-test" pipeline in the
+[QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror). These can be triggered either from a pipeline in the
+[Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) or [GitLab project](https://gitlab.com/gitlab-org/gitlab).
+
+This pipeline is intended to give the developers a package and an image to test
+their changes in addition to automatically doing a QA run against these
+artifacts. It also provides an option to do a QA run against an HA instance spun
+up using these artifacts.
+
+## CI Jobs
+
+### Development Jobs
+
+#### `danger-review`
+
+This job runs a check against the merge request to ensure it satisfies some
+basic requirements using the Danger tool.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
+and tag pipelines.
+
+#### `rubocop`
+
+Check the source code files to satisfy certain stylistic requirements.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
+and tag pipelines.
+
+#### `docs-lint`
+
+Check the docs files to satisfy certain stylistic requirements.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
+and tag pipelines.
+
+#### `review-docs-deploy`
+
+A manual job that will trigger a docs build in `gitlab-docs`, which will deploy a
+review app for the GitLab Docs with changes from the current commit.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) on branch pipelines.
+
+#### `review-docs-cleanup`
+
+A manual job to stop the environment created by `review-docs-deploy`.
+Automatically ran on merging a merge request.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) on branch pipelines.
+
+#### `<OS_NAME> knapsack`
+
+We have tests covering our Chef recipes and libraries using `RSpec` and
+`Chefspec`. They are parallelized with the help of
+[knapsack](https://github.com/KnapsackPro/knapsack).
+
+This job pulls in the knapsack reports of previous pipeline runs from the cache
+and gets them in place for the current spec runs.
+
+These jobs are run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
+and tag pipelines.
+
+#### `<OS_NAME> specs`
+
+Jobs that actually run `rspec` via `knapsack`. Parallelized to a count of 6
+using the [`parallel` keyword](https://docs.gitlab.com/ee/ci/yaml/#parallel)
+
+These jobs are run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
+and tag pipelines.
+
+#### `update-knapsack`
+
+This job combines the reports from each individual parallel runs of unit tests
+and prepares a final JSON report. This report is used by the MR widget to
+display spec status, and is also uploaded to the cache for the next pipeline run
+to use.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch and
+tag pipelines.
+
+#### `Trigger:ce-package`
+
+This is a manual job which when played triggers a pipeline in the [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) to
+run a package build and QA for development purposes. This can be used by
+developers to get a package or Docker image for testing, or to run a full QA
+suite against their MR's changes.
+
+To debug QA failures, refer to the [Investigate QA failures](https://handbook.gitlab.com/handbook/engineering/infrastructure/test-platform/self-managed-platform-team/distribution/#investigate-qa-failures)
+section. This job also generates Allure report, see more information and the demo at [Test Report Generation in Omnibus-GitLab](allure-test-report.md).
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
+pipelines.
+
+#### `Trigger:ee-package`
+
+Same as `Trigger:ce-package`, but builds an EE package.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
+pipelines.
+
+#### `fetch-assets`
+
+For building packages, we make use of the Rails assets that are already compiled
+by the [GitLab](https://gitlab.com/gitlab-org/gitlab) or
+[GitLab-FOSS](https://gitlab.com/gitlab-org/gitlab-foss) pipelines. Those
+pipelines push it as a Docker image, which we pull in here and copy the assets
+themselves to a predefined location.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on branch, tag and
+triggered pipelines.
+
+#### `Trigger:package`
+
+This job builds a single package that will be available as an artifact.
+
+This job is run only on [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on triggered pipelines.
+
+#### `Trigger:gitlab-docker`
+
+This job builds a GitLab Docker image using the package that was built by
+`Trigger:package` job.
+
+This job is run only on [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on triggered pipelines.
+
+#### `qa-subset-test`
+
+This job runs subset of test suite and is automatically triggered when either
+ `Trigger:CE-package` or `Trigger:EE-package` manual jobs are triggered manually.
+
+This job triggers a pipeline in the [GitLab QA Mirror](https://gitlab.com/gitlab-org/gitlab-qa-mirror), passing the GitLab Docker image created by `Trigger:gitlab-docker` job and the GitLab QA Docker image built by the GitLab Rails pipeline, so that a subset of test as mentioned in [this issue](https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1303#we-should-keep) will run using these images
+
+This job is run only on [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on triggered pipelines.
+
+### `qa-remaining-test-manual`
+
+This is a manual trigger job which runs the remaining tests that donot run in the `qa-subset-test` job.
+
+To run this QA jobs in MR pipeline, you need to either trigger `Trigger:CE-package` or `Trigger:EE-package` manual jobs.
+
+This job triggers a pipeline in the [GitLab QA Mirror](https://gitlab.com/gitlab-org/gitlab-qa-mirror), passing the GitLab
+Docker image created by `Trigger:gitlab-docker` job and the GitLab QA Docker
+image built by the GitLab Rails pipeline, so that the entire suite is run using these images.
+
+#### `RAT`
+
+This manual job triggers a pipeline in the
+[RAT](https://gitlab.com/gitlab-org/distribution/reference-architecture-tester)
+project passing the URL to the package built by `Trigger:package` job, which
+will spin up a PostgreSQL HA instance with that package using
+[GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit), and run
+QA against that instance.
+
+This job is run only on [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on triggered EE pipelines.
+
+#### `<OS_NAME>-branch`
+
+These jobs build packages for the specified OS, and pushes the resultant package
+to an S3 bucket in addition to making them available as artifacts.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on branch and nightly pipelines.
+
+`NOTE`: Raspberry Pi jobs are run only on CE branches while SLES jobs are run
+only on EE branches.
+
+#### `Docker-branch`
+
+This job builds a GitLab Docker image using the package built during the Ubuntu
+22.04-branch job. The image is pushed to the GitLab container registry.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on branch and nightly pipelines.
+
+#### `QA-branch`
+
+This job builds a GitLab QA Docker image from the
+[qa directory in the Rails codebase](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/Dockerfile).
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on branch and nightly pipelines.
+
+## Release jobs
+
+### `<OS_NAME>`
+
+These jobs build packages for the specified OS, and pushes the resultant package
+to an S3 bucket in addition to making them available as artifacts.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on tag pipelines.
+
+NOTE:
+Raspberry Pi jobs are run only on CE tags while SLES jobs are run only on EE tags.
+
+### `<OS_NAME>-staging`
+
+These jobs upload the packages built in previous job to an internal staging
+repository in [Packagecloud instance](https://packages.gitlab.com).
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on tag pipelines.
+
+NOTE:
+Raspberry Pi jobs are run only on CE tags while SLES jobs are run only on EE tags.
+
+### `<OS_NAME>-release`
+
+These jobs pulls the packages from the internal staging repository to a public
+repository in [Packagecloud instance](https://packages.gitlab.com).
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on tag pipelines.
+
+NOTE:
+Raspberry Pi jobs are run only on CE tags while SLES jobs are run only on EE tags.
+
+### `Docker`
+
+This job builds a GitLab Docker image using the package built during the Ubuntu
+22.04-branch job. The image is pushed to the internal GitLab container registry.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on tag pipelines.
+
+### `Docker-Release`
+
+This job pulls the GitLab Docker image from internal GitLab container registry
+and pushes it to Dockerhub.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on tag pipelines.
+
+### `Docker-QA`
+
+This job builds a GitLab QA Docker image from the
+[qa directory in the Rails codebase](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/Dockerfile) and
+pushes it to the internal GitLab container registry.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on tag pipelines.
+
+### `QA-Tag`
+
+This job pulls the GitLab QA Docker image from the internal GitLab container
+registry and pushes it to Dockerhub.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on tag pipelines.
+
+### `AWS`
+
+This job builds the unlicensed AWS AMIs using the Ubuntu 20.04 package.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on tag pipelines.
+
+### `AWS-Ultimate`
+
+This job builds an AWS AMI with embedded Ultimate licenses using the Ubuntu
+20.04 package.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on EE tag pipelines.
+
+### `AWS-Premium`
+
+This job builds an AWS AMI with embedded Premium licenses using the Ubuntu
+20.04 package.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on EE tag pipelines.
+
+### `AWS-CE-Release`
+
+This job updates the [GitLab Community Edition listing in AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-w6ykryurkesjq?ref_=srh_res_product_title) with the current version.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on CE tag pipelines.
+
+### `AWS-Ultimate-Release`
+
+This job updates the [GitLab Ultimate listing in AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-g6ktjmpuc33zk?ref_=srh_res_product_title) with the current version.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on EE tag pipelines.
+
+### `AWS-Premium-Release`
+
+This job updates the [GitLab Premium listing in AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-amk6tacbois2k?ref_=srh_res_product_title) with the current version.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on EE tag pipelines.
+
+### `create_omnibus_manifest`
+
+This job creates a `version-manifest.json` file to be used by
+`dependency_scanning` job.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on tag and nightly pipelines.
+
+### `dependency_scanning`
+
+This job runs a dependency scan on all the components of the package to check if
+any of them are vulnerable to any known vulnerabilities.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on tag and nightly pipelines.
+
+### `RAT-Nightly`
+
+This job triggers a pipeline in the
+[RAT](https://gitlab.com/gitlab-org/distribution/reference-architecture-tester)
+project passing the URL to the nightly Ubuntu 22.04 package built in this pipeline,
+which will spin up a PostgreSQL HA instance with that package using
+[GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit), and run
+QA against that instance.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on nightly pipelines.
+
+### `RAT-Tag`
+
+This job triggers a pipeline in the
+[RAT](https://gitlab.com/gitlab-org/distribution/reference-architecture-tester)
+project passing the URL to the Ubuntu 22.04 package built in this pipeline,
+which will spin up a PostgreSQL HA instance with that package using
+[GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit), and run
+QA against that instance.
+
+This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on EE tag pipelines.
+
+### `license-upload`
+
+This job compiles the license information of all the dependencies from the
+package and uploads it to the S3 bucket. This is used by
+[`pages`](#pages) scheduled job in [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) to populate the
+[License collection webpage](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html).
+
+## Housekeeping Jobs
+
+### `update-gems-cache` and `update-trigger-package-cache`
+
+Jobs that extends from the `.gems-cache` and `.trigger-package-cache` shared `cache` definitions only `pull` their cache.
+
+These caches are updated in scheduled pipelines by `update-gems-cache` and `update-trigger-package-cache` jobs respectively, when the `CACHE_UPDATE` exists.
+
+### `dependency_update`
+
+This job makes use of Dependencies.io to automatically check for version updates
+of various components that we include in our package and open merge request
+against [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) if an update is found.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab), on scheduled pipelines when
+`DEPS_PIPELINE` variable exists.
+
+### `dependencies_io_check`
+
+This job automatically triggers a e2e:package-and-test pipeline in [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror),
+(similar to `Trigger:ce-package` job), when a merge request is made by
+`dependency_update` job.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) when the branch name starts with
+`deps` (this is the format that `dependency_update` job uses for merge requests)
+
+### `valdiate_packer_changes`
+
+This job checks if the packer configuration files: `https://dev.gitlab.org/gitlab/omnibus-gitlab/-/tree/master/support/packer` are valid or not.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and Security mirror: `https://gitlab.com/gitlab-org/security/omnibus-gitlab`, when any of
+the packer configuration files: `https://dev.gitlab.org/gitlab/omnibus-gitlab/-/tree/master/support/packer` are modified.
+
+### `pages`
+
+This job is associated with GitLab Pages, and generates a static website
+containing license information of various components included in the package for
+each release of GitLab.
+
+This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and on scheduled pipelines which
+does not have the `DEPS_PIPELINE` variable (to differentiate it from
+`dependency_update` runs).

doc/development/public-attributes.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Working with `public_attributes.json`
+
+Chef stores a copy of a nodes attributes at the end of a reconfigure in `/opt/gitlab/embedded/nodes/$NODE_NAME.json`. Due to the sensitive nature of some of the attributes, it is only readable by the root user. To work around this, we've created a file (defaults to `/var/opt/gitlab/public_attributes.json`) which contains a set of attributes we've whitelisted for use of non-root services. This file is recreated on every run of `gitlab-ctl reconfigure`.
+
+## Adding an entry to `public_attributes.json`
+
+The `public_attributes.json` file is populated by merging the results of a call to the `public_attributes` method of a helper class. For example:
+
+```ruby
+class TestHelper < BaseHelper
+  attr_accessor :node
+
+  def public_attributes
+    {
+      'gitlab' => {
+        'test' => node['gitlab']['test']
+      }
+    }
+  end
+end
+```
+
+The file is generated as part of the [GitLabHandler](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/package/libraries/handlers/gitlab.rb#L36).
+
+## Reading an entry from `public_attributes.json` from a `gitlab-ctl` command
+
+In order to access the public nodes, you should use the provided [`GitlabCtl::Util.get_public_node_attributes` method](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-ctl-commands/lib/gitlab_ctl/util.rb#L60)
+
+```ruby
+attributes = GitlabCtl::Util.get_public_node_attributes
+
+puts attributes['gitlab']['test']
+```

doc/development/reconfigure_in_detail.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# What happens when `gitlab-ctl reconfigure` is run?
+
+Omnibus GitLab uses [Cinc](https://cinc.sh/) under the hood,
+which is a free-as-in-beer distribution of the open source software of [Chef Software Inc](https://docs.chef.io/).
+
+In very basic terms, a [Cinc client](https://cinc.sh/start/client/) run
+happens when `gitlab-ctl reconfigure` is run. This document elaborates
+the process and details the flow of control during a `gitlab-ctl reconfigure`
+run.
+
+`gitlab-ctl reconfigure` is defined in the
+[`omnibus-ctl` project](https://gitlab.com/gitlab-org/build/omnibus-mirror/omnibus-ctl/-/blob/0.6.0.1/lib/omnibus-ctl.rb#L517)
+and as mentioned above, it performs a
+[`cinc-client` run](https://gitlab.com/gitlab-org/build/omnibus-mirror/omnibus-ctl/-/blob/0.6.0.1/lib/omnibus-ctl.rb#L501)
+under the hood in [the local mode](https://docs.chef.io/ctl_chef_client/#run-in-local-mode) (using the `-z` flag). This invocation takes
+two files as inputs:
+
+- A configuration file named [`solo.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/solo.rb).
+- An attribute file named `dna.json`, which is created during build time and loads:
+  - For CE, the [`gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/files/gitlab-cookbooks/gitlab) cookbook.
+  - For EE, the [`gitlab-ee`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/files/gitlab-cookbooks/gitlab-ee) cookbook.
+
+Cinc then follows its [two-pass model of execution](https://coderanger.net/two-pass/) for the selected cookbook.
+
+In the load phase, the main cookbook and its dependency cookbooks (mentioned in
+the `metadata.rb` file) are loaded. The attributes mentioned in the default
+attribute files of these cookbooks are loaded (thus populating the `node` object
+with the default values specified in those attribute files) and the custom
+resources are all made available for use. Control then moves to the
+execution phase. In the `dna.json` file, we specify the cookbook name as the
+`run_list`, which makes Cinc use the default recipe as the only entry in the run
+list.
+
+The `gitlab-ee` cookbook extends the `gitlab` cookbook with EE-only
+features. For explanation purposes, let's first look at the
+[`gitlab` cookbook's default recipe](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab/recipes/default.rb).
+
+## The default recipe
+
+The functionality in the default recipe can be summarized as below:
+
+1. Call the `config` recipe to load the attributes from `gitlab.rb` and fully
+   populate the `node` object.
+1. Check for any deprecations and exit early.
+1. Check for any problematic settings in the run environment. For example,
+   if `LD_LIBRARY_PATH` is defined it can interfere with the included
+   libraries, how software links against them, and raise warnings.
+1. Check for non-UTF-8 locales and raise warnings.
+1. Create and configure necessary base directories like `/etc/gitlab` (unless
+   explicitly disabled), `/var/opt/gitlab`, and `/var/log/gitlab`.
+1. Call other necessary helper recipes and enable or disable recipes for different
+   services, etc.
+
+Note that this summary is not complete. Check out the default recipe code to
+learn more.
+
+### The config recipe
+
+The config recipe populates the `node` object with the
+final values for various settings after merging static default values,
+computed default values, and user values specified
+in `/etc/gitlab/gitlab.rb` file.
+
+In the above statement, we mention two types of default values:
+
+- **Static:** Static default values are specified in various
+  attribute files in different cookbooks and are independently set.
+- **Computed:** Computed default values are used in scenarios where the
+  default value for a setting depends on either the static default
+  value or user-specified value of another setting.
+
+For example, `gitlab_rails['gitlab_port']` defaults to the static
+value `80`. It translates to `production.gitlab.port` in the rendered
+`gitlab.yml` file that configures the GitLab Rails listener port. The
+`gitlab_rails['pages_host']` and `gitlab_rails['pages-port']` values,
+which inform GitLab Rails about GitLab Pages, depend on the user
+specified value in from `pages_external_url`. The computation of these
+default values may only happen **_after_** `gitlab.rb` gets parsed.
+
+#### What goes in to the `gitlab.rb` file?
+
+Omnibus GitLab uses a module named `Gitlab` to store the settings specified in
+`gitlab.rb`. This module, which extends `Mixlib::Config` module, can work as a
+configuration hash. In the
+[definition of this module](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/package/libraries/config/gitlab.rb),
+we register the various roles, attribute blocks, and top-level attributes that
+can be specified in `gitlab.rb`. The code for this registration is specified in
+the `SettingsDSL` module, and is extended by `GitLab` module.
+
+When `Gitlab.from_file` is called in the config recipe, the settings from
+`gitlab.rb` are parsed and loaded to the `Gitlab` object, and are accessible via
+`Gitlab['<setting_name>']`.
+
+Once `gitlab.rb` has been parsed and its values are available in `Gitlab`
+object, we can compute the default values of settings dependent on other settings.
+
+#### Computation of default values
+
+Each component with attributes that require calculation specify a library file
+at registration. One method in this file, `parse_variables`, validates user-provided
+input and, in typical usage, sets default values if the user did not already specify
+a value. It also detects bad configuration and raises errors.
+
+As mentioned above, `parse_variables` sets default values based on static
+defaults or user-provided values in related settings. The default values
+from these related settings are available in the `node` object after the load
+phase of the Cinc run. This `node` object, while available in the recipe, is
+not available in the libraries. To make the static default values available
+in the libraries, we attach the `node` object to the `GitLab` object in
+the config recipe with the code below:
+
+```ruby
+Gitlab[:node] = node
+```
+
+With this, the static default values of attributes can be accessed in the libraries
+using `Gitlab[:node]['<top-level-key>'][<setting>]`.
+
+It is important to note that:
+
+- The `Gitlab` object stores keys as they are mentioned in `gitlab.rb`.
+- `node` stores them based on the defined nesting attribute-block-attribute hierarchy.
+
+So, `gitlab_rails` settings from
+`gitlab.rb` are available as `Gitlab['gitlab_rails'][*]` while default values of
+those settings from attribute files are available at
+`Gitlab[:node]['gitlab']['gitlab_rails']`. The `gitlab_rails` key is specified
+under the `gitlab` attribute block, so an extra layer of nesting is present
+while accessing it via `node`.
+
+While the `Gitlab` object is technically only supposed to hold the settings
+specified in `gitlab.rb`, when computing default values of settings based on
+other settings, we usually put them under `Gitlab` key itself.
+[Issue #3932](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3923) is open to change
+this behavior.
+
+#### Handling of secrets
+
+Many attributes required for GitLab functionality are secrets that need to
+persist across reconfigure runs and, in multi-node setups, across different
+nodes. Moreover, if the user did not specify values for these secrets then
+Omnibus GitLab must create them to function. For this purpose, each library file
+specifies a `parse_secrets` method similar to the `parse_variables` method. This
+method generates secrets, unless explicitly disabled, if none have been specified
+in the `gitlab.rb` file.
+
+These secrets are written, unless explicitly disabled, to a file named
+`/etc/gitlab/gitlab-secrets.json`. This file is read by subsequent reconfigure
+runs and the secret persists across every reconfigure run.
+
+#### Update the node object with final attribute list
+
+After all libraries parse their respective attributes and secrets, the
+final configuration is ready to merge with default attributes already
+present in `node`. The `node.consume_attributes` method merges the
+final configuration with the default configuration populated in the load
+phase. Any configuration read from `gitlab.rb` or computed in the libraries
+overwrite the values for keys matched in the `node` object. At this point,
+the `node` object contains the final attribute list.
+
+#### `gitlab-cluster.json` file
+
+A user configures the system with `gitlab.rb` to match the requirement. In
+certain scenarios, however, we need to perform alterations without changes
+to the `gitlab.rb` file. Omnibus GitLab writes to a different file,
+`/etc/gitlab/gitlab-cluster.json`, that overrides user-specified values in
+`gitlab.rb`. The `gitlab-ctl` command or a reconfigure dynamically populates
+this file and it gets read and merged over the `node` attributes at the end
+of the config recipe.
+
+The `gitlab-ctl geo promote` command, when used on a multi-node PostgreSQL
+instance with Patroni, must disable the Patroni standby server. In this
+example, the standby server would normally be disabled via
+`patroni['standby_cluster']['enable']` in `gitlab.rb`. The `gitlab.rb` file
+should remain read-only for the duration of the Cinc run, so this setting
+is changed in the `gitlab-cluster.json` file. Future reconfigure runs parse
+the `gitlab-cluster.json` file at the end and `node['patroni']['standby_cluster']['enable']`
+will evaluate `false`.
+
+The Cinc run executes helper and service-specific recipes after
+the config recipe. Once these are complete, the node object is
+fully populated and may be used in recipes and resources.
+
+The default recipe in EE cookbook essentially calls `gitlab::default` recipe,
+and then handles the EE-specific components separately.

doc/development/s390x.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Developing for s390x architecture
+
+## Endianness
+
+[Linux on IBM Z](https://en.wikipedia.org/wiki/Linux_on_IBM_Z) runs on the s390x architecture. Unlike x86,
+the s390x architecture is [big-endian](https://en.wikipedia.org/wiki/Endianness). Many open source projects only
+test against little-endian systems, so there may be endian-related bugs
+lurking. For example, in the Ruby implementation of Protocol Buffers,
+we [fixed a handful of big-endian bugs](https://github.com/protocolbuffers/protobuf/issues/9141#issuecomment-962698503).
+
+## VM provisioning
+
+There are several ways to develop for s390x. You can use:
+
+- [A LinuxONE virtual server](https://community.ibm.com/zsystems/l1cc/).
+- [QEMU to emulate s390x](https://qemu-project.gitlab.io/qemu/system/target-s390x.html).
+
+  Use the following commands to spin up an emulated s390x virtual machine with Docker:
+
+  ```shell
+  docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
+  docker run --rm -it s390x/ubuntu bash
+  ```
+
+Note that the QEMU virtual machine is usually slower than a LinuxONE
+machine.
+
+## Bazel
+
+[Bazel](https://bazel.build/) is needed by a number of Google projects,
+such as Protocol Buffers. When compiling from source, you might run [into these fatal compiler warnings](https://github.com/protocolbuffers/protobuf/issues/9141#issuecomment-955997742).
+There is a [workaround to disable those warnings](https://github.com/bazelbuild/bazel/issues/13597#issuecomment-930606195).
+In addition, IBM [posted these instructions](https://github.com/linux-on-ibm-z/docs/wiki/Building-Bazel).
+
+## Omnibus build image
+
+To build Omnibus GitLab for s390x, you can use [this Dockerfile template](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/blob/master/docker/Dockerfile_debian_10_s390x.erb). Generate the Dockerfile with:
+
+```shell
+./docker/scripts/generate-dockerfile docker/Dockerfile_debian_10_s390x.erb
+```

doc/development/setup.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Set up your development environment
+
+Development of the Linux package can be done using an existing package available
+from the [Downloads page](https://about.gitlab.com/install/). To know how to set up
+a build environment to build these packages and use them, please read
+[Setting up a Build Environment](../build/build_package.md#prepare-a-build-environment)
+
+Choose one of the GitLab installation methods below. To provide isolation and
+to prevent rebuilding of the package for each and every change, it is preferred
+to use a container for development.
+
+## Set up a container
+
+1. Install Docker for your OS as per the [official Docker installation docs](https://docs.docker.com/get-docker/).
+
+1. Pull the GitLab CE nightly image:
+
+   ```shell
+   docker pull gitlab/gitlab-ce:nightly
+   ```
+
+1. Run the Docker image with a shell prompt:
+
+   ```shell
+   docker run -it --publish 443:443 --publish 80:80 --publish 22:22 gitlab/gitlab-ce:nightly bash
+   ```
+
+   This command runs Docker with the GitLab nightly image. You start with a
+   bash prompt, where you run the following commands.
+
+1. Initialize GitLab by first starting runsv, followed by `reconfigure`:
+
+   ```shell
+   /opt/gitlab/embedded/bin/runsvdir-start &
+   gitlab-ctl reconfigure
+   ```
+
+   If you have sysctl errors after running `reconfigure`, there is a workaround in the
+   [common installation problems doc](../troubleshooting.md#failed-to-modify-kernel-parameters-with-sysctl).
+
+1. (Optional) Take a snapshot of your container, so you can revert to this image if required. Run these commands on the Docker host:
+
+   ```shell
+   docker ps # Find the container ID of our container.
+   docker commit <container_id> gitlab_nightly_post_install
+   ```
+
+## Use an official nightly package
+
+1. Get the GitLab CE nightly package from the [Nightly Build repository](https://packages.gitlab.com/gitlab/nightly-builds)
+   and install it using the instructions given on that page.
+
+   NOTE:
+   On Ubuntu Xenial, you may have to install `tzdata`. This
+   [issue is reported in #4769](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4679).
+
+1. Configure and start GitLab.
+1. Check if you can access the GitLab instance from your host browser on `<ip address of host>`.
+1. Install the basic tools used for developing Omnibus GitLab:
+
+   ```shell
+   sudo apt-get install git
+   ```
+
+## Get the source of Omnibus GitLab
+
+1. Get the source code of Omnibus GitLab from the [repository on GitLab.com](https://gitlab.com/gitlab-org/omnibus-gitlab):
+
+   ```shell
+   git clone https://gitlab.com/gitlab-org/omnibus-gitlab.git ~/omnibus-gitlab
+   ```
+
+   We will be doing the development inside the `~/omnibus-gitlab` directory.
+
+1. Instructing GitLab to apply the changes we make to the cookbooks.
+
+   During development, we need the changes we make to the cookbooks to be
+   applied immediately to the running GitLab instance. So, we have to configure
+   GitLab to use those cookbooks instead of the ones shipped during
+   installation. This involves backing up of the existing cookbooks directories
+   and symlinking the directories where we make modifications to its location:
+
+   ```shell
+   cd ~/omnibus-gitlab/files/gitlab-cookbooks
+   for i in $(ls); do
+     mv "/opt/gitlab/embedded/cookbooks/${i}" "/opt/gitlab/embedded/cookbooks/${i}.$(date +%s)"
+     ln -s "$(pwd)/${i}" "/opt/gitlab/embedded/cookbooks/${i}"
+   done
+   ```
+
+   Now, you can make any necessary changes in the cookbooks inside `~/omnibus-gitlab/files/gitlab-cookbooks/`
+   and run `sudo gitlab-ctl reconfigure` for those changes to take effect.
+
+## Run GitLab QA Against Your Development Environment
+
+You can run [GitLab QA](https://gitlab.com/gitlab-org/gitlab-qa) tests against your development instance.
+
+This ensures that your new work is behaving as expected, and not breaking anything else. You can even add your own tests to QA to validate what you are working on.
+
+1. Create a user account on your development instance for GitLab QA to use
+
+   Then, from any machine that can reach your development instance:
+
+1. Clone the [GitLab EE](https://gitlab.com/gitlab-org/gitlab) repository
+
+   ```shell
+   git clone git@gitlab.com:gitlab-org/gitlab.git
+   ```
+
+1. Change to the `qa` directory
+
+   ```shell
+   cd gitlab/qa
+   ```
+
+1. Install the required gems
+
+   ```shell
+   bundle install
+   ```
+
+1. Run the tests
+
+   ```shell
+   GITLAB_USERNAME=$USERNAME GITLAB_PASSWORD=$PASSWORD bundle exec bin/qa Test::Instance $DEV_INSTANCE_URL
+   ```
+
+### Trigger QA pipeline against deployed instance
+
+If there is sustained network access to the deployed instance, you can trigger GitLab QA tests against the deployed instance using the [GitLab QA Executor](https://gitlab.com/gitlab-org/quality/gitlab-qa-executor) project. It contains CI configuration to run GitLab QA against self-managed GitLab environments with parallelization.
+
+## Run specific chefspec tests
+
+You can also test your changes against the current tests (or to test your newly added tests).
+
+1. Install `bundler` and `ruby-dev`, which are required to build the necessary gems:
+
+   ```shell
+   sudo apt install bundler ruby-dev
+   ```
+
+1. Change to the `omnibus-gitlab` directory:
+
+   ```shell
+   cd ~/omnibus-gitlab
+   ```
+
+1. Install the required gems inside the omnibus directory:
+
+   ```shell
+   /usr/bin/bundle install --path vendor/bundle
+   ```
+
+   If you use the GitLab Nightly Docker images, `/opt/gitlab/embedded/bin` is prepended to the `$PATH`, so using `bundle` alone uses the binary
+   that is included as part of GitLab. That's why we use the absolute path to the system `bundle`.
+
+1. Run your desired tests. The tests may need to run as root, as they need access to read the secrets file:
+
+   ```shell
+   sudo bundle exec rspec spec/<path_to_spec_file>
+   ```
+
+## Use chef-shell with `omnibus-gitlab` cookbooks
+
+You can use [chef-shell](https://docs.chef.io/workstation/chef_shell/) to debug changes to the cookbooks in your instance.
+
+As root in your development server run:
+
+```shell
+/opt/gitlab/embedded/bin/cinc-shell -z -c /opt/gitlab/embedded/cookbooks/solo.rb -s -j /opt/gitlab/embedded/cookbooks/dna.json
+```
+
+## Use Customers Portal Staging in GitLab
+
+To connect your GitLab instance to Customers Portal Staging, you can set the following
+[custom environment variables](../settings/environment-variables.md#setting-custom-environment-variables)
+in `/etc/gitlab/gitlab.rb` by supplying them in a `gitlab_rails['env']` hash. Set:
+
+- `GITLAB_LICENSE_MODE` to `test`
+- `CUSTOMER_PORTAL_URL` to `https://customers.staging.gitlab.com`
+
+For example:
+
+```ruby
+gitlab_rails['env'] = {
+    "GITLAB_LICENSE_MODE" => "test",
+    "CUSTOMER_PORTAL_URL" => "https://customers.staging.gitlab.com"
+}
+```
+
+NOTE:
+For [GitLab Geo](https://docs.gitlab.com/ee/administration/geo/), deployments using the above configuration must be added to each secondary site.
+
+## OpenShift GitLab Development Setup
+
+See Omnibus GitLab [development setup](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/doc/developer/guide.md) documentation.

doc/development/upgrading-chef.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Upgrading Chef
+
+Chef is a major part of `omnibus-gitlab`, and periodically needs to be updated. Follow these steps
+to upgrade to the latest version and verify the functionality.
+
+## Upgrade steps
+
+1. Create a new branch in `omnibus-gitlab`
+
+   ```shell
+   git switch -c upgrade-chef-to-X
+   ```
+
+1. Update the appropriate software entries in `config/templates/omnibus-gitlab-gems/Gemfile`.
+1. In the `config/templates/omnibus-gitlab-gems` directory, run `bundle update` for the updated gems:
+
+   ```shell
+   bundle update chef ohai
+   ```
+
+1. Update entries in `Gemfile` to the new version. At a minimum, the `chef` and `ohai` entries will need to be updated.
+1. Update the bundle
+
+   1. If this is a major version upgrade
+
+   ```shell
+   bundle update chef ohai
+   ```
+
+   1. If this is a minor version upgrade
+
+   ```shell
+   bundle update chef ohai --conservative
+   ```
+
+   It may be necessary to chase down errors related to dependencies being upgraded
+
+1. Commit the changes
+
+   ```shell
+   git add config/templates/omnibus-gitlab-gems/Gemfile{,.lock}
+   git add Gemfile{,.lock}
+   git commit
+   git push
+   ```
+
+1. Ensure the pipelines pass on GitLab.com
+1. Trigger an EE package pipeline to ensure we get a `gitlab-qa` run
+1. When available, trigger an HA validation job
+1. Check QA jobs for the package pipeline, and the HA validation job, ensure pipelines are green, or any failures are unrelated.
+1. Push to `omnibus-gitlab` on `dev.gitlab.org` and ensure the package builds on all platforms
+1. Download a package to a dev environment, and verify you can upgrade from an older version of `omnibus-gitlab`, to the newer package
+1. Verify a Geo installation is successful using the newer package
+
+## Bonus points
+
+1. Read through the Changelog and Release notes, identify any improvements, new features, or bug fixes which may apply to omnibus and open follow up issues

doc/development/upgrading-software-components.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Upgrading software components
+
+The Linux package is created from a series of software components, some of which
+are developed by GitLab and others which are sourced from free and open source
+projects. Software components can be updated individually as new features, bug
+fixes, and security vulnerabilities become available.
+
+Software component upgrades can be risky, especially when non-backwards
+compatible changes are made. Considering [Semantic versioning](https://semver.org/)], examining changelogs, and examining release
+notes can give a sense of the amount of risk involved in an upgrade. In all
+cases, upgrades should be thoroughly tested before merging.
+
+The [CNG](https://gitlab.com/gitlab-org/build/CNG) project uses some of these
+same software components. Components common to both projects should be updated
+in both.
+
+## Types of software components
+
+There are two types of software components used in the Linux package:
+
+- External software components
+- Internal software components
+
+### External software components
+
+External software components source is downloaded directly from an external site
+or copied from the
+[`omnibus-mirror`](https://gitlab.com/gitlab-org/build/omnibus-mirror) repository.
+A component can be provided using a `git clone`, extracting from a source
+tarball, performing a `gem install` (for Ruby modules), or performing a `pip
+install` (for Python modules).
+
+### Internal software components
+
+Internal software component are developed by GitLab and external contributors.
+Source for internal software components is downloaded from a project's GitLab
+repository. Versions used in a build are set by the Git reference contained in
+the project's corresponding `*VERSION` files. These versions can be overridden
+by environment variables. For more information, see
+[Use specific branches or versions of a GitLab component](../build/team_member_docs.md#use-specific-branches-or-versions-of-a-gitlab-component).
+
+Updates to internal software components are done by merge requests in the
+corresponding repository.
+
+## Internal software component update workflow
+
+A typical workflow for updating an internal software component.
+
+### Create a fork/branch
+
+External contributors should create a fork of the
+[`gitlab-org/omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) repository.
+
+Create a new branch from the target branch (usually `master` of the
+[`gitlab-org/omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) repository.
+
+### Modify `config/software/<component.rb>`
+
+1. Find the corresponding configure file for the component that you want to
+ update in the `config/software` directory. For example
+ `config/software/prometheus.rb` is used for the Prometheus component.
+
+1. Change the version to the version you want to update to. If applicable, also
+change the corresponding `sha256` to the value of the corresponding version
+source tarball.
+
+### Add or modify any required patches
+
+The new component version may require:
+
+- Adding new patches.
+- Removing existing patches.
+- Changing existing patches.
+
+All patch files go in `config/patches/<component name>`. They are then
+referenced in the corresponding `config/software/<component name>.rb` file.
+Examples can be found at:
+
+- [unzip component](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/config/software/unzip.rb)
+- [unzip patches](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/config/patches/unzip).
+
+### Push branch
+
+Push the branch to the to the upstream repository.
+
+### Create merge request (MR)
+
+Create a merge request using your development branch and the target branch.
+
+### Build
+
+Build the Linux package either:
+
+- Using the CI/CD system.
+- Building locally using [Build local Linux package](../build/build_package.md).
+
+You must build on all architectures using the CI/CD system before a merge
+request for an updated software component can be accepted.
+
+### Test
+
+Install the resulting Linux package and test the component changes.
+
+## Testing software component updates
+
+### Minimum test requirements
+
+At a minimum, the following tests should be performed when updating a software
+component.
+
+- Perform a successful GitLab Enterprise Edition (EE) build on all supported platforms.
+- Run `qa-subset-test` as well as manual `qa-remaining-test-manual` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
+- Install and verify that component version has been upgraded.
+- Verify basic functionality of the software component.
+
+Additional testing is almost always required and varies depending on the
+software component.
+
+### Test plans
+
+Here are test plans for individual components. They are meant to be copied into
+the merge request where their execution can be recorded.
+
+Not every component is listed here. Please consider creating a merge request to
+add one for a component upgrade that you are working on. Use
+`test-plans/upgrade-component-testplan-template.md` as a starting point.
+
+These test plans are not exhaustive. The might need to be supplemented depending
+on the degree of change made to the component. Record these additions in the
+merge request and consider adding them here. Use the following filename pattern
+when creating the test plan file:
+
+```plaintext
+upgrade-<component-name>-testplan.md
+```
+
+And add a link in `test-plans/index.md`.

doc/development/vulnerabilities.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Handle vulnerabilities detected by dependency scanning
+
+A scheduled pipeline runs `dependency_scanning` job every night. This job adds new
+vulnerabilities to the
+[Vulnerability Report](https://gitlab.com/gitlab-org/omnibus-gitlab/-/security/vulnerability_report/).
+
+Slack notifications tell `#g_distribution` on Slack when new
+vulnerabilities are detected. Complete the following steps when you receive this notification.
+
+1. Visit the [Omnibus Vulnerability Report](https://gitlab.com/gitlab-org/omnibus-gitlab/-/security/vulnerability_report)
+   and locate the appropriate vulnerability. If the vulnerability is legitimate:
+
+   - Select `Create Issue` to open a confidential issue in the
+   [`omnibus-gitlab` issue tracker](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues).
+   - Change the vulnerability status to `Confirmed`. If the vulnerability turns
+   out to be a false positive, duplicate, or otherwise not actionable, change the
+   status to `Dismiss`.
+
+1. Label the issue with the `security` and `For Scheduling` labels. The GitLab
+   Security team is then made aware of this issue due to the automation by
+   escalator: `https://gitlab.com/gitlab-com/gl-security/automation/escalator`.
+
+1. The Security team triages and schedules the issue with the help of Distribution.
+
+1. If the issue is actionable for us, the Security team:
+
+   - Schedules the issue based on its severity and priority.
+   - Creates the needed merge requests (MRs) to target all relevant branches.
+
+1. After the MR that fixes the vulnerability has been merged, and the corresponding
+   issue is closed:
+
+   - Visit the [Omnibus Vulnerability Report](https://gitlab.com/gitlab-org/omnibus-gitlab/-/security/vulnerability_report).
+   - Locate the appropriate vulnerability and set the status to `Resolved` if not
+   already done automatically.
+
+1. If the issue is a no-op for our use case, set its status to `Dismissed` in the
+   [Vulnerability Report](https://gitlab.com/gitlab-org/omnibus-gitlab/-/security/vulnerability_report)
+   page and close the corresponding issue.

doc/index.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Linux package documentation
+
+DETAILS:
+**Tier:** Free, Premium, Ultimate
+**Offering:** Self-managed
+
+The Linux package has different services and tools required to run GitLab. Most users can install it without laborious
+configuration.
+
+## Package information
+
+- [Checking the versions of bundled software](https://docs.gitlab.com/ee/administration/package_information/index.html#checking-the-versions-of-bundled-software)
+- [Package defaults](https://docs.gitlab.com/ee/administration/package_information/defaults.html)
+- [Components included](https://docs.gitlab.com/ee/development/architecture.html#component-list)
+- [Deprecated Operating Systems](https://docs.gitlab.com/ee/administration/package_information/supported_os.html#os-versions-that-are-no-longer-supported)
+- [Signed Packages](https://docs.gitlab.com/ee/administration/package_information/signed_packages.html)
+- [Deprecation Policy](https://docs.gitlab.com/ee/administration/package_information/deprecation_policy.html)
+- [Licenses of bundled dependencies](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html)
+
+## Installation
+
+For installation details, see [Install GitLab with the Linux package](installation/index.md).
+
+## Running on a low-resource device (like a Raspberry Pi)
+
+You can run GitLab on supported low-resource computers like the Raspberry Pi 3, but you must tune the settings
+to work best with the available resources. Check out the [documentation](settings/rpi.md) for suggestions on what to adjust.
+
+## Maintenance
+
+- [Get service status](maintenance/index.md#get-service-status)
+- [Starting and stopping](maintenance/index.md#starting-and-stopping)
+- [Invoking Rake tasks](maintenance/index.md#invoking-rake-tasks)
+- [Starting a Rails console session](maintenance/index.md#starting-a-rails-console-session)
+- [Starting a PostgreSQL superuser `psql` session](maintenance/index.md#starting-a-postgresql-superuser-psql-session)
+- [Container registry garbage collection](maintenance/index.md#container-registry-garbage-collection)
+
+## Configuring
+
+- [Configuring the external URL](settings/configuration.md#configure-the-external-url-for-gitlab)
+- [Configuring a relative URL for GitLab (experimental)](settings/configuration.md#configure-a-relative-url-for-gitlab)
+- [Storing Git data in an alternative directory](settings/configuration.md#store-git-data-in-an-alternative-directory)
+- [Changing the name of the Git user group](settings/configuration.md#change-the-name-of-the-git-user-or-group)
+- [Specify numeric user and group identifiers](settings/configuration.md#specify-numeric-user-and-group-identifiers)
+- [Start Linux package installation services only after a given file system is mounted](settings/configuration.md#start-linux-package-installation-services-only-after-a-given-file-system-is-mounted)
+- [Disable user and group account management](settings/configuration.md#disable-user-and-group-account-management)
+- [Disable storage directory management](settings/configuration.md#disable-storage-directories-management)
+- [Failed authentication ban](settings/configuration.md#configure-a-failed-authentication-ban)
+- [SMTP](settings/smtp.md)
+- [NGINX](settings/nginx.md)
+- [LDAP](https://docs.gitlab.com/ee/administration/auth/ldap/index.html)
+- [Puma](https://docs.gitlab.com/ee/administration/operations/puma.html)
+- [ActionCable](settings/actioncable.md)
+- [Redis](settings/redis.md)
+- [Logs](settings/logs.md)
+- [Database](settings/database.md)
+- [Reply by email](https://docs.gitlab.com/ee/administration/reply_by_email.html)
+- [Environment variables](settings/environment-variables.md)
+- [`gitlab.yml`](settings/gitlab.yml.md)
+- [Backups](settings/backups.md)
+- [Pages](https://docs.gitlab.com/ee/administration/pages/index.html)
+- [SSL](settings/ssl/index.md)
+- [GitLab and Registry](https://docs.gitlab.com/ee/administration/packages/container_registry.html)
+- [Configuring an asset proxy server](https://docs.gitlab.com/ee/security/asset_proxy.html)
+- [Image scaling](settings/image_scaling.md)
+- [GitLab Agent](https://docs.gitlab.com/ee/administration/clusters/kas.html)
+
+## Upgrading
+
+- [Upgrade guidance](https://docs.gitlab.com/ee/update/package/), including [supported upgrade paths](https://docs.gitlab.com/ee/update/index.html#upgrade-paths).
+- [Upgrade from Community Edition to Enterprise Edition](https://docs.gitlab.com/ee/update/package/convert_to_ee.html)
+- [Upgrade to the latest version](https://docs.gitlab.com/ee/update/package/#upgrade-using-the-official-repositories)
+- [Downgrade to an earlier version](https://docs.gitlab.com/ee/update/package/downgrade.html)
+- [Upgrade from a non-Linux package PostgreSQL to a Linux package installation using a backup](update/convert_to_omnibus.md#upgrading-from-a-non-linux-package-postgresql-to-a-linux-package-installation-using-a-backup)
+- [Upgrade from a non-Linux package PostgreSQL to a Linux package installation in-place](update/convert_to_omnibus.md#upgrading-from-a-non-linux-package-postgresql-to-a-linux-package-installation-in-place)
+
+## Troubleshooting
+
+For troubleshooting details, see [Troubleshooting Omnibus GitLab installation issues](troubleshooting.md).
+
+## Omnibus GitLab developer documentation
+
+See the [development documentation](development/index.md)

doc/installation/index.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Install GitLab with the Linux package
+
+DETAILS:
+**Tier:** Free, Premium, Ultimate
+**Offering:** Self-managed
+
+## Prerequisites
+
+- [Installation Requirements](https://docs.gitlab.com/ee/install/requirements.html).
+- If you want to access your GitLab instance by using a domain name, like `mygitlabinstance.com`,
+  make sure the domain correctly points to the IP of the server where GitLab is being
+  installed. You can check this using the command `host mygitlabinstance.com`.
+- If you want to use HTTPS on your GitLab instance, make sure you have the SSL
+  certificates for the domain ready. (Note that certain components like
+  Container Registry which can have their own subdomains requires certificates for
+  those subdomains also.)
+- If you want to send notification emails, install and configure a mail server (MTA)
+  like Sendmail or Postfix. Alternatively, you can use other [third party SMTP servers](../settings/smtp.md).
+
+## Installation and Configuration
+
+These configuration settings are commonly used when configuring a Linux package installation.
+For a complete list of settings, see the [README](../index.md#configuring) file.
+
+- [Installing GitLab](https://about.gitlab.com/install/).
+  - [Manually downloading and installing a GitLab package](https://docs.gitlab.com/ee/update/package/#upgrade-using-a-manually-downloaded-package).
+- [Setting up a domain name/URL](../settings/configuration.md#configure-the-external-url-for-gitlab)
+  for the GitLab Instance so that it can be accessed easily.
+- [Enabling HTTPS](../settings/nginx.md#enable-https).
+- [Enabling notification emails](../settings/smtp.md#smtp-settings).
+- [Enabling replying via email](https://docs.gitlab.com/ee/administration/reply_by_email.html#set-it-up).
+  - [Installing and configuring Postfix](https://docs.gitlab.com/ee/administration/reply_by_email_postfix_setup.html).
+- [Enabling container registry on GitLab](https://docs.gitlab.com/ee/administration/packages/container_registry.html#container-registry-domain-configuration).
+  - You require SSL certificates for the domain used for container registry.
+- [Enabling GitLab Pages](https://docs.gitlab.com/ee/administration/pages/).
+  - If you want HTTPS enabled, you must get wildcard certificates.
+- [Enabling Elasticsearch](https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch.html).
+- [GitLab Mattermost](https://docs.gitlab.com/ee/integration/mattermost/). Set up the Mattermost messaging app that ships with the Linux package.
+- [GitLab Prometheus](https://docs.gitlab.com/ee/administration/monitoring/prometheus/index.html)
+  Set up the Prometheus monitoring included in the Linux package.
+- [GitLab High Availability Roles](../roles/index.md).
+
+### Set up the initial account
+
+By default, a Linux package installation automatically generates a password for the
+initial administrator user account (`root`) and stores it to
+`/etc/gitlab/initial_root_password` for at least 24 hours. For security reasons,
+after 24 hours, this file is automatically removed by the first `gitlab-ctl reconfigure`.
+
+The default account is tied to a randomly-generated email address. To override
+this, pass the `GITLAB_ROOT_EMAIL` environment variable to the installation command.
+
+NOTE:
+If GitLab can't detect a valid hostname for the server during the
+installation, a reconfigure does not run.
+
+To provide a custom initial root password, you have two options:
+
+- Pass the `GITLAB_ROOT_PASSWORD` environment variable to the
+  [installation command](https://about.gitlab.com/install/) provided
+  the hostname for the server is set up correctly:
+
+  ```shell
+  sudo GITLAB_ROOT_EMAIL="<gitlab_admin@example.com>" GITLAB_ROOT_PASSWORD="<strongpassword>" EXTERNAL_URL="http://gitlab.example.com" apt install gitlab-ee
+  ```
+
+  If during the installation GitLab doesn't automatically perform a
+  reconfigure, you have to pass the `GITLAB_ROOT_PASSWORD` or `GITLAB_ROOT_EMAIL`
+  variable to the first `gitlab-ctl reconfigure` run.
+
+- Before the first reconfigure, edit `/etc/gitlab/gitlab.rb` (create it if it
+  doesn't exist) and set:
+
+  ```ruby
+  gitlab_rails['initial_root_password'] = '<my_strong_password>'
+  ```
+
+Both of these methods apply only during the initial database seeding, which happens
+during the first reconfigure. For subsequent reconfigure runs, neither of
+the aforementioned methods have any effect. In that case, use the random
+password in `/etc/gitlab/initial_root_password` to log in, or
+[reset the root password](https://docs.gitlab.com/ee/security/reset_user_password.html).
+
+## Using Docker image
+
+You can also use the Docker images provided by GitLab to install and configure a GitLab instance.
+Check the [documentation](https://docs.gitlab.com/ee/install/docker.html) to know more.
+
+## Uninstall the Linux package (Omnibus)
+
+To uninstall the Linux package, you can opt to either keep your data (repositories,
+database, configuration) or remove all of them:
+
+1. Optional. To remove
+   [all users and groups created by the Linux package](../settings/configuration.md#disable-user-and-group-account-management)
+   before removing the package (with `apt` or `yum`):
+
+   ```shell
+   sudo gitlab-ctl stop && sudo gitlab-ctl remove-accounts
+   ```
+
+   NOTE:
+   If you have problems removing accounts or groups, run `userdel` or `groupdel` manually
+   to delete them. You might also want to manually remove the leftover user home directories
+   from `/home/`.
+
+1. Choose whether to keep your data or remove all of them:
+
+   - To preserve your data (repositories, database, configuration), stop GitLab and
+     remove its supervision process:
+
+     ```shell
+     sudo systemctl stop gitlab-runsvdir
+     sudo systemctl disable gitlab-runsvdir
+     sudo rm /usr/lib/systemd/system/gitlab-runsvdir.service
+     sudo systemctl daemon-reload
+     sudo systemctl reset-failed
+     sudo gitlab-ctl uninstall
+     ```
+
+   - To remove all data:
+
+     ```shell
+     sudo gitlab-ctl cleanse && sudo rm -r /opt/gitlab
+     ```
+
+1. Uninstall the package (replace with `gitlab-ce` if you have GitLab FOSS installed):
+
+   ```shell
+   # Debian/Ubuntu
+   sudo apt remove gitlab-ee
+
+   # RedHat/CentOS
+   sudo yum remove gitlab-ee
+   ```

doc/jihu_edition.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# JiHu Edition
+
+NOTE:
+This section is only relevant if you are a customer in the Chinese market.
+
+GitLab licensed its technology to a new independent Chinese company, called JiHu.
+This independent company will help drive adoption of the GitLab complete DevOps
+platform in China and foster the GitLab community and open source contributions.
+
+For more information, see the
+[blog post announcement](https://about.gitlab.com/blog/2021/03/18/gitlab-licensed-technology-to-new-independent-chinese-company/)
+and the [FAQ](https://about.gitlab.com/pricing/faq-jihu/).
+
+## Prerequisites
+
+Before installing GitLab JiHu Edition, it is of critical importance to review the system [requirements](https://docs.gitlab.com/ee/install/requirements.html). The system requirements include details on the minimum hardware, software, database, and additional requirements to support GitLab.
+
+Once you have contracted with JiHu, a JiHu representative will contact you to supply you with a license
+that you can use as part of the installation process.
+
+## Install or update a JiHu Edition package
+
+NOTE:
+If you are installing for the first time, you have to pass the
+`EXTERNAL_URL="<GitLab URL>"` variable to set your preferred domain name. Installation
+automatically configures and starts GitLab at that URL. Enabling HTTPS requires
+[additional configuration](settings/nginx.md#enable-https) to specify the certificates.
+
+Please refer to the [GitLab Jihu Edition Install](https://gitlab.cn/install/) page
+for more details on installing or updating a JiHu Edition package.
+
+### Set initial password and apply license
+
+The first time GitLab JiHu Edition is installed, you are redirected to a password reset screen. Provide
+the password for the initial administrator account and you are redirected
+back to the login screen. Use the default account's username `root` to log in.
+
+For detailed instructions, see [installation and configuration](installation/index.md#installation-and-configuration).
+
+Additionally, you can navigate to the GitLab administration panel of your server and
+[upload your JiHu Edition license file](https://docs.gitlab.com/ee/administration/license.html#uploading-your-license).
+
+## Update GitLab Enterprise Edition to JiHu Edition
+
+To update an existing GitLab Enterprise Edition (EE) server installed using the Omnibus GitLab
+packages to GitLab JiHu Edition (JH), you install the JiHu Edition (JH)
+package on top of EE.
+
+The available options are:
+
+- (Recommended) Updating from the same version of EE to JH.
+- Updating from a lower version of EE to a higher version of JH, provided that this is a supported [upgrade path](https://docs.gitlab.com/ee/update/index.html#upgrade-paths) (for example, EE 13.5.4 to JH 13.10.0).
+
+In the following steps we assume that
+you are updating the same version (for example, EE 13.10.0 to JH 13.10.0).
+
+To update EE to JH:
+
+- If you installed GitLab using a deb/rpm package:
+
+  1. Take a [backup](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html).
+  1. Find the currently installed GitLab version:
+
+     **For Debian/Ubuntu**
+
+     ```shell
+     sudo apt-cache policy gitlab-ee | grep Installed
+     ```
+
+     The output should be similar to `Installed: 13.10.0-ee.0`, so the installed
+     version is `13.10.0-ee.0`.
+
+     **For CentOS/RHEL**
+
+     ```shell
+     sudo rpm -q gitlab-ee
+     ```
+
+     The output should be similar to `gitlab-ee-13.10.0-ee.0.el8.x86_64`, so
+     the installed version is `13.10.0-ee.0`.
+
+  1. Follow the same steps as when
+     [installing the JiHu Edition package](#install-or-update-a-jihu-edition-package) for
+     your operating system, and make sure to pick the same version as the one
+     noted in the previous step. Replace `<url>` with the URL of your package.
+
+  1. Reconfigure GitLab:
+
+     ```shell
+     sudo gitlab-ctl reconfigure
+     ```
+
+  1. Go to the GitLab administration panel of your server (`/admin/license/new`) and
+     upload your JiHu Edition license file. If you have an EE license already installed
+     prior to updating to JiHu, the EE license is automatically deactivated when JH
+     is installed.
+
+  1. Confirm that GitLab is working as expected, then remove the old
+     Enterprise Edition repository:
+
+     **For Debian/Ubuntu**
+
+     ```shell
+     sudo rm /etc/apt/sources.list.d/gitlab_gitlab-ee.list
+     ```
+
+     **For CentOS/RHEL**
+
+     ```shell
+     sudo rm /etc/yum.repos.d/gitlab_gitlab-ee.repo
+     sudo dnf config-manager --disable gitlab_gitlab-ee
+     ```
+
+- If you installed GitLab using Docker:
+
+   1. Follow the [Docker update guide](https://docs.gitlab.com/ee/install/docker.html)
+      and replace `gitlab/gitlab-ee:latest` with the following:
+
+      ```shell
+      registry.gitlab.com/gitlab-jh/omnibus-gitlab/gitlab-jh:<version>
+      ```
+
+      Where `<version>` is the currently installed GitLab version, which
+      you can find with:
+
+      ```shell
+      sudo docker ps | grep gitlab/gitlab-ee | awk '{print $2}'
+      ```
+
+      The output should be similar to: `gitlab/gitlab-ee:13.10.0-ee.0`, so
+      in this case, `<version>` equals to `13.10.0`.
+
+   1. Go to the GitLab administration panel of your server (`/admin/license/new`) and
+      upload your JiHu Edition license file. If you have an EE license already installed
+      prior to updating to JiHu, the EE license is automatically deactivated when JH
+      is installed.
+
+That's it! You can now use GitLab JiHu Edition! To update to a newer
+version, see [Install or update a JiHu Package](#install-or-update-a-jihu-edition-package).
+
+## Go back to GitLab Enterprise Edition
+
+To downgrade the JiHu Edition installation to GitLab Enterprise Edition (EE), install the same version of the Enterprise Edition package on top of the currently installed one.
+
+Depending on the preferred installation method for GitLab EE, either:
+
+- Use the official GitLab package repository and [install GitLab EE](https://about.gitlab.com/install/?version=ee), or
+- Download the GitLab EE package and [manually install it](https://docs.gitlab.com/ee/update/package/#upgrade-using-a-manually-downloaded-package).

doc/maintenance/index.md

+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Maintenance commands
+
+DETAILS:
+**Tier:** Free, Premium, Ultimate
+**Offering:** Self-managed
+
+The following commands can be run after installation.
+
+## Get service status
+
+Run `sudo gitlab-ctl status` to see the current state and uptime of each GitLab component.
+
+The output will look similar to this:
+
+```plaintext
+run: nginx: (pid 972) 7s; run: log: (pid 971) 7s
+run: postgresql: (pid 962) 7s; run: log: (pid 959) 7s
+run: redis: (pid 964) 7s; run: log: (pid 963) 7s
+run: sidekiq: (pid 967) 7s; run: log: (pid 966) 7s
+run: puma: (pid 961) 7s; run: log: (pid 960) 7s
+```
+
+As a demonstration, the first line of the previous example can be interpreted as follows:
+
+- `Nginx` is the process name.
+- `972` is the process identifier.
+- NGINX has been running for 7 seconds (`7s`).
+- `log` indicates a [svlogd logging process](https://manpages.ubuntu.com/manpages/lunar/en/man8/svlogd.8.html) attached to the preceding process.
+- `971` is the process identifier for the logging process.
+- The logging process has been running for 7 seconds (`7s`).
+
+## Tail process logs
+
+See [settings/logs.md.](../settings/logs.md)
+
+## Starting and stopping
+
+After Omnibus GitLab is installed and configured, your server has a runit
+service directory (`runsvdir`) process running that gets started at boot via
+`/etc/inittab` or the `/etc/init/gitlab-runsvdir.conf` Upstart resource. You
+should not have to deal with the `runsvdir` process directly; you can use the
+`gitlab-ctl` front-end instead.
+
+You can start, stop or restart GitLab and all of its components with the
+following commands.
+
+```shell
+# Start all GitLab components
+sudo gitlab-ctl start
+
+# Stop all GitLab components
+sudo gitlab-ctl stop
+
+# Restart all GitLab components
+sudo gitlab-ctl restart
+```
+
+Note that on a single-core server it may take up to a minute to restart Puma and
+Sidekiq. Your GitLab instance will give a 502 error until Puma is up again.
+
+It is also possible to start, stop or restart individual components.
+
+```shell
+sudo gitlab-ctl restart sidekiq
+```
+
+Puma does support almost zero-downtime reloads. These can be triggered as
+follows:
+
+```shell
+sudo gitlab-ctl hup puma
+```
+
+Note that you must wait for the `hup` command to finish. This could take some time. Leave the node out of the pool and do not restart services on the node where this is invoked until this completes. You also cannot use a Puma reload to update the Ruby runtime.
+
+Puma has the following signals to control application behavior:
+
+| Signal   | Puma                                                                |
+| -------- | ------                                                              |
+| `HUP`    | reopen log files defined, or stop the process to force restart      |
+| `INT`    | gracefully stops requests processing                                |
+| `USR1`   | restart workers in phases, a rolling restart, without config reload |
+| `USR2`   | restart workers and reload config                                   |
+| `QUIT`   | exit the main process                                               |
+
+For Puma, `gitlab-ctl hup puma` will send a sequence of `SIGINT` and `SIGTERM`
+(if process does not restart) signals. Puma stops accepting new connections as
+soon as `SIGINT` is received. It finishes all running requests. Then `runit`
+restarts the service.
+
+## Invoking Rake tasks
+
+To invoke a GitLab Rake task, use `gitlab-rake`. For example:
+
+```shell
+sudo gitlab-rake gitlab:check
+```
+
+Leave out `sudo` if you are the `git` user.
+
+Contrary to with a traditional GitLab installation, there is no need to change
+the user or the `RAILS_ENV` environment variable; this is taken care of by the
+`gitlab-rake` wrapper script.
+
+## Starting a Rails console session
+
+For more information, see
+[Rails console](https://docs.gitlab.com/ee/administration/operations/rails_console.html#starting-a-rails-console-session).
+
+## Starting a PostgreSQL superuser `psql` session
+
+If you need superuser access to the bundled PostgreSQL service you can
+use the `gitlab-psql` command. It takes the same arguments as the
+regular `psql` command.
+
+```shell
+# Superuser psql access to GitLab's database
+sudo gitlab-psql -d gitlabhq_production
+```
+
+This will only work after you have run `gitlab-ctl reconfigure` at
+least once. The `gitlab-psql` command cannot be used to connect to a
+remote PostgreSQL server, nor to connect to a local non-Omnibus PostgreSQL
+server.
+
+### Starting a PostgreSQL superuser `psql` session in Geo tracking database
+
+Similar to the previous command, if you need superuser access to the bundled
+Geo tracking database (`geo-postgresql`), you can use the `gitlab-geo-psql`.
+It takes the same arguments as the regular `psql` command. For HA, see more
+about the necessary arguments in:
+[Checking Configuration](https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html#checking-configuration)
+
+```shell
+# Superuser psql access to GitLab's Geo tracking database
+sudo gitlab-geo-psql -d gitlabhq_geo_production
+```
+
+## Container Registry garbage collection
+
+Container Registry can use considerable amounts of disk space. To clear up
+unused layers, the registry includes a
+[garbage collect command](https://docs.gitlab.com/ee/administration/packages/container_registry.html#container-registry-garbage-collection).
+
+## Restrict users from logging into GitLab
+
+If you need to temporarily restrict users from logging into GitLab, you can use
+`sudo gitlab-ctl deploy-page up`. When a user goes to your GitLab URL, they
+will be shown an arbitrary `Deploy in progress` page.
+
+To remove the page, you simply run `sudo gitlab-ctl deploy-page down`. You can also check the status of the deploy page with `sudo gitlab-ctl deploy-page status`.
+
+As a side note, if you would like to restrict logging into GitLab and restrict
+changes to projects, you can [set projects as read-only](https://docs.gitlab.com/ee/administration/troubleshooting/gitlab_rails_cheat_sheet.html#make-a-project-read-only-can-only-be-done-in-the-console)
+, then put up the `Deploy in progress` page.
+
+## Rotate the secrets file
+
+If required for security purposes, you can rotate the `/etc/gitlab/gitlab-secrets.json` secrets file. In this file:
+
+- Do not rotate the `gitlab_rails` secrets because it contains the database encryption keys. If this secret is rotated, you see the same
+  behavior as [when the secrets file is lost](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#when-the-secrets-file-is-lost).
+- You can rotate all other secrets.
+
+If you have multiple nodes in your GitLab environment, choose one of your Rails node to perform the initial steps.
+
+To rotate the secrets:
+
+1. [Verify that the database values can be decrypted](https://docs.gitlab.com/ee/administration/raketasks/check.html#verify-database-values-can-be-decrypted-using-the-current-secrets) and either make note of any decryption errors shown, or resolve
+   them before proceeding.
+
+1. Recommended. Extract your current secrets for `gitlab_rails`. Save the output because you need this later:
+
+   ```shell
+   sudo grep "secret_key_base\|db_key_base\|otp_key_base\|encrypted_settings_key_base\|openid_connect_signing_key\|ci_jwt_signing_key" /etc/gitlab/gitlab-secrets.json
+   ```
+
+1. Move your current secrets file to a different location:
+
+   ```shell
+   sudo mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.old
+   ```
+
+1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#reconfigure-a-linux-package-installation). GitLab will
+   then generate a new `/etc/gitlab/gitlab-secrets.json` file with new secret values.
+
+1. If you extracted the previous secrets for `gitlab_rails`, edit the new `/etc/gitlab/gitlab-secrets.json` file and replace the key/value pairs
+   under `gitlab_rails` with the previous secrets output obtained earlier.
+
+1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#reconfigure-a-linux-package-installation) again
+   so the changes made to the secrets file are applied.
+
+1. [Restart GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#restart-a-linux-package-installation) to ensure all services
+   are using the new secrets.
+
+1. If you have multiple nodes in your GitLab environment, you must copy the secrets to all of your other nodes:
+
+   1. On all other nodes, move your current secrets file to a different location:
+
+      ```shell
+      sudo mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.old
+      ```
+
+   1. Copy the new `/etc/gitlab/gitlab-secrets.json` file from your Rails node onto all of your other GitLab nodes.
+
+   1. On all other nodes, [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#reconfigure-a-linux-package-installation)
+      on each node.
+
+   1. On all other nodes, [restart GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#restart-a-linux-package-installation) on each
+      node to ensure all services are using the new secrets.
+
+   1. On all nodes, run a checksum match on the `/etc/gitlab/gitlab-secrets.json` file to confirm that the secrets match:
+
+      ```shell
+      sudo md5sum /etc/gitlab/gitlab-secrets.json
+      ```
+
+1. [Verify that the database values can be decrypted](https://docs.gitlab.com/ee/administration/raketasks/check.html#verify-database-values-can-be-decrypted-using-the-current-secrets). The output should match with the previous execution.
+  
+1. Confirm that GitLab is working as expected. If it is, it should be safe to delete the old secrets.