Skip to content

GitLab

Commit edb06ab2 authored by 徐豪's avatar 徐豪
Browse files

init

parents
Hide whitespace changes
Inline Side-by-side
Showing with 5337 additions and 0 deletions

Too many changes to show.

To preserve performance only 532 of 532+ files are displayed.
Plain diff Email patch
doc/release/index.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Omnibus GitLab release process
Our main goal is to make it clear which version of GitLab is in an omnibus
package.
## How is the official Omnibus GitLab package built
The official package build is fully automated by GitLab Inc.
We can differentiate between two types of build:
- Packages for release to <https://packages.gitlab.com>.
- Test packages built from branches available in S3 buckets.
Both types are built on the same infrastructure.
## Infrastructure
Each package is built on the platform it is intended for (CentOS 6 packages are
built on CentOS6 servers, Debian 8 packages on Debian 8 servers and so on).
The number of build servers varies but there is always at least one build
server per platform.
The Omnibus GitLab project fully utilizes GitLab CI. This means that each push
to the Omnibus GitLab repository will trigger a build in GitLab CI which will
then create a package.
Since we deploy GitLab.com using Omnibus GitLab packages, we need a separate
remote to build the packages in case of a problem with GitLab.com or due to
a security release of a package.
This remote is located on `https://dev.gitlab.org`. The only difference between the
Omnibus GitLab project on `https://dev.gitlab.org` and other public remotes is that the
project has active GitLab CI and has specific runners assigned to the project
which run on the build servers. This is also the case for all GitLab components,
eg. GitLab Shell is exactly the same on `https://dev.gitlab.org` as it is on GitLab.com.
All build servers run [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) and all runners use a deploy key
to connect to the projects on `https://dev.gitlab.org`. The build servers also have
access to official package repository at <https://packages.gitlab.com> and to a special
Amazon S3 bucket which stores the test packages.
## Build process
GitLab Inc is using the [release-tools project](https://gitlab.com/gitlab-org/release-tools/tree/master) to automate the release tasks
for every release. When the release manager starts the release process, a couple
of important things for Omnibus GitLab will be done:
1. All remotes of the project will be synced.
1. The versions of components will be read from GitLab CE/EE repository
(eg. `VERSION`, `GITLAB_SHELL_VERSION`) and written to the Omnibus GitLab repository.
1. A specific Git tag will be created and synced to Omnibus GitLab repositories.
When the Omnibus GitLab repository on `https://dev.gitlab.org` gets updated, GitLab CI
build gets triggered.
The specific steps can be seen in the `.gitlab-ci.yml` file in the Omnibus GitLab
repository. The builds are executed on all platforms at the same time.
During the build, Omnibus GitLab will pull external libraries from their source
locations and GitLab components like GitLab, GitLab Shell, GitLab Workhorse, and
so on will be pulled from `https://dev.gitlab.org`.
Once the build completes and the .deb or .rpm packages are built, depending on
the build type package will be pushed to <https://packages.gitlab.com> or to a temporary
(files older than 30 days are purged) S3 bucket.
## Specifying component versions manually
### On your development machine
1. Pick a tag of GitLab to package (e.g. `v6.6.0`).
1. Create a release branch in your Omnibus GitLab repo (e.g. `6-6-stable`).
1. If the release branch already exists, for instance because you are doing a
patch release, make sure to pull the latest changes to your local machine:
```shell
git pull https://gitlab.com/gitlab-org/omnibus-gitlab.git 6-6-stable # existing release branch
```
1. Use `support/set-revisions` to set the revisions of files in
`config/software/`. It will take tag names and look up the Git SHA1's, and set
the download sources to `https://dev.gitlab.org`. Use `set-revisions --ee` for an EE
release:
```shell
# usage: set-revisions [--ee] GITLAB_RAILS_REF GITLAB_SHELL_REF GITALY_REF GITLAB_ELASTICSEARCH_INDEXER_REF
# For GitLab CE:
support/set-revisions v1.2.3 v1.2.3 1.2.3 1.2.3 1.2.3
# For GitLab EE:
support/set-revisions --ee v1.2.3-ee v1.2.3 1.2.3 1.2.3 1.2.3
```
1. Commit the new version to the release branch:
```shell
git add VERSION GITLAB_SHELL_VERSION GITALY_SERVER_VERSION
git commit
```
1. Create an annotated tag in Omnibus GitLab corresponding to the GitLab tag.
The omnibus tag looks like: `MAJOR.MINOR.PATCH+OTHER.OMNIBUS_RELEASE`, where
`MAJOR.MINOR.PATCH` is the GitLab version, `OTHER` can be something like `ce`,
`ee` or `rc1` (or `rc1.ee`), and `OMNIBUS_RELEASE` is a number (starting at 0):
```shell
git tag -a 6.6.0+ce.0 -m 'Pin GitLab to v6.6.0'
```
WARNING:
Do NOT use a hyphen `-` anywhere in the Omnibus GitLab tag.
Examples of converting an upstream tag to an omnibus tag sequence:
| upstream tag | omnibus tag sequence |
| ------------ | -------------------- |
| `v7.10.4` | `7.10.4+ce.0`, `7.10.4+ce.1`, `...` |
| `v7.10.4-ee` | `7.10.4+ee.0`, `7.10.4+ee.1`, `...` |
| `v7.11.0.rc1-ee` | `7.11.0+rc1.ee.0`, `7.11.0+rc1.ee.1`, `...` |
1. Push the branch and the tag to both <https://gitlab.com> and `https://dev.gitlab.org`:
```shell
git push git@gitlab.com:gitlab-org/omnibus-gitlab.git 6-6-stable 6.6.0+ce.0
git push git@dev.gitlab.org:gitlab/omnibus-gitlab.git 6-6-stable 6.6.0+ce.0
```
Pushing an annotated tag to `https://dev.gitlab.org` triggers a package release.
### Publishing the packages
You can track the progress of package building on `https://dev.gitlab.org/gitlab/omnibus-gitlab/builds`.
They are pushed to [packagecloud repositories](https://packages.gitlab.com/gitlab/) automatically after
successful builds.
### Updating cloud images
The cloud image release process is documented here: <https://handbook.gitlab.com/handbook/alliances/cloud-images/>.
New image are released when:
1. There is a new monthly release of GitLab.
1. A security vulnerability has been fixed in a patch release.
1. There is a patch that fixes a critical issue impacting the image.
New images should be released within 3 business days of the package release.
Image specific release documentation:
- (**Deprecated**) [OpenShift](https://docs.gitlab.com/charts/development/release.html).
doc/roles/index.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Omnibus GitLab High Availability Roles
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
Omnibus GitLab includes various software components/services to support running GitLab in
a high availability configuration. By default, some of these supporting services
are disabled, and Omnibus GitLab is configured to run as a single node installation.
Each service can be enabled or disabled using configuration settings in `/etc/gitlab/gitlab.rb`,
but the introduction of `roles` allows you to easily enable a group of services,
and provides better default configuration based on the high availability roles you
have enabled.
## Not specifying any Roles (the default configuration)
When you don't configure GitLab with any roles, GitLab enables the default services for
a single node install. These include things like PostgreSQL, Redis, Puma, Sidekiq,
Gitaly, GitLab Workhorse, NGINX, etc.
These can still be individually enable/disabled by the settings in your `/etc/gitlab/gitlab.rb`.
## Specifying Roles
Roles are passed as an array in `/etc/gitlab/gitlab.rb`
Example specifying multiple roles:
```ruby
roles ['redis_sentinel_role', 'redis_master_role']
```
Example specifying a single role:
```ruby
roles ['geo_primary_role']
```
## Roles
The majority of the following roles will only work on a
[GitLab Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/), meaning
a `gitlab-ee` Omnibus package. It will be mentioned next to each role.
### GitLab App Role
- **application_role** (`gitlab-ce`/`gitlab-ee`)
The GitLab App role is used to configure an instance where only GitLab is running. Redis, PostgreSQL, and Consul services are disabled by default.
### Redis Server Roles
Documentation on the use of the Redis Roles can be found in [Configuring Redis for Scaling](https://docs.gitlab.com/ee/administration/redis/index.html)
- **redis_sentinel_role** (`gitlab-ee`)
Enables the sentinel service on the machine,
*By default, enables no other services.*
- **redis_master_role** (`gitlab-ee`)
Enables the Redis service and monitoring, and allows configuring the master password
*By default, enables no other services.*
- **redis_replica_role** (`gitlab-ee`)
Enables the Redis service and monitoring
*By default, enables no other services.*
### GitLab Geo Roles
The GitLab Geo roles are used for configuration of GitLab Geo sites. See the
[Geo Setup Documentation](https://docs.gitlab.com/ee/administration/geo/setup/index.html)
for configuration steps.
- **geo_primary_role** (`gitlab-ee`)
This role:
- Configures a single-node PostgreSQL database as a leader for streaming replication.
- Prevents automatic upgrade of PostgreSQL since it requires downtime of streaming replication to Geo secondary sites.
- Enables all single-node GitLab services including NGINX, Puma, Redis, and Sidekiq. If you are segregating services, then you must explicitly disable unwanted services in `/etc/gitlab/gitlab.rb`. Therefore, this role is only useful on a single-node PostgreSQL in a Geo primary site.
- Cannot be used to set up a PostgreSQL cluster in a Geo primary site. Instead, see [Geo multi-node database replication](https://docs.gitlab.com/ee/administration/geo/setup/database.html#multi-node-database-replication).
*By default, enables standard single-node GitLab services including NGINX, Puma, Redis, and Sidekiq.*
- **geo_secondary_role** (`gitlab-ee`)
- Configures the secondary read-only replica database for incoming
replication.
- Configures the Rails connection to the Geo tracking database.
- Enables the Geo tracking database `geo-postgresql`.
- Enables the Geo Log Cursor `geo-logcursor`.
- Disables automatic database migrations on the read-only replica database
during reconfigure.
- Reduces the number of Puma workers to save memory for other services.
- Sets `gitlab_rails['enable'] = true`.
This role is intended to be used in a Geo secondary site running on a single
node. If using this role in a Geo site with multiple nodes, undesired
services will need to be explicitly disabled in `/etc/gitlab/gitlab.rb`. See
[Geo for multiple nodes](https://docs.gitlab.com/ee/administration/geo/replication/multiple_servers.html).
This role should not be used to set up a PostgreSQL cluster in a Geo secondary
site. Instead, see [Geo multi-node database replication](https://docs.gitlab.com/ee/administration/geo/setup/database.html#multi-node-database-replication).
*By default, enables all of the GitLab default single node services. (NGINX, Puma, Redis, Sidekiq, etc)*
### Monitoring Roles
Monitoring roles are used to set up monitoring of installations. For additional information, see the [Monitoring documentation](https://docs.gitlab.com/ee/administration/monitoring/prometheus/index.html).
- **monitoring_role** (`gitlab-ce`/`gitlab-ee`)
Configures a central monitoring server to collect metrics and provide dashboards.
Enables Prometheus and Alertmanager.
### PostgreSQL Roles
Documentation on the usage of the PostgreSQL Roles can be found in [Configuring PostgreSQL for Scaling](https://docs.gitlab.com/ee/administration/postgresql/index.html)
- **postgres_role** (`gitlab-ce`/`gitlab-ee`)
Enables the PostgreSQL service on the machine
*By default, enables no other services.*
- **patroni_role** (`gitlab-ee`)
Enables the PostgreSQL, patroni, and Consul services on the machine
*By default, enables no other services.*
- **pgbouncer_role** (`gitlab-ee`)
Enables the PgBouncer and Consul services on the machine
*By default, enables no other services.*
- **consul_role** (`gitlab-ee`)
Enables the Consul service on the machine
*By default, enables no other services.*
### GitLab Pages Roles
GitLab Pages roles are used to set up and configure GitLab Pages. For additional
information, see the
[GitLab Pages Administration documentation](https://docs.gitlab.com/ee/administration/pages/index.html)
- **pages_role** (`gitlab-ce`/`gitlab-ee`)
Configures the server with a GitLab Pages instance.
*By default, enables no other services.*
### Sidekiq Roles
Sidekiq roles are used to set up and configure Sidekiq. For additional
information, see the
[Sidekiq Administration documentation](https://docs.gitlab.com/ee/administration/sidekiq/index.html)
- **sidekiq_role** (`gitlab-ce`/`gitlab-ee`)
Configures the server with Sidekiq service.
*By default, enables no other services.*
### Spamcheck Roles
Spamcheck roles are used to set up and configure Spamcheck services. For additional
information, see the
[Spamcheck documentation](https://docs.gitlab.com/ee/administration/reporting/spamcheck.html)
- **spamcheck_role** (`gitlab-ee`)
Configures the server with spamcheck and spam-classifier services.
*By default, enables no other services.*
doc/settings/actioncable.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Action Cable
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
Action Cable is a Rails engine that handles websocket connections.
## Configuring the worker pool size
Action Cable uses a separate thread pool per Puma worker. The number of threads can be configured
using the `actioncable['worker_pool_size']` option.
doc/settings/backups.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Backup
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
## Backup and restore configuration on a Linux package installation
It is recommended to keep a copy of `/etc/gitlab`, or at least of
`/etc/gitlab/gitlab-secrets.json`, in a safe place. If you ever
need to restore a GitLab application backup you need to also restore
`gitlab-secrets.json`. If you do not, GitLab users who are using
two-factor authentication will lose access to your GitLab server
and 'secure variables' stored in GitLab CI will be lost.
It is not recommended to store your configuration backup in the
same place as your application data backup, see below.
All configuration for Linux package installations is stored in `/etc/gitlab`. To backup your
configuration, just run `sudo gitlab-ctl backup-etc`. It creates a tar
archive in `/etc/gitlab/config_backup/`. Directory and backup files will be
readable only to root.
NOTE:
Running `sudo gitlab-ctl backup-etc --backup-path <DIRECTORY>` will place
the backup in the specified directory. The directory will be created if it
does not exist. Absolute paths are recommended.
To create a daily application backup, edit the cron table for user root:
```shell
sudo crontab -e -u root
```
The cron table will appear in an editor.
Enter the command to create a tar file containing the contents of
`/etc/gitlab/`. For example, schedule the backup to run every morning after a
weekday, Tuesday (day 2) through Saturday (day 6):
```plaintext
15 04 * * 2-6 gitlab-ctl backup-etc && cd /etc/gitlab/config_backup && cp $(ls -t | head -n1) /secret/gitlab/backups/
```
NOTE:
Make sure that `/secret/gitlab/backups/` exists.
You can extract the tar file as follows.
```shell
# Rename the existing /etc/gitlab, if any
sudo mv /etc/gitlab /etc/gitlab.$(date +%s)
# Change the example timestamp below for your configuration backup
sudo tar -xf gitlab_config_1487687824_2017_02_21.tar -C /
```
Remember to run `sudo gitlab-ctl reconfigure` after restoring a configuration
backup.
NOTE:
Your machines SSH host keys are stored in a separate location at `/etc/ssh/`. Be sure to also [backup and restore those keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079) to avoid man-in-the-middle attack warnings if you have to perform a full machine restore.
### Limit backup lifetime for configuration backups (prune old backups)
GitLab configuration backups can be pruned using the same `backup_keep_time` setting that is
[used for the GitLab application backups](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#limit-backup-lifetime-for-local-files-prune-old-backups)
To make use of this setting, edit `/etc/gitlab/gitlab.rb`:
```ruby
## Limit backup lifetime to 7 days - 604800 seconds
gitlab_rails['backup_keep_time'] = 604800
```
The default `backup_keep_time` setting is `0` - which keeps all GitLab configuration and application backups.
Once a `backup_keep_time` is set - you can run `sudo gitlab-ctl backup-etc --delete-old-backups` to prune all
backups older than the current time minus the `backup_keep_time`.
You can provide the parameter `--no-delete-old-backups` if you want to keep all existing backups.
WARNING:
If no parameter is provided the default is `--delete-old-backups`, which will delete any backups
older than the current time minus the `backup_keep_time`, if `backup_keep_time` is greater than 0.
### Separate configuration backups from application data
Do not store your GitLab application backups (Git repositories, SQL
data) in the same place as your configuration backup (`/etc/gitlab`).
The `gitlab-secrets.json` file (and possibly also the `gitlab.rb`
file) contain database encryption keys to protect sensitive data
in the SQL database:
- GitLab two-factor authentication (2FA) user secrets ('QR codes')
- GitLab CI 'secure variables'
If you separate your configuration backup from your application data backup,
you reduce the chance that your encrypted application data will be
lost/leaked/stolen together with the keys needed to decrypt it.
## Creating an application backup
To create a backup of your repositories and GitLab metadata, follow the
[backup create documentation](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html).
Backup create will store a tar file in `/var/opt/gitlab/backups`.
If you want to store your GitLab backups in a different directory, add the
following setting to `/etc/gitlab/gitlab.rb` and run `sudo gitlab-ctl
reconfigure`:
```ruby
gitlab_rails['backup_path'] = '/mnt/backups'
```
## Creating backups for GitLab instances in Docker containers
WARNING:
The backup command requires [additional parameters](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#back-up-and-restore-for-installations-using-pgbouncer)
when your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster.
Backups can be scheduled on the host by prepending `docker exec -t <your container name>` to the commands.
Backup application:
```shell
docker exec -t <your container name> gitlab-backup
```
Backup configuration and secrets:
```shell
docker exec -t <your container name> /bin/sh -c 'gitlab-ctl backup-etc && cd /etc/gitlab/config_backup && cp $(ls -t | head -n1) /secret/gitlab/backups/'
```
NOTE:
To persist these backups outside the container, mount volumes in the following directories:
1. `/secret/gitlab/backups`.
1. `/var/opt/gitlab` for [all application data](https://docs.gitlab.com/ee/install/docker.html#set-up-the-volumes-location), which includes backups.
1. `/var/opt/gitlab/backups` (optional). The `gitlab-backup` tool writes to this directory [by default](#creating-an-application-backup).
While this directory is nested inside `/var/opt/gitlab`, [Docker sorts these mounts](https://github.com/moby/moby/pull/8055), allowing them to work in harmony.
This configuration enables, for example:
- Application data on regular local storage (through the second mount).
- A backup volume on network storage (through the third mount).
## Restoring an application backup
See [restore documentation](https://docs.gitlab.com/ee/administration/backup_restore/restore_gitlab.html).
## Backup and restore using non-packaged database
If you are using non-packaged database see [documentation on using non-packaged database](database.md#using-a-non-packaged-postgresql-database-management-server).
## Upload backups to remote (cloud) storage
For details check [backup documentation](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#upload-backups-to-a-remote-cloud-storage).
## Manually manage backup directory
Linux package installations create the backup directory set with `gitlab_rails['backup_path']`. The directory is owned by the user that is running GitLab and it has strict permissions set to be accessible to only that user.
That directory will hold backup archives and they contain sensitive information.
In some organizations permissions need to be different because of, for example, shipping the backup archives offsite.
To disable backup directory management, in `/etc/gitlab/gitlab.rb` set:
```ruby
gitlab_rails['manage_backup_path'] = false
```
WARNING:
If you set this configuration option, it is up to you to create the directory specified in `gitlab_rails['backup_path']` and to set permissions
which will allow user specified in `user['username']` to have correct access. Failing to do so will prevent GitLab from creating the backup archive.
doc/settings/configuration.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Configuration options for Linux package installations
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
To configure GitLab, set the relevant options in the `/etc/gitlab/gitlab.rb` file.
[`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template)
contains a complete list of available options. New installations have all the
options of the template listed in `/etc/gitlab/gitlab.rb` by default.
NOTE:
The examples provided when you edit `/etc/gitlab/gitlab.rb` might not always reflect the default settings for an instance.
For a list of default settings, see the
[package defaults](https://docs.gitlab.com/ee/administration/package_information/defaults.html).
## Configure the external URL for GitLab
To display the correct repository clone links to your users,
you must provide GitLab with the URL your users use to reach the repository.
You can use the IP of your server, but a Fully Qualified Domain Name (FQDN)
is preferred. See the [DNS documentation](dns.md)
for more details about the use of DNS in a self-managed GitLab instance.
To change the external URL:
1. Optional. Before you change the external URL, determine if you have previously
defined a [custom **Home page URL** or **After sign-out path**](https://docs.gitlab.com/ee/administration/settings/sign_in_restrictions.html#sign-in-information).
Both of these settings might cause unintentional redirecting after configuring
a new external URL. If you have defined any URLs, remove them completely.
1. Edit `/etc/gitlab/gitlab.rb` and change `external_url` to your preferred URL:
```ruby
external_url "http://gitlab.example.com"
```
Alternatively, you can use the IP address of your server:
```ruby
external_url "http://10.0.0.1"
```
In the previous examples we use plain HTTP. If you want to use HTTPS, see
how to [configure SSL](ssl/index.md).
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
1. Optional. If you had been using GitLab for a while, after you change the
external URL, you should also
[invalidate the Markdown cache](https://docs.gitlab.com/ee/administration/invalidate_markdown_cache.html).
### Specify the external URL at the time of installation
If you use the Linux package, you can set up your GitLab instance
with the minimum number of commands by using the `EXTERNAL_URL` environment variable.
If this variable is set, it is automatically detected and its value is written
as `external_url` in the `gitlab.rb` file.
The `EXTERNAL_URL` environment variable affects only the installation and upgrade
of packages. For regular reconfigure runs, the value
in `/etc/gitlab/gitlab.rb` is used.
As part of package updates, if you have `EXTERNAL_URL` variable set
inadvertently, it replaces the existing value in `/etc/gitlab/gitlab.rb`
without any warning. So, we recommended not to set the variable globally, but
rather pass it specifically to the installation command:
```shell
sudo EXTERNAL_URL="https://gitlab.example.com" apt-get install gitlab-ee
```
## Configure a relative URL for GitLab
NOTE:
For self-compiled (source) installations, there is a
[separate document](https://docs.gitlab.com/ee/install/relative_url.html).
While we recommended installing GitLab in its own (sub)domain, sometimes
it is not possible. In that case, GitLab can also
be installed under a relative URL, for example, `https://example.com/gitlab`.
By changing the URL, all remote URLs change as well, so you must
manually edit them in any local repository that points to your GitLab instance.
To enable relative URL in GitLab:
1. Set the `external_url` in `/etc/gitlab/gitlab.rb`:
```ruby
external_url "https://example.com/gitlab"
```
In this example, the relative URL under which GitLab is served is
`/gitlab`. Change it to your liking.
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
If you have any issues, see the [troubleshooting section](#relative-url-troubleshooting).
## Load external configuration file from non-root user
Linux package installations load all configuration from `/etc/gitlab/gitlab.rb` file.
This file has strict file permissions and is owned by the `root` user. The reason for strict permissions
and ownership is that `/etc/gitlab/gitlab.rb` is being executed as Ruby code by the `root` user during `gitlab-ctl reconfigure`. This means
that users who have write access to `/etc/gitlab/gitlab.rb` can add a configuration that is executed as code by `root`.
In certain organizations, it is allowed to have access to the configuration files but not as the root user.
You can include an external configuration file inside `/etc/gitlab/gitlab.rb` by specifying the path to the file:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
from_file "/home/admin/external_gitlab.rb"
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
When you use `from_file`:
- Code you include into `/etc/gitlab/gitlab.rb` using `from_file` runs with
`root` privileges when you reconfigure GitLab.
- Any configuration that is set in `/etc/gitlab/gitlab.rb` after `from_file` is
included, takes precedence over the configuration from the included file.
## Read certificate from file
Certificates can be stored as separate files and loaded into memory when running `sudo gitlab-ctl reconfigure`. Files containing
certificates must be plaintext.
In this example, the [PostgreSQL server certificate](database.md#configuring-ssl) is read directly from a file rather than copying and pasting into `/etc/gitlab/gitlab.rb` directly.
```ruby
postgresql['internal_certificate'] = File.read('/path/to/server.crt')
```
## Store Git data in an alternative directory
By default, Linux package installations store the Git repository data under
`/var/opt/gitlab/git-data`. The repositories are stored in a subfolder called
`repositories`.
To change the location of the `git-data` parent directory:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
git_data_dirs({ "default" => { "path" => "/mnt/nas/git-data" } })
```
You can also add more than one Git data directory:
```ruby
git_data_dirs({
"default" => { "path" => "/var/opt/gitlab/git-data" },
"alternative" => { "path" => "/mnt/nas/git-data" }
})
```
The target directories and any of its subpaths must not be a symlink.
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
1. Optional. If you already have existing Git repositories in `/var/opt/gitlab/git-data`, you
can move them to the new location:
1. Prevent users from writing to the repositories while you move them:
```shell
sudo gitlab-ctl stop
```
1. Sync the repositories to the new location. Note there is _no_ slash behind
`repositories`, but there _is_ a slash behind `git-data`:
```shell
sudo rsync -av --delete /var/opt/gitlab/git-data/repositories /mnt/nas/git-data/
```
1. Reconfigure to start the necessary processes and fix any wrong permissions:
```shell
sudo gitlab-ctl reconfigure
```
1. Double-check the directory layout in `/mnt/nas/git-data/`. The expected output
should be `repositories`:
```shell
sudo ls /mnt/nas/git-data/
```
1. Start GitLab and verify that you can browse through the repositories in
the web interface:
```shell
sudo gitlab-ctl start
```
If you're running Gitaly on a separate server, remember to also include the
`gitaly_address` for each Git data directory. See
[the documentation on configuring Gitaly](https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#configure-gitaly-clients).
If you're not looking to move all repositories, but instead want to move specific
projects between existing repository storages, use the
[Edit Project API](https://docs.gitlab.com/ee/api/projects.html#edit-project)
endpoint and specify the `repository_storage` attribute.
## Change the name of the Git user or group
WARNING:
We do not recommend changing the user or group of an existing installation because it can cause unpredictable side effects.
By default, Linux package installations use the user name `git` for Git GitLab Shell login,
ownership of the Git data itself, and SSH URL generation on the web interface.
Similarly, the `git` group is used for group ownership of the Git data.
To change the user and group on a new Linux package installation:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
user['username'] = "gitlab"
user['group'] = "gitlab"
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
If you are changing the username of an existing installation, the reconfigure run
doesn't change the ownership of the nested directories, so you must do that manually.
At the very least, you must change ownership of the repositories and uploads
directories:
```shell
sudo chown -R gitlab:gitlab /var/opt/gitlab/git-data/repositories
sudo chown -R gitlab:gitlab /var/opt/gitlab/gitlab-rails/uploads
```
## Specify numeric user and group identifiers
Linux package installations create users for GitLab, PostgreSQL, Redis, NGINX, etc. To
specify the numeric identifiers for these users:
1. Write down the old user and group identifiers, as you might need them later:
```shell
sudo cat /etc/passwd
```
1. Edit `/etc/gitlab/gitlab.rb` and change any of the identifiers you want:
```ruby
user['uid'] = 1234
user['gid'] = 1234
postgresql['uid'] = 1235
postgresql['gid'] = 1235
redis['uid'] = 1236
redis['gid'] = 1236
web_server['uid'] = 1237
web_server['gid'] = 1237
registry['uid'] = 1238
registry['gid'] = 1238
mattermost['uid'] = 1239
mattermost['gid'] = 1239
prometheus['uid'] = 1240
prometheus['gid'] = 1240
```
1. Stop, reconfigure, and then start GitLab:
```shell
sudo gitlab-ctl stop
sudo gitlab-ctl reconfigure
sudo gitlab-ctl start
```
1. Optional. If you're changing `user['uid']` and `user['gid']`, make sure to update the uid/guid of any files not managed by the Linux package
directly, for example, the logs:
```shell
find /var/log/gitlab -uid <old_uid> | xargs -I:: chown git ::
find /var/log/gitlab -gid <old_uid> | xargs -I:: chgrp git ::
find /var/opt/gitlab -uid <old_uid> | xargs -I:: chown git ::
find /var/opt/gitlab -gid <old_uid> | xargs -I:: chgrp git ::
```
## Disable user and group account management
By default, Linux package installations create system user and group accounts,
as well as keeping the information updated.
These system accounts run various components of the package.
Most users don't need to change this behavior.
However, if your system accounts are managed by other software, for example, LDAP, you
might need to disable account management done by the GitLab package.
By default, the Linux package installations expect the following users and groups to exist:
| Linux user and group | Required | Description | Default home directory | Default shell |
| -------------------- | --------------------------------------- | --------------------------------------------------------------------- | ---------------------------- | ------------- |
| `git` | Yes | GitLab user/group | `/var/opt/gitlab` | `bin/sh` |
| `gitlab-www` | Yes | Web server user/group | `/var/opt/gitlab/nginx` | `/bin/false` |
| `gitlab-prometheus` | Yes | Prometheus user/group for Prometheus monitoring and various exporters | `/var/opt/gitlab/prometheus` | `/bin/sh` |
| `gitlab-redis` | Only when using the packaged Redis | Redis user/group for GitLab | `/var/opt/gitlab/redis` | `/bin/false` |
| `gitlab-psql` | Only when using the packaged PostgreSQL | PostgreSQL user/group | `/var/opt/gitlab/postgresql` | `/bin/sh` |
| `gitlab-consul` | Only when using GitLab Consul | GitLab Consul user/group | `/var/opt/gitlab/consul` | `/bin/sh` |
| `registry` | Only when using GitLab Registry | GitLab Registry user/group | `/var/opt/gitlab/registry` | `/bin/sh` |
| `mattermost` | Only when using GitLab Mattermost | GitLab Mattermost user/group | `/var/opt/gitlab/mattermost` | `/bin/sh` |
| `gitlab-backup` | Only when using `gitlab-backup-cli` | GitLab Backup Cli User | `/var/opt/gitlab/backups` | `/bin/sh` |
To disable user and group accounts management:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
manage_accounts['enable'] = false
```
1. Optional. You can also use different user/group names, but then you must specify the user/group details:
```ruby
# GitLab
user['username'] = "git"
user['group'] = "git"
user['shell'] = "/bin/sh"
user['home'] = "/var/opt/custom-gitlab"
# Web server
web_server['username'] = 'webserver-gitlab'
web_server['group'] = 'webserver-gitlab'
web_server['shell'] = '/bin/false'
web_server['home'] = '/var/opt/gitlab/webserver'
# Prometheus
prometheus['username'] = 'gitlab-prometheus'
prometheus['group'] = 'gitlab-prometheus'
prometheus['shell'] = '/bin/sh'
prometheus['home'] = '/var/opt/gitlab/prometheus'
# Redis (not needed when using external Redis)
redis['username'] = "redis-gitlab"
redis['group'] = "redis-gitlab"
redis['shell'] = "/bin/false"
redis['home'] = "/var/opt/redis-gitlab"
# Postgresql (not needed when using external Postgresql)
postgresql['username'] = "postgres-gitlab"
postgresql['group'] = "postgres-gitlab"
postgresql['shell'] = "/bin/sh"
postgresql['home'] = "/var/opt/postgres-gitlab"
# Consul
consul['username'] = 'gitlab-consul'
consul['group'] = 'gitlab-consul'
consul['dir'] = "/var/opt/gitlab/registry"
# Registry
registry['username'] = "registry"
registry['group'] = "registry"
registry['dir'] = "/var/opt/gitlab/registry"
# Mattermost
mattermost['username'] = 'mattermost'
mattermost['group'] = 'mattermost'
mattermost['home'] = '/var/opt/gitlab/mattermost'
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Move the home directory for a user
For the GitLab user, we recommended that the home directory
is set on a local disk and not on shared storage like NFS, for better performance. When setting it in
NFS, Git requests must make another network request to read the Git
configuration and this increases latency in Git operations.
To move an existing home directory, GitLab services need to be stopped and some downtime is required:
1. Stop GitLab:
```shell
sudo gitlab-ctl stop
```
1. Stop the runit server:
```shell
sudo systemctl stop gitlab-runsvdir
```
1. Change the home directory:
```shell
sudo usermod -d /path/to/home <username>
```
If you had existing data, you need to manually copy/rsync it to the new location:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
user['home'] = "/var/opt/custom-gitlab"
```
1. Start the runit server:
```shell
sudo systemctl start gitlab-runsvdir
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Disable storage directories management
The Linux package takes care of creating all the necessary directories
with the correct ownership and permissions, as well as keeping this updated.
Some of the directories hold large amounts of data so in certain setups,
those directories are most likely mounted on an NFS (or some other) share.
Some types of mounts don't allow the automatic creation of directories by the root user
(default user for initial setup), for example, NFS with `root_squash` enabled on the
share. To work around this, the Linux package attempts to create
those directories using the directory's owner user.
### Disable the `/etc/gitlab` directory management
If you have the `/etc/gitlab` directory mounted, you can turn off the management of
that directory:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
manage_storage_directories['manage_etc'] = false
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
### Disable the `/var/opt/gitlab` directory management
If you are mounting all GitLab storage directories, each on a separate mount,
you should completely disable the management of storage directories.
Linux package installations expect these directories to exist
on the file system. It is up to you to create and set correct
permissions if this setting is set.
Enabling this setting prevents the creation of the following directories:
| Default location | Permissions | Ownership | Purpose |
|--------------------------------------------------------|---------------|------------------|------------------------------------|
| `/var/opt/gitlab/git-data` | `2770` | `git:git` | Holds repositories directory |
| `/var/opt/gitlab/git-data/repositories` | `2770` | `git:git` | Holds Git repositories |
| `/var/opt/gitlab/gitlab-rails/shared` | `0751` | `git:gitlab-www` | Holds large object directories |
| `/var/opt/gitlab/gitlab-rails/shared/artifacts` | `0700` | `git:git` | Holds CI artifacts |
| `/var/opt/gitlab/gitlab-rails/shared/external-diffs` | `0700` | `git:git` | Holds external merge request diffs |
| `/var/opt/gitlab/gitlab-rails/shared/lfs-objects` | `0700` | `git:git` | Holds LFS objects |
| `/var/opt/gitlab/gitlab-rails/shared/packages` | `0700` | `git:git` | Holds package repository |
| `/var/opt/gitlab/gitlab-rails/shared/dependency_proxy` | `0700` | `git:git` | Holds dependency proxy |
| `/var/opt/gitlab/gitlab-rails/shared/terraform_state` | `0700` | `git:git` | Holds terraform state |
| `/var/opt/gitlab/gitlab-rails/shared/ci_secure_files` | `0700` | `git:git` | Holds uploaded secure files |
| `/var/opt/gitlab/gitlab-rails/shared/pages` | `0750` | `git:gitlab-www` | Holds user pages |
| `/var/opt/gitlab/gitlab-rails/uploads` | `0700` | `git:git` | Holds user attachments |
| `/var/opt/gitlab/gitlab-ci/builds` | `0700` | `git:git` | Holds CI build logs |
| `/var/opt/gitlab/.ssh` | `0700` | `git:git` | Holds authorized keys |
To disable the management of storage directories:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
manage_storage_directories['enable'] = false
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Start Linux package installation services only after a given file system is mounted
If you want to prevent Linux package installation services (NGINX, Redis, Puma, etc.)
from starting before a given file system is mounted, you can set the
`high_availability['mountpoint']` setting:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
# wait for /var/opt/gitlab to be mounted
high_availability['mountpoint'] = '/var/opt/gitlab'
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
NOTE:
If the mount point doesn't exist, GitLab fails to reconfigure.
## Configure the runtime directory
When Prometheus monitoring is enabled, the GitLab Exporter conducts measurements
of each Puma process (Rails metrics). Every Puma process needs to write
a metrics file to a temporary location for each controller request.
Prometheus then collects all these files and processes their values.
To avoid creating disk I/O, the Linux package uses a
runtime directory.
During `reconfigure`, the package check if `/run` is a `tmpfs` mount.
If it is not, the following warning is shown and Rails metrics are disabled:
```plaintext
Runtime directory '/run' is not a tmpfs mount.
```
To enable the Rails metrics again:
1. Edit `/etc/gitlab/gitlab.rb` to create a `tmpfs` mount
(note that there is no `=` in the configuration):
```ruby
runtime_dir '/path/to/tmpfs'
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Configure a failed authentication ban
You can configure a
[failed authentication ban](https://docs.gitlab.com/ee/security/rate_limits.html#failed-authentication-ban-for-git-and-container-registry)
for Git and the container registry. When a client is banned, a 403 error code
is returned.
The following settings can be configured:
| Setting | Description |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled` | `false` by default. Set this to `true` to enable the Git and registry authentication ban. |
| `ip_whitelist` | IPs to not block. They must be formatted as strings in a Ruby array. You can use either single IPs or CIDR notation, for example, `["127.0.0.1", "127.0.0.2", "127.0.0.3", "192.168.0.1/24"]`. |
| `maxretry` | The maximum amount of times a request can be made in the specified time. |
| `findtime` | The maximum amount of time in seconds that failed requests can count against an IP before it's added to the denylist. |
| `bantime` | The total amount of time in seconds that an IP is blocked. |
To configure the Git and container registry authentication ban:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['rack_attack_git_basic_auth'] = {
'enabled' => true,
'ip_whitelist' => ["127.0.0.1"],
'maxretry' => 10, # Limit the number of Git HTTP authentication attempts per IP
'findtime' => 60, # Reset the auth attempt counter per IP after 60 seconds
'bantime' => 3600 # Ban an IP for one hour (3600s) after too many auth attempts
}
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Disable automatic cache cleaning during installation
If you have a large GitLab installation, you might not want to run a `rake cache:clear` task
as it can take a long time to finish. By default, the cache clear task runs automatically
during reconfiguring.
To disable automatic cache cleaning during installation:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
# This is an advanced feature used by large gitlab deployments where loading
# whole RAILS env takes a lot of time.
gitlab_rails['rake_cache_clear'] = false
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Error Reporting and Logging with Sentry
WARNING:
From GitLab 17.0, only Sentry versions 21.5.0 or later will be supported. If
you use an earlier version of a Sentry instance that you host, you must
[upgrade Sentry](https://develop.sentry.dev/self-hosted/releases/) to continue
collecting errors from your GitLab environments.
Sentry is an open source error reporting and logging tool which can be used as
SaaS (<https://sentry.io>) or [host it yourself](https://develop.sentry.dev/self-hosted/).
To configure Sentry:
1. Create a project in Sentry.
1. Find the
[Data Source Name (DSN)](https://docs.sentry.io/concepts/key-terms/dsn-explainer/)
of the project you created.
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['sentry_enabled'] = true
gitlab_rails['sentry_dsn'] = 'https://<public_key>@<host>/<project_id>' # value used by the Rails SDK
gitlab_rails['sentry_clientside_dsn'] = 'https://<public_key>@<host>/<project_id>' # value used by the Browser JavaScript SDK
gitlab_rails['sentry_environment'] = 'production'
```
The [Sentry environment](https://docs.sentry.io/concepts/key-terms/environments/)
can be used to track errors and issues across several deployed GitLab
environments, for example, lab, development, staging, and production.
1. Optional. To set custom [Sentry tags](https://docs.sentry.io/concepts/key-terms/enrich-data/)
on every event sent from a particular server, the `GITLAB_SENTRY_EXTRA_TAGS`
an environment variable can be set. This variable is a JSON-encoded hash representing any
tags that should be passed to Sentry for all exceptions from that server.
For instance, setting:
```ruby
gitlab_rails['env'] = {
'GITLAB_SENTRY_EXTRA_TAGS' => '{"stage": "main"}'
}
```
Would add the `stage` tag with a value of `main`.
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Set a Content Delivery Network URL
Service static assets with a Content Delivery Network (CDN) or asset host
using `gitlab_rails['cdn_host']`. This configures a [Rails asset host](https://guides.rubyonrails.org/configuring.html#config-asset-host).
To set a CDN/asset host:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['cdn_host'] = 'https://mycdnsubdomain.fictional-cdn.com'
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
Additional documentation for configuring common services to act as an asset host
is tracked in [this issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5708).
## Set a Content Security Policy
Setting a Content Security Policy (CSP) can help thwart JavaScript
cross-site scripting (XSS) attacks. See
[the Mozilla documentation on CSP](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) for more
details.
[CSP and nonce-source with inline JavaScript](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src) is available on GitLab.com.
It is [not configured by default](https://gitlab.com/gitlab-org/gitlab/-/issues/30720) on self-managed.
NOTE:
Improperly configuring the CSP rules could prevent GitLab from working
properly. Before rolling out a policy, you may also want to change
`report_only` to `true` to test the configuration.
To add a CSP:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['content_security_policy'] = {
enabled: true,
report_only: false
}
```
GitLab automatically provides secure default values for the CSP.
Explicitly setting the `<default_value>` value for a directive is equivalent to
not setting a value and will use the default values.
To add a custom CSP:
```ruby
gitlab_rails['content_security_policy'] = {
enabled: true,
report_only: false,
directives: {
default_src: "'none'",
script_src: "https://example.com"
}
}
```
Secure default values are used for directives that aren't explicitly configured.
To unset a CSP directive, set a value of `false`.
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Set initial root password on installation
The initial password for the administrator user `root` can be set at installation time. For more information, see
[Set up the initial password](../installation/index.md#set-up-the-initial-account).
## Set allowed hosts to prevent host header attacks
To prevent GitLab from accepting a host header other than
what's intended:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['allowed_hosts'] = ['gitlab.example.com']
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
There are no known security issues in GitLab caused by not configuring `allowed_hosts`,
but it's recommended for defense in depth against potential [HTTP Host header attacks](https://portswigger.net/web-security/host-header).
If using a custom external proxy such as Apache, it may be necessary to add the localhost address or name (`localhost` or `127.0.0.1`). You should add filters to the external proxy to mitigate potential HTTP Host header attacks passed through the proxy to workhorse.
```ruby
gitlab_rails['allowed_hosts'] = ['gitlab.example.com', '127.0.0.1', 'localhost']
```
## Session cookie configuration
To change the prefix of the generated web session cookie values:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['session_store_session_cookie_token_prefix'] = 'custom_prefix_'
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
The default value is an empty string `""`.
## Provide sensitive configuration to components without plain text storage
Some components expose an `extra_config_command` option in `gitlab.rb`. This allows an external script to provide secrets
dynamically rather than read them from plain text storage.
The available options are:
| `gitlab.rb` setting | Responsibility |
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `redis['extra_config_command']` | Provides extra configuration to the Redis server configuration file. |
| `gitlab_rails['redis_extra_config_command']` | Provides extra configuration to the Redis configuration files used by GitLab Rails application. (`resque.yml`, `redis.yml`, `redis.<redis_instance>.yml` files) |
| `gitlab_rails['db_extra_config_command']` | Provides extra configuration to the DB configuration file used by GitLab Rails application. (`database.yml`) |
| `gitlab_kas['extra_config_command']` | Provides extra configuration to GitLab agent server for Kubernetes (KAS). |
| `gitlab_workhorse['extra_config_command']` | Provides extra configuration to GitLab Workhorse. |
| `gitlab_exporter['extra_config_command']` | Provides extra configuration to GitLab Exporter. |
The value assigned to any of these options should be an absolute path to an executable script
that writes the sensitive configuration in the required format to STDOUT. The
components:
1. Execute the supplied script.
1. Replace values set by user and default configuration files with those emitted
by the script.
### Provide Redis password to Redis server and client components
As an example, you can use the script and `gitlab.rb` snippet below to specify
the password to Redis server and components that need to connect to Redis.
NOTE:
When specifying password to Redis server, this method only saves the user from
having the plaintext password in `gitlab.rb` file. The password will end up in
plaintext in the Redis server configuration file present at
`/var/opt/gitlab/redis/redis.conf`.
1. Save the script below as `/opt/generate-redis-conf`
```ruby
#!/opt/gitlab/embedded/bin/ruby
require 'json'
require 'yaml'
class RedisConfig
REDIS_PASSWORD = `echo "toomanysecrets"`.strip # Change the command inside backticks to fetch Redis password
class << self
def server
puts "requirepass '#{REDIS_PASSWORD}'"
puts "masterauth '#{REDIS_PASSWORD}'"
end
def rails
puts YAML.dump({
'password' => REDIS_PASSWORD
})
end
def kas
puts YAML.dump({
'redis' => {
'password' => REDIS_PASSWORD
}
})
end
def workhorse
puts JSON.dump({
redis: {
password: REDIS_PASSWORD
}
})
end
def gitlab_exporter
puts YAML.dump({
'probes' => {
'sidekiq' => {
'opts' => {
'redis_password' => REDIS_PASSWORD
}
}
}
})
end
end
end
def print_error_and_exit
$stdout.puts "Usage: generate-redis-conf <COMPONENT>"
$stderr.puts "Supported components are: server, rails, kas, workhorse, gitlab_exporter"
exit 1
end
print_error_and_exit if ARGV.length != 1
component = ARGV.shift
begin
RedisConfig.send(component.to_sym)
rescue NoMethodError
print_error_and_exit
end
```
1. Ensure the script created above is executable:
```shell
chmod +x /opt/generate-redis-conf
```
1. Add the snippet below to `/etc/gitlab/gitlab.rb`:
```ruby
redis['extra_config_command'] = '/opt/generate-redis-conf server'
gitlab_rails['redis_extra_config_command'] = '/opt/generate-redis-conf rails'
gitlab_workhorse['extra_config_command'] = '/opt/generate-redis-conf workhorse'
gitlab_kas['extra_config_command'] = '/opt/generate-redis-conf kas'
gitlab_exporter['extra_config_command'] = '/opt/generate-redis-conf gitlab_exporter'
```
1. Run `sudo gitlab-ctl reconfigure`.
### Provide the PostgreSQL user password to GitLab Rails
As an example, you can use the script and configuration below to provide the
password that GitLab Rails should use to connect to the PostgreSQL server.
1. Save the script below as `/opt/generate-db-config`:
```ruby
#!/opt/gitlab/embedded/bin/ruby
require 'yaml'
db_password = `echo "toomanysecrets"`.strip # Change the command inside backticks to fetch DB password
puts YAML.dump({
'main' => {
'password' => db_password
},
'ci' => {
'password' => db_password
}
})
```
1. Ensure the script created above is executable:
```shell
chmod +x /opt/generate-db-config
```
1. Add the snippet below to `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['db_extra_config_command'] = '/opt/generate-db-config'
```
1. Run `sudo gitlab-ctl reconfigure`.
## Related topics
- [Disable impersonation](https://docs.gitlab.com/ee/api/index.html#disable-impersonation)
- [Set up LDAP sign-in](https://docs.gitlab.com/ee/administration/auth/ldap/index.html)
- [Smartcard authentication](https://docs.gitlab.com/ee/administration/auth/smartcard.html)
- [Set up NGINX](nginx.md) for things like:
- Set up HTTPS
- Redirect `HTTP` requests to `HTTPS`
- Change the default port and the SSL certificate locations
- Set the NGINX listen-address or addresses
- Insert custom NGINX settings into the GitLab server block
- Insert custom settings into the NGINX configuration
- Enable `nginx_status`
- [Use a non-packaged web-server](nginx.md#using-a-non-bundled-web-server)
- [Use a non-packaged PostgreSQL database management server](database.md)
- [Use a non-packaged Redis instance](redis.md)
- [Add `ENV` vars to the GitLab runtime environment](environment-variables.md)
- [Changing `gitlab.yml` and `application.yml` settings](gitlab.yml.md)
- [Send application email via SMTP](smtp.md)
- [Set up OmniAuth (Google, Twitter, GitHub login)](https://docs.gitlab.com/ee/integration/omniauth.html)
- [Adjust Puma settings](https://docs.gitlab.com/ee/administration/operations/puma.html)
## Troubleshooting
### Relative URL troubleshooting
If you notice any issues with GitLab assets appearing broken after moving to a
relative URL configuration (like missing images or unresponsive components),
please raise an issue in [GitLab](https://gitlab.com/gitlab-org/gitlab)
with the `Frontend` label.
### `Mixlib::ShellOut::ShellCommandFailed: linux_user[GitLab user and group]`
When [moving the home directory for a user](#move-the-home-directory-for-a-user),
if the runit service is not stopped and the home directories are not manually
moved for the user, GitLab will encounter an error while reconfiguring:
```plaintext
account[GitLab user and group] (gitlab::users line 28) had an error: Mixlib::ShellOut::ShellCommandFailed: linux_user[GitLab user and group] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/account.rb line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '8'
---- Begin output of ["usermod", "-d", "/var/opt/gitlab", "git"] ----
STDOUT:
STDERR: usermod: user git is currently used by process 1234
---- End output of ["usermod", "-d", "/var/opt/gitlab", "git"] ----
Ran ["usermod", "-d", "/var/opt/gitlab", "git"] returned 8
```
Make sure to stop `runit` before moving the home directory.
### GitLab responds with 502 after changing the name of the Git user or group
If you changed the [name of the Git user or group](#change-the-name-of-the-git-user-or-group)
on an existing installation, this can cause many side effects.
You can check for errors that relate to files unable to access and try to
fix their permissions:
```shell
gitlab gitlab-ctl tail -f
```
doc/settings/database.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Database settings
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
GitLab supports only the PostgreSQL database management system.
Thus you have two options for database servers to use with a Linux package installation:
- Use the packaged PostgreSQL server included with the Linux package installation (no
configuration required, recommended).
- Use an [external PostgreSQL server](#using-a-non-packaged-postgresql-database-management-server).
## Using the PostgreSQL database service shipped with the Linux package
### Reconfigure and PostgreSQL restarts
Linux package installations normally restart any service on reconfigure if configuration settings for that service were
changed in the `gitlab.rb` file. PostgreSQL is unique in that some of its settings take effect
with a reload (HUP), while others require PostgreSQL to be restarted. Because administrators
frequently want more control over exactly when PostgreSQL is restarted, Linux package installations are configured
to do a reload of PostgreSQL on reconfigure, and not a restart. This means that if you modify any
PostgreSQL setting that requires a restart, you will need to restart PostgreSQL manually after you
reconfigure.
The [GitLab config template](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template)
identifies which PostgreSQL settings require a restart and which require only a reload. You can also
run a query against your database to determine if any individual setting requires a restart. Start a
database console with `sudo gitlab-psql`, then replace `<setting name>` in the following query
with the setting you are changing:
```sql
SELECT name,setting FROM pg_settings WHERE context = 'postmaster' AND name = '<setting name>';
```
If changing the setting will require a restart, the query will return the name of the setting and the current value
of that setting in the running PostgreSQL instance.
#### Automatic restart when the PostgreSQL version changes
By default, Linux package installations automatically restart PostgreSQL when the underlying
version changes, as suggested by the [upstream documentation](https://www.postgresql.org/docs/14/upgrading.html).
This behavior can be controlled using the `auto_restart_on_version_change` setting
available for `postgresql` and `geo-postgresql`.
To disable automatic restarts when the PostgreSQL version changes:
1. Edit `/etc/gitlab/gitlab.rb` and add the following line:
```ruby
# For PostgreSQL/Patroni
postgresql['auto_restart_on_version_change'] = false
# For Geo PostgreSQL
geo_postgresql['auto_restart_on_version_change'] = false
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
NOTE:
It is highly recommended to restart PostgreSQL when the underlying version changes,
to avoid errors like the [one related to loading necessary libraries](#could-not-load-library-plpgsqlso).
### Configuring SSL
Linux package installations automatically enable SSL on the PostgreSQL server, but it will accept
both encrypted and unencrypted connections by default. Enforcing SSL requires
using the `hostssl` configuration in `pg_hba.conf`. For more details, see the
[`pg_hba.conf` documentation](https://www.postgresql.org/docs/14/auth-pg-hba-conf.html).
SSL support depends on the following files:
- The public SSL certificate for the database (`server.crt`).
- The corresponding private key for the SSL certificate (`server.key`).
- A root certificate bundle that validates the server's certificate (`root.crt`).
By default, Linux package installations use the embedded certificate bundle in
`/opt/gitlab/embedded/ssl/certs/cacert.pem`. This isn't required for
self-signed certificates.
A 10-year self-signed certificate and private key are generated by a Linux package installation for use. If you'd
prefer to use a CA-signed certificate or replace this with your own self-signed certificate, use the following steps.
Note that the location of these files can be configurable, but the private key
_must_ be readable by the `gitlab-psql` user. Linux package installations manage the permissions of
the files for you, but if the paths are customized, you *must* ensure that the
`gitlab-psql` can access the directory in the files are placed in.
For more details, see the [PostgreSQL documentation](https://www.postgresql.org/docs/11/ssl-tcp.html).
Note that `server.crt` and `server.key` may be different from the default SSL
certificates used to access GitLab. For example, suppose the external hostname
of your database is `database.example.com`, and your external GitLab hostname
is `gitlab.example.com`. You will either need a wildcard certificate for
`*.example.com` or two different SSL certificates.
The `ssl_cert_file`, `ssl_key_file`, and `ssl_ca_file` files direct PostgreSQL to where
on the filesystem to find the certificate, key, and bundle. These changes are applied to
`postgresql.conf`. The directives `internal_certificate` and `internal_key` are used to
populate the contents of these files. The contents can be added directly or
loaded from file as shown in the following example.
After you have these files, enable SSL:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
postgresql['ssl_cert_file'] = '/custom/path/to/server.crt'
postgresql['ssl_key_file'] = '/custom/path/to/server.key'
postgresql['ssl_ca_file'] = '/custom/path/to/bundle.pem'
postgresql['internal_certificate'] = File.read('/custom/path/to/server.crt')
postgresql['internal_key'] = File.read('/custom/path/to/server.key')
```
Relative paths will be rooted in the PostgreSQL data directory
(`/var/opt/gitlab/postgresql/data` by default).
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) to apply the configuration changes.
1. Restart PostgreSQL for the changes to take effect:
```shell
gitlab-ctl restart postgresql
```
If PostgreSQL fails to start, check the logs (for example,
`/var/log/gitlab/postgresql/current`) for more details.
#### Require SSL
1. Add the following to `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['db_sslmode'] = 'require'
```
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) to apply the configuration changes.
1. Restart PostgreSQL for the changes to take effect:
```shell
gitlab-ctl restart postgresql
```
If PostgreSQL fails to start, check the logs (for example,
`/var/log/gitlab/postgresql/current`) for more details.
#### Disabling SSL
1. Add the following to `/etc/gitlab/gitlab.rb`:
```ruby
postgresql['ssl'] = 'off'
```
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) to apply the configuration changes.
1. Restart PostgreSQL for the changes to take effect:
```shell
gitlab-ctl restart postgresql
```
If PostgreSQL fails to start, check the logs (for example,
`/var/log/gitlab/postgresql/current`) for more details.
#### Verifying that SSL is being used
To determine whether SSL is being used by clients, you can run:
```shell
sudo gitlab-rails dbconsole --database main
```
At startup, you should see a banner as the following:
```plaintext
psql (13.14)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: on)
Type "help" for help.
```
To determine if clients are using SSL, issue this SQL query:
```sql
SELECT * FROM pg_stat_ssl;
```
For example:
```plaintext
gitlabhq_production=> select * from pg_stat_ssl;
pid | ssl | version | cipher | bits | compression | clientdn
------+-----+---------+------------------------+------+-------------+------------
384 | f | | | | |
386 | f | | | | |
998 | t | TLSv1.3 | TLS_AES_256_GCM_SHA384 | 256 | f | /CN=gitlab
933 | f | | | | |
1003 | t | TLSv1.3 | TLS_AES_256_GCM_SHA384 | 256 | f | /CN=gitlab
1016 | t | TLSv1.3 | TLS_AES_256_GCM_SHA384 | 256 | f | /CN=gitlab
1022 | t | TLSv1.3 | TLS_AES_256_GCM_SHA384 | 256 | f | /CN=gitlab
1211 | t | TLSv1.3 | TLS_AES_256_GCM_SHA384 | 256 | f | /CN=gitlab
1214 | t | TLSv1.3 | TLS_AES_256_GCM_SHA384 | 256 | f | /CN=gitlab
1213 | t | TLSv1.3 | TLS_AES_256_GCM_SHA384 | 256 | f | /CN=gitlab
1215 | t | TLSv1.3 | TLS_AES_256_GCM_SHA384 | 256 | f | /CN=gitlab
1252 | t | TLSv1.3 | TLS_AES_256_GCM_SHA384 | 256 | f |
1280 | t | TLSv1.3 | TLS_AES_256_GCM_SHA384 | 256 | f | /CN=gitlab
382 | f | | | | |
381 | f | | | | |
383 | f | | | | |
(16 rows)
```
1. Rows that have `t` listed under the `ssl` column are enabled.
1. Rows that have a value in the `clientdn` are using the `cert` authentication method
#### Configure SSL client authentication
Client SSL certificates can be used to authenticate to the database server. Creating the certificates
is beyond the scope of `omnibus-gitlab`. But users who have an existing SSL certificate management solution
can use this.
##### Configure the database server
1. Create a certificate and key for the server, the common name should equal the DNS name of the server
1. Copy the server certificate, key, and CA file to the PostgreSQL server, and ensure the permissions are correct
1. The certificate should be owned by the database user (default: `gitlab-psql`)
1. The key file should be owned by the database user, and its permissions should be `0400`
1. The CA file should be owned by the database user, and its permissions should be `0400`
NOTE:
Don't use the file names `server.crt` or `server.key` for these files. These
file names are reserved for the internal use of `omnibus-gitlab`.
1. Ensure the following is set in `gitlab.rb`:
```ruby
postgresql['ssl_cert_file'] = 'PATH_TO_CERTIFICATE'
postgresql['ssl_key_file'] = 'PATH_TO_KEY_FILE'
postgresql['ssl_ca_file'] = 'PATH_TO_CA_FILE'
postgresql['listen_address'] = 'IP_ADDRESS'
postgresql['cert_auth_addresses'] = {
'IP_ADDRESS' => {
'database' => 'gitlabhq_production',
'user' => 'gitlab'
}
```
Set `listen_address` as the IP address of the server that the clients will use
to connect to the database.
Ensure `cert_auth_addresses` contains a list of IP addresses and the
databases and users that are allowed to connect to the database. You can use
CIDR notation when specifying the key for `cert_auth_addresses` to
incorporate an IP address range.
1. Run `gitlab-ctl reconfigure`, and then `gitlab-ctl restart postgresql` for
the new settings to take effect.
#### Configure the Rails client
For the rails client to connect to the server, you will need a certificate and key with the `commonName` set to `gitlab`, which is signed by a certificate authority trusted in the CA file specified in `ssl_ca_file` on the database server.
1. Configure `gitlab.rb`
```ruby
gitlab_rails['db_host'] = 'IP_ADDRESS_OR_HOSTNAME_OF_DATABASE_SERVER'
gitlab_rails['db_sslcert'] = 'PATH_TO_CERTIFICATE_FILE'
gitlab_rails['db_sslkey'] = 'PATH_TO_KEY_FILE'
gitlab_rails['db_rootcert'] = 'PATH_TO_CA_FILE'
```
1. Run `gitlab-ctl reconfigure` for the rails client to use the new settings
1. Follow the steps in [Verifying that SSL is being used](#verifying-that-ssl-is-being-used) to ensure the authentication is working.
### Configure packaged PostgreSQL server to listen on TCP/IP
The packaged PostgreSQL server can be configured to listen for TCP/IP connections,
with the caveat that some non-critical scripts expect UNIX sockets and may misbehave.
To configure the use of TCP/IP for the database service, make changes to both
the `postgresql` and `gitlab_rails` sections of `gitlab.rb`.
#### Configure PostgreSQL block
The following settings are affected in the `postgresql` block:
- `listen_address`: Controls the address on which PostgreSQL will listen.
- `port`: Controls the port on which PostgreSQL listens to. The default is `5432`.
- `md5_auth_cidr_addresses`: A list of CIDR address blocks that are allowed to
connect to the server, after authentication with a password.
- `trust_auth_cidr_addresses`: A list of CIDR address blocks that are allowed
to connect to the server, without authentication of any kind. You should
only set this setting to allow connections from nodes that need to connect,
such as GitLab Rails or Sidekiq. This includes local connections when deployed
on the same node or from components such as Postgres Exporter (`127.0.0.1/32`).
- `sql_user`: Controls the expected username for MD5 authentication. This
defaults to `gitlab`, and isn't a required setting.
- `sql_user_password`: Sets the password that PostgreSQL will accept for MD5
authentication.
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
postgresql['listen_address'] = '0.0.0.0'
postgresql['port'] = 5432
postgresql['md5_auth_cidr_addresses'] = %w()
postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/24)
postgresql['sql_user'] = "gitlab"
##! SQL_USER_PASSWORD_HASH can be generated using the command `gitlab-ctl pg-password-md5 'gitlab'`,
##! where 'gitlab' (single-quoted to avoid shell interpolation) is the name of the SQL user that connects to GitLab.
##! You will be prompted for a password which other clients will use to authenticate with database, such as `securesqlpassword` in the below section.
postgresql['sql_user_password'] = "SQL_USER_PASSWORD_HASH"
# force ssl on all connections defined in trust_auth_cidr_addresses and md5_auth_cidr_addresses
postgresql['hostssl'] = true
```
1. Reconfigure GitLab and restart PostrgreSQL:
```shell
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart postgresql
```
Any client or GitLab service which will connect over the network will need to
provide the values of `sql_user` for the username, and password provided to the
configuration when connecting to the PostgreSQL server. They must also be in the
network block provided to `md5_auth_cidr_addresses`
#### Configure GitLab Rails block
To configure the `gitlab-rails` application to connect to the PostgreSQL database
over the network, several settings must be configured:
- `db_host`: Needs to be set to the IP address of the database server. If this is
on the same instance as the PostgreSQL service, this can be `127.0.0.1` and _will
not require_ password authentication.
- `db_port`: Sets the port on the PostgreSQL server to connect to, and _must be set_
if `db_host` is set.
- `db_username`: Configures the username with which to connect to PostgreSQL. This
defaults to `gitlab`.
- `db_password`: Must be provided if connecting to PostgreSQL over TCP/IP, and from
an instance in the `postgresql['md5_auth_cidr_addresses']` block from settings
above. This is not required if you are connecting to `127.0.0.1` and have configured
`postgresql['trust_auth_cidr_addresses']` to include it.
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['db_host'] = '127.0.0.1'
gitlab_rails['db_port'] = 5432
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['db_password'] = "securesqlpassword"
```
1. Reconfigure GitLab and restart PostrgreSQL:
```shell
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart postgresql
```
#### Apply and restart services
After making the previous changes, an administrator should run `gitlab-ctl reconfigure`.
If you experience any issues in regards to the service not listening on TCP, try
directly restarting the service with `gitlab-ctl restart postgresql`.
Some included scripts of the Linux package (such as `gitlab-psql`) expect the
connections to PostgreSQL to be handled over the UNIX socket, and may not function
properly. You can enable TCP/IP without disabling UNIX sockets.
To test access from other clients, you can run:
```shell
sudo gitlab-rails dbconsole --database main
```
### Enabling PostgreSQL WAL (Write Ahead Log) Archiving
By default, WAL archiving of the packaged PostgreSQL isn't enabled. Consider the
following when seeking to enable WAL archiving:
- The WAL level needs to be 'replica' or higher (9.6+ options are `minimal`,
`replica`, or `logical`)
- Increasing the WAL level will increase the amount of storage consumed in
regular operations
To enable WAL Archiving:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
# Replication settings
postgresql['sql_replication_user'] = "gitlab_replicator"
postgresql['wal_level'] = "replica"
...
...
# Backup/Archive settings
postgresql['archive_mode'] = "on"
postgresql['archive_command'] = "/your/wal/archiver/here"
postgresql['archive_timeout'] = "60"
```
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) for the changes to take effect. This will result in a database restart.
### Store PostgreSQL data in a different directory
By default, everything is stored under `/var/opt/gitlab/postgresql`, controlled
by the `postgresql['dir']` attribute.
This consists of:
- The database socket will be `/var/opt/gitlab/postgresql/.s.PGSQL.5432`. This
is controlled by `postgresql['unix_socket_directory']`.
- The `gitlab-psql` system user will have its `HOME` directory set to this. This
is controlled by `postgresql['home']`.
- The actual data will be stored in `/var/opt/gitlab/postgresql/data`.
To change the location of the PostgreSQL data
WARNING:
If you have an existing database, you need to move the data to the new location
first.
WARNING:
This is an intrusive operation. It cannot be done without downtime on an
existing installation
1. If this is an existing installation, stop GitLab: `gitlab-ctl stop`.
1. Update `postgresql['dir']` to the desired location.
1. Run `gitlab-ctl reconfigure`.
1. Start GitLab `gitlab-ctl start`.
### Upgrade packaged PostgreSQL server
The Linux package provides the `gitlab-ctl pg-upgrade` command to update the
packaged PostgreSQL server to a later version (if one is included in the
package). This updates PostgreSQL to the [default shipped version](https://docs.gitlab.com/ee/administration/package_information/postgresql_versions.html)
during package upgrades, unless specifically [opted out](#opt-out-of-automatic-postgresql-upgrades).
Before upgrading GitLab to a newer version, refer to the [version-specific changes](https://docs.gitlab.com/ee/update/#version-specific-upgrading-instructions)
of the Linux package to see either:
- When a database version has changed.
- When an upgrade is warranted.
WARNING:
Before upgrading, it's important that you fully read this section before running any commands. For
single-node installations, this upgrade needs downtime, as the database must be
down while the upgrade is being performed. The length of time depends on the
size of your database.
NOTE:
If you encounter any problems during the upgrade, raise an issue with a full
description at the [`omnibus-gitlab` issue tracker](https://gitlab.com/gitlab-org/omnibus-gitlab).
To upgrade the PostgreSQL version, be sure that:
- You're running the latest version of GitLab that supports your current version of PostgreSQL.
- If you recently upgraded, you ran `sudo gitlab-ctl reconfigure` successfully
before you proceed.
- You have sufficient disk space for two copies of your database. _Do not attempt
to upgrade unless you have enough free space available._
- Check your database size using `sudo du -sh /var/opt/gitlab/postgresql/data`
(or update your database path).
- Check the space available using `sudo df -h`. If the partition where the
database resides doesn't have enough space, pass the argument `--tmp-dir $DIR`
to the command. The upgrade task includes an available disk space check and aborts
the upgrade if the requirements aren't met.
After you confirm that the above checklist is satisfied, you can proceed with
the upgrade:
```shell
sudo gitlab-ctl pg-upgrade
```
To upgrade to a specific PostgreSQL version, use the `-V` flag to append the
version. For example, to upgrade to PostgreSQL 14:
```shell
sudo gitlab-ctl pg-upgrade -V 14
```
NOTE:
`pg-upgrade` can take arguments; for example, you can set the timeout for the
execution of the underlying commands (`--timeout=1d2h3m4s5ms`). Run `gitlab-ctl pg-upgrade -h`
to see the full list.
`gitlab-ctl pg-upgrade` performs the following steps:
1. Checks to ensure the database is in a known good state.
1. Checks if there's enough free disk space and abort otherwise. You can skip this by appending the `--skip-disk-check` flag.
1. Shuts down the existing database and any unnecessary services, and enables GitLab to deploy page.
1. Changes the symlinks in `/opt/gitlab/embedded/bin/` for PostgreSQL to point to the newer version of the database.
1. Creates a new directory containing a new, empty database with a locale matching the existing database.
1. Uses the `pg_upgrade` tool to copy the data from the old database to the new database.
1. Moves the old database out of the way.
1. Moves the new database to the expected location.
1. Calls `sudo gitlab-ctl reconfigure` to do the required configuration changes and starts the new database server.
1. Runs `ANALYZE` to generate database statistics.
1. Starts the remaining services and removes the deploy page.
1. If any errors are detected during this process, it reverts to the old version of the database.
After the upgrade is complete, verify that everything is working as expected.
If there was an error in the output while running the `ANALYZE` step, your upgrade
will still be working but will have poor database performance until the
database statistics are generated. Use `gitlab-psql` to determine whether `ANALYZE` should be run manually:
```shell
sudo gitlab-psql -c "SELECT relname, last_analyze, last_autoanalyze FROM pg_stat_user_tables WHERE last_analyze IS NULL AND last_autoanalyze IS NULL;"
```
You can run `ANALYZE` manually if the query above returned any rows:
```shell
sudo gitlab-psql -c 'SET statement_timeout = 0; ANALYZE VERBOSE;'
```
_After you have verified that your GitLab instance is running correctly_, you
can clean up the old database files:
```shell
sudo rm -rf /var/opt/gitlab/postgresql/data.<old_version>
sudo rm -f /var/opt/gitlab/postgresql-version.old
```
You can find details of PostgreSQL versions shipped with various GitLab versions
in [PostgreSQL versions shipped with the Linux package](https://docs.gitlab.com/ee/administration/package_information/postgresql_versions.html).
#### Opt out of automatic PostgreSQL upgrades
To opt out of automatic PostgreSQL upgrades during GitLab package upgrades, run:
```shell
sudo touch /etc/gitlab/disable-postgresql-upgrade
```
### Revert packaged PostgreSQL server to the previous version
WARNING:
This operation will revert your current database, _including its data_, to its state
before your last upgrade. Be sure to create a backup before attempting to downgrade
your packaged PostgreSQL database.
On GitLab versions which ship multiple PostgreSQL versions, users can downgrade
an already upgraded PostgreSQL version to the earlier version using the `gitlab-ctl
revert-pg-upgrade` command. This command also supports the `-V` flag to specify
a target version for scenarios where more than two PostgreSQL versions are shipped in
the package (for example: GitLab 12.8 where PostgreSQL 9.6.x, 10.x, and 11.x are
shipped).
To specify a target PostgreSQL version of 12:
```shell
gitlab-ctl revert-pg-upgrade -V 12
```
If the target version is not specified, it will use the version in `/var/opt/gitlab/postgresql-version.old`
if available. Otherwise, it falls back to the default version shipped with GitLab.
On other GitLab versions that ship only one PostgreSQL version, you can't
downgrade your PostgreSQL version. You must downgrade GitLab to an older version for
this.
### Configuring multiple database connections
> - The `gitlab:db:decomposition:connection_status` Rake task was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111927) in GitLab 15.11.
> - Support for single database will be [removed in GitLab 18.0](https://docs.gitlab.com/ee/update/deprecations.html#running-a-single-database-is-deprecated).
In GitLab 16.0, GitLab defaults to using two database connections that point to the same PostgreSQL database.
Before upgrading to GitLab 16.0, check that the PostgreSQL `max_connections` setting is high enough so that more than 50% of available connections show as being unused.
For example, if `max_connections` is set to 100 and you see 75 connections in use, you must increase `max_connections` to at least 150 before upgrading because after
upgrading, the in-use connections will double to 150.
You can verify this by running the following Rake task:
```shell
sudo gitlab-rake gitlab:db:decomposition:connection_status
```
If the task indicates that `max_connections` is high enough, then you can proceed with the upgrade.
## Using a non-packaged PostgreSQL database management server
By default, GitLab is configured to use the PostgreSQL server that's included
in the Linux package. You can also reconfigure it to use an external instance of
PostgreSQL.
WARNING:
If you are using a non-packaged PostgreSQL server, you need to make
sure that PostgreSQL is set up according to the [database requirements document](https://docs.gitlab.com/ee/install/requirements.html#database).
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
# Disable the built-in Postgres
postgresql['enable'] = false
# Fill in the connection details for database.yml
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
gitlab_rails['db_host'] = '127.0.0.1'
gitlab_rails['db_port'] = 5432
gitlab_rails['db_username'] = 'USERNAME'
gitlab_rails['db_password'] = 'PASSWORD'
```
Don't forget to remove the `#` comment characters at the beginning of these
lines.
Note that:
- `/etc/gitlab/gitlab.rb` should have file permissions `0600` because it contains
plain-text passwords.
- PostgreSQL allows listening on [multiple addresses](https://www.postgresql.org/docs/11/runtime-config-connection.html)
If you use multiple addresses in `gitlab_rails['db_host']`, comma-separated, the first address in the list will be used for the connection.
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) for the changes to take effect.
1. [Seed the database](#seed-the-database-fresh-installs-only).
### UNIX socket configuration for non-packaged PostgreSQL
If you want to use your system's PostgreSQL server (installed on the same system
as GitLab) instead of the one bundled with GitLab, you can do so by using a UNIX
socket:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
# Disable the built-in Postgres
postgresql['enable'] = false
# Fill in the connection details for database.yml
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
# The path where the socket lives
gitlab_rails['db_host'] = '/var/run/postgresql/'
```
1. Reconfigure GitLab for the changes to take effect:
```ruby
sudo gitlab-ctl-reconfigure
```
### Configuring SSL
#### Require SSL
1. Add the following to `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['db_sslmode'] = 'require'
```
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) to apply the configuration changes.
1. Restart PostgreSQL for the changes to take effect:
```shell
gitlab-ctl restart postgresql
```
If PostgreSQL fails to start, check the logs (for example,
`/var/log/gitlab/postgresql/current`) for more details.
#### Require SSL and verify server certificate against CA bundle
PostgreSQL can be configured to require SSL and verify the server certificate
against a CA bundle to prevent spoofing.
The CA bundle that's specified in `gitlab_rails['db_sslrootcert']` must contain
both the root and intermediate certificates.
1. Add the following to `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['db_sslmode'] = "verify-full"
gitlab_rails['db_sslrootcert'] = "<full_path_to_your_ca-bundle.pem>"
```
If you are using Amazon RDS for your PostgreSQL server, ensure you
download and use the [combined CA bundle](https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem)
for `gitlab_rails['db_sslrootcert']`. More information on this can be found
in the [using SSL/TLS to Encrypt a Connection to a DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html)
article on AWS.
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) to apply the configuration changes.
1. Restart PostgreSQL for the changes to take effect:
```shell
gitlab-ctl restart postgresql
```
If PostgreSQL fails to start, check the logs (for example,
`/var/log/gitlab/postgresql/current`) for more details.
### Backup and restore a non-packaged PostgreSQL database
When using the [backup](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#backup-command)
and [restore](https://docs.gitlab.com/ee/administration/backup_restore/restore_gitlab.html#restore-for-linux-package-installations)
commands, GitLab will
attempt to use the packaged `pg_dump` command to create a database backup file
and the packaged `psql` command to restore a backup. This will only work if
they are the correct versions. Check the versions of the packaged `pg_dump` and
`psql`:
```shell
/opt/gitlab/embedded/bin/pg_dump --version
/opt/gitlab/embedded/bin/psql --version
```
If these versions are different from your non-packaged external PostgreSQL, you may encounter the following error output when attempting to run the [backup command](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#backup-command).
```plaintext
Dumping PostgreSQL database gitlabhq_production ... pg_dump: error: server version: 13.3; pg_dump version: 12.6
pg_dump: error: aborting because of server version mismatch
```
In this example, the error occurs on GitLab 14.1 when using PostgreSQL version 13.3, instead of the [default shipped PostgreSQL version](https://docs.gitlab.com/ee/administration/package_information/postgresql_versions.html) of 12.6.
In this case, you will need to install tools that match your database version and then follow the
steps below. There are multiple ways to install PostgreSQL client tools. See
<https://www.postgresql.org/download/> for options.
Once the correct `psql` and `pg_dump` tools are available on your system, follow
these steps, using the correct path to the location you installed the new tools:
1. Add symbolic links to the non-packaged versions:
```shell
ln -s /path/to/new/pg_dump /path/to/new/psql /opt/gitlab/bin/
```
1. Check the versions:
```shell
/opt/gitlab/bin/pg_dump --version
/opt/gitlab/bin/psql --version
```
They should now be the same as your non-packaged external PostgreSQL.
After this is done, ensure that the backup and restore tasks are using the
correct executables by running both the [backup](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#backup-command) and
[restore](https://docs.gitlab.com/ee/administration/backup_restore/restore_gitlab.html#restore-for-linux-package-installations) commands.
### Upgrade a non-packaged PostgreSQL database
You can upgrade the external database after stopping all the processes that are connected to the database (Puma, Sidekiq):
```shell
sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq
```
Before proceeding with the upgrade, note the following:
- Check compatibility between GitLab releases and PostgreSQL versions:
- Read about which GitLab versions introduced a requirement for a
[minimum PostgreSQL version](https://docs.gitlab.com/ee/install/requirements.html#postgresql-requirements).
- Read about significant changes to the PostgreSQL versions which
[shipped with the Linux package](https://docs.gitlab.com/ee/administration/package_information/postgresql_versions.html):
The Linux package is tested for compatibility with the major releases of PostgreSQL that it ships with.
- When using GitLab backup or restore, you _must_ keep the same version of GitLab.
If you plan to upgrade to a later GitLab version as well, upgrade PostgreSQL first.
- The [backup and restore commands](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#backup-command)
can be used to back up and restore the database to a later version of PostgreSQL.
- If a PostgreSQL version is specified with `postgresql['version']` that doesn't ship
with that Linux package release, the
[default version in the compatibility table](https://docs.gitlab.com/ee/administration/package_information/postgresql_versions.html)
determines which client binaries (such as the PostgreSQL backup/restore binaries) are active.
The following example demonstrates upgrading from a database host running PostgreSQL 13 to another database host running PostgreSQL 14 and incurs downtime:
1. Spin up a new PostgreSQL 14 database server that's set up according to the [database requirements](https://docs.gitlab.com/ee/install/requirements.html#database).
1. Ensure that the compatible versions of `pg_dump` and `pg_restore` are being
used on the GitLab Rails instance. To amend GitLab configuration, edit
`/etc/gitlab/gitlab.rb` and specify the value of `postgresql['version']`:
```ruby
postgresql['version'] = 14
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
1. Stop GitLab (note that this step causes downtime):
```shell
sudo gitlab-ctl stop
```
WARNING:
The backup command requires [additional parameters](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#back-up-and-restore-for-installations-using-pgbouncer)
when your installation is using PgBouncer.
1. Run the backup Rake task using the SKIP options to back up only the database.
Make a note of the backup file name; you'll use it later to restore.
```shell
sudo gitlab-backup create SKIP=repositories,uploads,builds,artifacts,lfs,pages,registry
```
1. Shutdown the PostgreSQL 13 database host.
1. Edit `/etc/gitlab/gitlab.rb` and update the `gitlab_rails['db_host']` setting
to point to the PostgreSQL database 14 host.
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
WARNING:
The backup command requires [additional parameters](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#back-up-and-restore-for-installations-using-pgbouncer)
when your installation is using PgBouncer.
1. Restore the database using the database backup file created earlier, and be
sure to answer **no** when asked "This task will now rebuild the authorized_keys file":
```shell
# Use the backup timestamp https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#backup-timestamp
sudo gitlab-backup restore BACKUP=<backup-timestamp>
```
1. Start GitLab:
```shell
sudo gitlab-ctl start
```
1. After upgrading PostgreSQL to a new major release, recreate the table statistics to ensure efficient query plans are picked and
to reduce database server CPU load.
If the upgrade was "in-place" using `pg_upgrade`, run the following query on the PostgreSQL database console:
```SQL
SET statement_timeout = 0; ANALYZE VERBOSE;
```
If the upgrade used `pg_dump` and `pg_restore`, run the following query on the PostgreSQL database console:
```SQL
SET statement_timeout = 0; VACUUM VERBOSE ANALYZE;
```
### Seed the database (fresh installs only)
WARNING:
This is a destructive command; do not run it on an existing database.
The Linux package installation does not seed your external database. Run the following command
to import the schema and create the first administration user:
```shell
# Remove 'sudo' if you are the 'git' user
sudo gitlab-rake gitlab:setup
```
If you want to specify a password for the default `root` user, specify the
`initial_root_password` setting in `/etc/gitlab/gitlab.rb` before running the
`gitlab:setup` command above:
```ruby
gitlab_rails['initial_root_password'] = 'nonstandardpassword'
```
If you want to specify the initial registration token for shared GitLab Runners,
specify the `initial_shared_runners_registration_token` setting in `/etc/gitlab/gitlab.rb`
before running the `gitlab:setup` command:
```ruby
gitlab_rails['initial_shared_runners_registration_token'] = 'token'
```
### Pin the packaged PostgreSQL version (fresh installs only)
The Linux package ships with [different PostgreSQL versions](https://docs.gitlab.com/ee/administration/package_information/postgresql_versions.html)
and initializes the default version if not specified otherwise.
To initialize PostgreSQL with a non-default version, you can set `postgresql['version']` to the major version one of
the [packaged PostgreSQL versions](https://docs.gitlab.com/ee/administration/package_information/postgresql_versions.html) prior to the initial reconfigure.
For example, in GitLab 15.0 you can use `postgresql['version'] = 12` to use PostgreSQL 12 instead of the default of PostgreSQL 13.
WARNING:
Setting `postgresql['version']` while using the PostgreSQL packaged with the Linux package after the initial reconfigure will
throw errors about the data directory being initialized on a different version of PostgreSQL. If this is encountered,
see [Revert packaged PostgreSQL server to the previous version](#revert-packaged-postgresql-server-to-the-previous-version).
If you are doing a fresh install on an environment that previously had GitLab installed on it and you are using a pinned PostgreSQL version, first make
sure that any folders that relate to PostgreSQL are deleted and that there are no PostgreSQL processes running on the instance.
## Provide sensitive data configuration to GitLab Rails without plain text storage
For more information, see the example in [configuration documentation](../settings/configuration.md#provide-the-postgresql-user-password-to-gitlab-rails).
## Application Settings for the Database
### Disabling automatic database migration
If you have multiple GitLab servers sharing a database, you will want to limit the
number of nodes that are performing the migration steps during reconfiguration.
Edit `/etc/gitlab/gitlab.rb` to append:
```ruby
# Enable or disable automatic database migrations
# on all hosts except the designated deploy node
gitlab_rails['auto_migrate'] = false
```
`/etc/gitlab/gitlab.rb` should have file permissions `0600` because it contains
plain-text passwords.
The next time hosts carrying the above configuration are reconfigured, the migration steps are not performed.
To avoid schema-related post-upgrade errors, the host marked as [the deploy node](https://docs.gitlab.com/ee/update/zero_downtime.html#multi-node--ha-deployment) must have `gitlab_rails['auto_migrate'] = true` during upgrades.
### Setting client statement_timeout
The amount of time that Rails will wait for a database transaction to complete
before timing out can now be adjusted with the `gitlab_rails['db_statement_timeout']`
setting. By default, this setting is not used.
Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['db_statement_timeout'] = 45000
```
In this case, the client `statement_timeout` is set to 45 seconds. The value
is specified in milliseconds.
### Setting connection timeout
The amount of time that Rails will wait for a PostgreSQL connection attempt to succeed
before timing out can be adjusted with the `gitlab_rails['db_connect_timeout']`
setting. By default, this setting is not used:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['db_connect_timeout'] = 5
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
In this case, the client `connect_timeout` is set to 5 seconds. The value
is specified in seconds. A minimum value of 2 seconds applies. Setting this to `<= 0`
or not specifying the setting at all disables the timeout.
### Setting TCP controls
The Rails PostgreSQL adapter provides a series of TCP connection controls
that may be tuned to improve performance. Consult the
[PostgreSQL upstream documentation for more information about each parameter](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-KEEPALIVES).
The Linux package sets no defaults for these values and instead uses the defaults
provided by the PostgreSQL adapter. Override them in `gitlab.rb` using the
parameters noted in the table below and then run `gitlab-ctl reconfigure`.
| PostgreSQL parameter | `gitlab.rb` parameter |
|-|-|
| `keepalives` | `gitlab_rails['db_keepalives']` |
| `keepalives_idle` | `gitlab_rails['db_keepalives_idle']` |
| `keepalives_interval` | `gitlab_rails['db_keepalives_interval']` |
| `keepalives_count` | `gitlab_rails['db_keepalives_count']` |
| `tcp_user_timeout` | `gitlab_rails['db_tcp_user_timeout']` |
## Automatic database reindexing
WARNING:
This is an experimental feature that isn't enabled by default.
Recreates database indexes in the background (called "reindexing"). This can
be used to remove bloated space that has accumulated in indexes and helps to maintain healthy and
efficient indexes.
The reindexing task can be started regularly through a cronjob. To configure the cronjob,
`gitlab_rails['database_reindexing']['enable']` should be set to `true`.
In a multi-node environment, this feature should only be enabled on an application host.
The reindexing process cannot go through PgBouncer, it has to have a direct database connection.
By default, this starts the cronjob every hour during weekends (likely a low-traffic time) only.
You can change the schedule by refining the following settings:
1. Edit `/etc/gitlab/gitlab.rb`:
```shell
gitlab_rails['database_reindexing']['hour'] = '*'
gitlab_rails['database_reindexing']['minute'] = 0
gitlab_rails['database_reindexing']['month'] = '*'
gitlab_rails['database_reindexing']['day_of_month'] = '*'
gitlab_rails['database_reindexing']['day_of_week'] = '0,6'
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Packaged PostgreSQL deployed in an HA/Geo Cluster
### Upgrading a GitLab HA cluster
To upgrade the PostgreSQL version in a Patroni cluster see [Upgrading PostgreSQL major version in a Patroni cluster](https://docs.gitlab.com/ee/administration/postgresql/replication_and_failover.html#upgrading-postgresql-major-version-in-a-patroni-cluster).
### Upgrading a GitLab HA Repmgr cluster
NOTE:
If you are upgrading to PostgreSQL 12, you need to switch from Repmgr to Patroni first see [Switching from Repmgr to Patroni](https://docs.gitlab.com/ee/administration/postgresql/replication_and_failover.html#switching-from-repmgr-to-patroni).
These instructions are provided for upgrading an older GitLab cluster to PostgreSQL 11 when using Repmgr.
If [PostgreSQL is configured for high availability](https://docs.gitlab.com/ee/administration/postgresql/index.html),
`pg-upgrade` should be run on all the nodes running PostgreSQL. Other nodes can be
skipped but must be running the same GitLab version as the database nodes.
Follow the steps below to upgrade the database nodes:
1. Secondary nodes must be upgraded before the primary node.
1. On the secondary nodes, edit `/etc/gitlab/gitlab.rb` to include the following:
```shell
# Replace X with the number of DB nodes + 1
postgresql['max_replication_slots'] = X
```
1. Run `gitlab-ctl reconfigure` to update the configuration.
1. Run `sudo gitlab-ctl restart postgresql` to get PostgreSQL restarted with the new configuration.
1. On running `pg-upgrade` on a PostgreSQL secondary node, the node will be removed
from the cluster.
1. Once all the secondary nodes are upgraded using `pg-upgrade`, the user
will be left with a single-node cluster that has only the primary node.
1. `pg-upgrade`, on secondary nodes will not update the existing data to
match the new version, as that data will be replaced by the data from
the primary node. It will however move the existing data to a backup
location.
1. Once all secondary nodes are upgraded, run `pg-upgrade` on the primary node.
1. On the primary node, edit `/etc/gitlab/gitlab.rb` to include the following:
```shell
# Replace X with the number of DB nodes + 1
postgresql['max_replication_slots'] = X
```
1. Run `gitlab-ctl reconfigure` to update the configuration.
1. Run `sudo gitlab-ctl restart postgresql` to get PostgreSQL restarted with the new configuration.
1. On a primary node, `pg-upgrade` will update the existing data to match
the new PostgreSQL version.
1. Recreate the secondary nodes by running the following command on each of them
```shell
gitlab-ctl repmgr standby setup MASTER_NODE_NAME
```
1. Check if the repmgr cluster is back to the original state
```shell
gitlab-ctl repmgr cluster show
```
### Troubleshooting upgrades in an HA cluster
If at some point, the bundled PostgreSQL had been running on a node before upgrading to an HA setup, the old data directory may remain. This will cause `gitlab-ctl reconfigure` to downgrade the version of the PostgreSQL utilities it uses on that node. Move (or remove) the directory to prevent this:
- `mv /var/opt/gitlab/postgresql/data/ /var/opt/gitlab/postgresql/data.$(date +%s)`
If you encounter the following error when recreating the secondary nodes with `gitlab-ctl repmgr standby setup MASTER_NODE_NAME`, ensure that you have `postgresql['max_replication_slots'] = X` (where `X` is the number of DB nodes + 1), is included in `/etc/gitlab/gitlab.rb`:
```shell
pg_basebackup: could not create temporary replication slot "pg_basebackup_12345": ERROR: all replication slots are in use
HINT: Free one or increase max_replication_slots.
```
### Upgrading a Geo instance
Since Geo depends on PostgreSQL streaming replication by default, there are
additional considerations when upgrading GitLab and/or when upgrading
PostgreSQL is described below.
#### Caveats when upgrading PostgreSQL with Geo
WARNING:
When using Geo, upgrading PostgreSQL **requires downtime on all secondaries**.
When using Geo, upgrading PostgreSQL **requires downtime on all secondaries**
because it requires re-initializing PostgreSQL replication to Geo
**secondaries**. This is due to the way PostgreSQL streaming replication works.
Re-initializing replication copies all data from the primary again, so it can
take a long time depending mostly on the size of the database and available
bandwidth. For example, at a transfer speed of 30 Mbps, and a database size of
100 GB, resynchronization could take approximately 8 hours. See
[PostgreSQL documentation](https://www.postgresql.org/docs/11/pgupgrade.html)
for more.
#### How to upgrade PostgreSQL when using Geo
To upgrade PostgreSQL, you will need the name of the replication slot, and the
replication user's password.
1. Find the name of the existing replication slot on the Geo primary's database
node, run:
```shell
sudo gitlab-psql -qt -c 'select slot_name from pg_replication_slots'
```
If you can't find your `slot_name` here, or there is no output returned, your Geo secondaries may not be healthy. In that case, make sure the [secondaries are healthy and replication is working](https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html#check-the-health-of-the-secondary-node).
1. Gather the replication user's password. It was set while setting up Geo in
[Step 1. Configure the primary site](https://docs.gitlab.com/ee/administration/geo/setup/database.html#step-1-configure-the-primary-site).
1. Manually upgrade PostgreSQL on the Geo primary. Run on the Geo primary's
database node:
```shell
sudo gitlab-ctl pg-upgrade
```
Wait for the **primary database** to finish upgrading before
beginning the following step, so the secondary can remain ready as a backup.
Afterward, you can upgrade the **tracking database** in parallel with the
**secondary database**.
1. Manually upgrade PostgreSQL on the Geo secondaries. Run on the Geo
**secondary database** and also on the **tracking database**:
```shell
sudo gitlab-ctl pg-upgrade
```
1. Restart the database replication on the Geo **secondary database** using the
command:
```shell
sudo gitlab-ctl replicate-geo-database --slot-name=SECONDARY_SLOT_NAME --host=PRIMARY_HOST_NAME --sslmode=verify-ca
```
You will be prompted for the replication user's password of the primary. Replace `SECONDARY_SLOT_NAME` with the slot name retrieved from the first step above.
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) on the Geo **secondary database** to update the
`pg_hba.conf` file. This is needed because `replicate-geo-database`
replicates the primary's file to the secondary.
1. Restart `puma`, `sidekiq`, and `geo-logcursor`.
```shell
sudo gitlab-ctl hup puma
sudo gitlab-ctl restart sidekiq
sudo gitlab-ctl restart geo-logcursor
```
1. Navigate to `https://your_primary_server/admin/geo/nodes` and ensure that all nodes are healthy.
## Connecting to the PostgreSQL database
If you need to connect to the PostgreSQL database, you can connect as the
application user:
```shell
sudo gitlab-rails dbconsole --database main
```
## Troubleshooting
### Set `default_transaction_isolation` into `read committed`
If you see errors similar to the following in your `production/sidekiq` log:
```plaintext
ActiveRecord::StatementInvalid PG::TRSerializationFailure: ERROR: could not serialize access due to concurrent update
```
Chances are your database's `default_transaction_isolation` configuration is not
in line with the GitLab application requirement. You can check this configuration by
connecting to your PostgreSQL database and run `SHOW default_transaction_isolation;`.
GitLab application expects `read committed` to be configured.
This `default_transaction_isolation` configuration is set in your
`postgresql.conf` file. You will need to restart/reload the database once you
changed the configuration. This configuration comes by default in the packaged
PostgreSQL server included with the Linux package.
### Could not load library `plpgsql.so`
You might see errors similar to the following while running Database migrations
or in the PostgreSQL/Patroni logs:
```plaintext
ERROR: could not load library "/opt/gitlab/embedded/postgresql/12/lib/plpgsql.so": /opt/gitlab/embedded/postgresql/12/lib/plpgsql.so: undefined symbol: EnsurePortalSnapshotExists
```
This error is caused due to not restarting PostgreSQL after the underlying
version changed. To fix this error:
1. Run one of the following commands:
```shell
# For PostgreSQL
sudo gitlab-ctl restart postgresql
# For Patroni
sudo gitlab-ctl restart patroni
# For Geo PostgreSQL
sudo gitlab-ctl restart geo-postgresql
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
### Database CPU load very high
If the database CPU load is very high, it could be caused by the [auto cancel redundant pipelines setting](https://docs.gitlab.com/ee/ci/pipelines/settings.html#auto-cancel-redundant-pipelines). For more details, see [issue 435250](https://gitlab.com/gitlab-org/gitlab/-/issues/435250).
To work around this issue:
- You can allocate more CPU resources to the database server.
- If Sidekiq is overloaded, you might need to [add more Sidekiq processes](https://docs.gitlab.com/ee/administration/sidekiq/extra_sidekiq_processes.html#start-multiple-processes) for the `ci_cancel_redundant_pipelines` queue if your projects have a very large number of pipelines.
- You can enable the `disable_cancel_redundant_pipelines_service` feature flag to disable this setting instance-wide and see if the CPU load goes down. This disables the feature for all projects, and can lead to increased resource use by pipelines that are no longer being cancelled automatically.
doc/settings/dns.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# DNS settings
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
The Domain Name System (DNS) is the naming system used to match IP addresses
with domain names.
Although you can run a GitLab instance using only its IP address, using a
domain name is:
- Easier to remember and use.
- Required for HTTPS.
NOTE:
To take advantage of the [Let's Encrypt integration](ssl/index.md#enable-the-lets-encrypt-integration) (automatic SSL certificates),
your instance's domain name must be resolvable over the public internet.
## Use a name registrar
To associate a domain name with your instance's IP address, you must specify
one or more DNS records.
Adding a DNS record to your domain's DNS configuration is entirely dependent
on your chosen provider, and out of scope for this document.
Generally, the process is similar to:
1. Visit the control panel of your DNS registrar and add the DNS record.
It should be one of type:
- `A`
- `AAAA`
- `CNAME`
The type depends on the underlying architecture of your instance. The most
common one is the A record.
1. [Test](#successful-dns-query) that the configuration was applied.
1. Use SSH to connect to the server where GitLab is installed.
1. Edit the configuration file `(/etc/gitlab/gitlab.rb)` with your preferred [GitLab settings](#gitlab-settings-that-use-dns).
To learn more about the DNS records, see the
[DNS records overview](https://docs.gitlab.com/ee/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.html).
## Use a dynamic DNS service
For non-production use, you can use a dynamic DNS service, such as [nip.io](https://nip.io).
We do not recommend these for any production or long-lived instances, as they are often:
- [Insecure](https://github.com/publicsuffix/list/issues/335#issuecomment-261825647)
- [Rate-limited](https://letsencrypt.org/docs/rate-limits/) by Let's Encrypt
## GitLab settings that use DNS
The following GitLab settings correspond to DNS entries.
| GitLab setting | Description | Configuration |
| -------------- | ----------- | ------------- |
| `external_url` | This URL interacts with the main GitLab instance. It's used when cloning over SSH/HTTP/HTTPS and when accessing the web UI. GitLab Runner uses this URL to communicate with the instance. | [Configure the `external_url`](configuration.md#configure-the-external-url-for-gitlab). |
| `registry_external_url` | This URL is used to interact with the [Container Registry](https://docs.gitlab.com/ee/user/packages/container_registry/). It can be used by the Let's Encrypt integration. This URL can also use the same DNS entry as `external_url` but on a different port. | [Configure the `registry_external_url`](https://docs.gitlab.com/ee/administration/packages/container_registry.html#container-registry-domain-configuration). |
| `mattermost_external_url` | This URL is used for the [bundled Mattermost](https://docs.gitlab.com/ee/integration/mattermost/) software. It can be used by the Let's Encrypt integration. | [Configure the `mattermost_external_url`](https://docs.gitlab.com/ee/integration/mattermost/#getting-started). |
| `pages_external_url` | By default, projects that use [GitLab Pages](https://docs.gitlab.com/ee/user/project/pages/) deploy to a sub-domain of this value. | [Configure the `pages_external_url`](https://docs.gitlab.com/ee/administration/pages/#configuration). |
| Auto DevOps domain | If you use Auto DevOps to deploy projects, this domain can be used to deploy software. It can be defined at an instance, or cluster level. This is configured using the GitLab UI, and not in `/etc/gitlab/gitlab.rb`. | [Configure the Auto DevOps domain](https://docs.gitlab.com/ee/topics/autodevops/requirements.html#auto-devops-base-domain). |
## Troubleshooting
If you have issues accessing a particular component, or if the Let's
Encrypt integration is failing, you might have a DNS issue. You can use the
[dig](https://en.wikipedia.org/wiki/Dig_(command)) tool to determine if
DNS is causing a problem.
### Successful DNS query
This example uses the [Public Cloudflare DNS resolver](https://www.cloudflare.com/en-gb/learning/dns/what-is-1.1.1.1/) to ensure that the query is globally resolvable. However, other public resolvers like the [Google Public DNS resolver](https://developers.google.com/speed/public-dns) are also available.
```shell
$ dig registry.gitlab.com @1.1.1.1
; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> registry.gitlab.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3934
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;registry.gitlab.com. IN A
;; ANSWER SECTION:
registry.gitlab.com. 58 IN A 35.227.35.254
;; Query time: 8 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Wed Jan 31 11:16:51 CET 2024
;; MSG SIZE rcvd: 64
```
Make sure that the status is `NOERROR`, and that the `ANSWER SECTION` has the actual results.
### Failed DNS query
```shell
$ dig fake.gitlab.com @1.1.1.1
; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> fake.gitlab.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25693
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;fake.gitlab.com. IN A
;; AUTHORITY SECTION:
gitlab.com. 1800 IN SOA diva.ns.cloudflare.com. dns.cloudflare.com. 2331688399 10000 2400 604800 1800
;; Query time: 12 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Wed Jan 31 11:17:46 CET 2024
;; MSG SIZE rcvd: 103
```
In this example, the `status` is `NXDOMAIN`, and there is no `ANSWER SECTION`. The `SERVER` field tells you which DNS server was queried for the answer, in this case the [Public Cloudflare DNS resolver](https://www.cloudflare.com/en-gb/learning/dns/what-is-1.1.1.1/).
### Use a wildcard DNS entry
It is possible use a wildcard DNS for the [URL attributes](#gitlab-settings-that-use-dns),
but you must provide the full domain name for each one.
The Let's Encrypt integration does not fetch a wildcard certificate. You must do this
[on your own](https://certbot.eff.org/faq/#does-let-s-encrypt-issue-wildcard-certificates).
doc/settings/environment-variables.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Setting custom environment variables
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
If necessary you can set custom environment variables to be used by Puma,
Sidekiq, Rails and Rake via `/etc/gitlab/gitlab.rb`. This can be useful in
situations where you need to use a proxy to access the internet and need to
clone externally hosted repositories directly into GitLab. In
`/etc/gitlab/gitlab.rb` supply a `gitlab_rails['env']` with a hash value. For
example:
```ruby
gitlab_rails['env'] = {
"http_proxy" => "http://USERNAME:PASSWORD@example.com:8080",
"https_proxy" => "http://USERNAME:PASSWORD@example.com:8080"
# "no_proxy" => ".yourdomain.com" # Wildcard syntax if you need your internal domain to bypass proxy
}
```
You can also override environment variables from other GitLab components which
might be required if you are behind a proxy:
```ruby
# Needed for proxying Git clones
gitaly['env'] = {
"http_proxy" => "http://USERNAME:PASSWORD@example.com:8080",
"https_proxy" => "http://USERNAME:PASSWORD@example.com:8080"
}
gitlab_workhorse['env'] = {
"http_proxy" => "http://USERNAME:PASSWORD@example.com:8080",
"https_proxy" => "http://USERNAME:PASSWORD@example.com:8080"
}
gitlab_pages['env'] = {
"http_proxy" => "http://USERNAME:PASSWORD@example.com:8080",
"https_proxy" => "http://USERNAME:PASSWORD@example.com:8080"
}
# If you use the docker registry
registry['env'] = {
"http_proxy" => "http://USERNAME:PASSWORD@example.com:8080",
"https_proxy" => "http://USERNAME:PASSWORD@example.com:8080"
}
```
GitLab will attempt to use HTTP Basic Authentication when a username and password is included in the proxy URL.
Proxy settings use the `.` syntax for globing.
Proxy URL values should generally be `http://` only, unless
your proxy has its own SSL certificate and SSL enabled. This means, even for
the `https_proxy` value, you should usually specify a value as
`http://USERNAME:PASSWORD@example.com:8080`.
NOTE:
DNS rebind protection is disabled when either the HTTP_PROXY or the HTTPS_PROXY environment variable is set,
and the domain DNS can't be resolved.
## Applying the changes
Any change made to the environment variables requires a reconfigure for it
to take effect.
Perform a reconfigure:
```shell
sudo gitlab-ctl reconfigure
```
doc/settings/gitlab.yml.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Changing `gitlab.yml` and `application.yml` settings
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
Some GitLab features can be customized through
[`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/config/gitlab.yml.example). If you want to change a `gitlab.yml` setting
for a Linux package installation, you need to do so with `/etc/gitlab/gitlab.rb`. The
translation works as follows. For a complete list of available options, visit the
[`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template).
All the options of the template listed in `/etc/gitlab/gitlab.rb` are avilable by default.
In `gitlab.yml`, you find structure like this:
```yaml
production: &base
gitlab:
default_theme: 2
```
In `gitlab.rb`, this translates to:
```ruby
gitlab_rails['gitlab_default_theme'] = 2
```
What happens here is that we forget about `production: &base`, and join
`gitlab:` with `default_theme:` into `gitlab_default_theme`.
Note that not all `gitlab.yml` settings can be changed via `gitlab.rb` yet; see
the [`gitlab.yml.erb` template](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/gitlab.yml.erb).
If you think an attribute is missing please create a merge request on the `omnibus-gitlab` repository.
Run `sudo gitlab-ctl reconfigure` for changes in `gitlab.rb` to take effect.
Do not edit the generated file in `/var/opt/gitlab/gitlab-rails/etc/gitlab.yml`
because it is overwritten on the next `gitlab-ctl reconfigure` run.
## Adding a new setting to `gitlab.yml`
First, consider not adding a setting to `gitlab.yml`. See **Settings** under [GitLab-specific concerns](https://docs.gitlab.com/ee/development/code_review.html#gitlab-specific-concerns).
Don't forget to update the following 5 files when adding a new setting:
- the [`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template)
file to expose the setting to the end user via `/etc/gitlab/gitlab.rb`.
- the [`default.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb)
file to provide a sane default for the new setting.
- the [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example)
file to actually use the setting's value from `gitlab.rb`.
- the [`gitlab.yml.erb`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/gitlab.yml.erb)
file
- the [`gitlab-rails_spec.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/spec/chef/cookbooks/gitlab/recipes/gitlab-rails_spec.rb)
file
doc/settings/image_scaling.md 0 → 100644
View file @ edb06ab2
---
stage: Data Stores
group: Tenant Scale
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Image scaling
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
GitLab runs a built-in image scaler to improve site rendering performance. It is enabled by default.
## Configure the scaler
We strive to always set sensible defaults that work with the vast majority of GitLab deployments.
However, we provide several settings that allow you to tweak image scaling to best match your
desired performance profile.
### Maximum number of image scalers
Rescaling images results in additional, short-lived processes that run on the same node Workhorse
runs on. By default, we limit the number of these processes allowed to execute simultaneously
to half the number of CPU cores on that machine or VM, but no less than two.
You may choose to set this to a fixed value instead:
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
gitlab_workhorse['image_scaler_max_procs'] = 10
```
1. Reconfigure for the changes to take effect:
```shell
sudo gitlab-ctl reconfigure
```
This would mean that if 10 images are already being processed, then the 11th request would not be
rescaled, and would be served in the original size instead. Putting a ceiling on this is important
to ensure that the system remains available even under high load.
### Maximum image file size
By default, GitLab only rescales images that are at most 250kB in size. This is to prevent excessive
memory consumption on Workhorse nodes and to keep latencies in reasonable bounds. Beyond a certain
file size, it is in fact faster overall to just serve the original image instead.
If you want to lower or raise the maximum allowed file size:
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
gitlab_workhorse['image_scaler_max_filesize'] = 1024 * 1024
```
1. Reconfigure for the changes to take effect:
```shell
sudo gitlab-ctl reconfigure
```
This would allow images up to 1MB to be rescaled (the unit is Byte).
### Disabling the image scaler
You may decide to turn off image scaling entirely. This can be accomplished by switching the respective
feature toggle off:
```ruby
Feature.disable(:dynamic_image_resizing)
```
Refer to the [Feature Flags documentation](https://docs.gitlab.com/ee/administration/feature_flags.html)
to learn how to work with feature flags.
doc/settings/index.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Configuring a Linux package installation
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
- [Asset proxy server](https://docs.gitlab.com/ee/security/asset_proxy.html)
- [Backups](backups.md)
- [Configuration options](configuration.md)
- [Database](database.md)
- [Environment variables](environment-variables.md)
- [Gitaly Cluster](praefect.md)
- [`gitlab.yml`](gitlab.yml.md)
- [LDAP](https://docs.gitlab.com/ee/administration/auth/ldap/index.html)
- [Logs](logs.md)
- [Mattermost](https://docs.gitlab.com/ee/integration/mattermost/)
- [NGINX](nginx.md)
- [Pages](https://docs.gitlab.com/ee/administration/pages/index.html)
- [Prometheus Monitoring](prometheus.md)
- [Puma](https://docs.gitlab.com/ee/administration/operations/puma.html)
- [Raspberry Pi](rpi.md)
- [Redis](redis.md)
- [SMTP](smtp.md)
- [SSL](ssl/index.md)
- [Image scaling](image_scaling.md)
doc/settings/logs.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Logs on Linux package installations
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
GitLab includes an [advanced log system](https://docs.gitlab.com/ee/administration/logs/index.html) where every service and component within GitLab will output system logs.
Here are the configuration settings and tools for managing these logs on Linux package installations.
## Tail logs in a console on the server
If you want to 'tail', i.e. view live log updates of GitLab logs you can use
`gitlab-ctl tail`.
```shell
# Tail all logs; press Ctrl-C to exit
sudo gitlab-ctl tail
# Drill down to a sub-directory of /var/log/gitlab
sudo gitlab-ctl tail gitlab-rails
# Drill down to an individual file
sudo gitlab-ctl tail nginx/gitlab_error.log
```
### Tail logs in a console and save to a file
Oftentimes, it is useful to both display the logs in the console and save them to a file for later debugging/analysis. You can use the [`tee`](https://en.wikipedia.org/wiki/Tee_(command)) utility to accomplish this.
```shell
# Use 'tee' to tail all the logs to STDOUT and write to a file at the same time
sudo gitlab-ctl tail | tee --append /tmp/gitlab_tail.log
```
## Configure default log directories
In your `/etc/gitlab/gitlab.rb` file, there are many `log_directory` keys for
the various types of logs. Uncomment and update the values for all the logs
you want to place elsewhere:
```ruby
# For example:
gitlab_rails['log_directory'] = "/var/log/gitlab/gitlab-rails"
puma['log_directory'] = "/var/log/gitlab/puma"
registry['log_directory'] = "/var/log/gitlab/registry"
...
```
Gitaly and Mattermost have different log directory configs:
```ruby
gitaly['configuration'] = {
logging: {
dir: "/var/log/gitlab/registry"
}
}
mattermost['log_file_directory'] = "/var/log/gitlab/registry"
```
Run `sudo gitlab-ctl reconfigure` to configure your instance with these settings.
## runit logs
The [runit-managed](../architecture/index.md#runit) services in Linux package installations generate log data using
`svlogd`.
- Logs are written to a file called `current`.
- Periodically, this log is compressed and renamed using the TAI64N format, for
example: `@400000005f8eaf6f1a80ef5c.s`.
- The filesystem datestamp on the compressed logs will be consistent with the time
GitLab last wrote to that file.
- `zmore` and `zgrep` allow viewing and searching through both compressed or uncompressed logs.
Read the [svlogd documentation](https://smarden.org/runit/svlogd.8) for more information
about the files it generates.
You can modify svlogd settings via `/etc/gitlab/gitlab.rb` with the following settings:
```ruby
# Below are the default values
logging['svlogd_size'] = 200 * 1024 * 1024 # rotate after 200 MB of log data
logging['svlogd_num'] = 30 # keep 30 rotated log files
logging['svlogd_timeout'] = 24 * 60 * 60 # rotate after 24 hours
logging['svlogd_filter'] = "gzip" # compress logs with gzip
logging['svlogd_udp'] = nil # transmit log messages via UDP
logging['svlogd_prefix'] = nil # custom prefix for log messages
# Optionally, you can override the prefix for e.g. Nginx
nginx['svlogd_prefix'] = "nginx"
```
## Logrotate
The **logrotate** service built into GitLab manages all logs except those
captured by **runit**. This service will rotate, compress, and eventually delete
the log data such as `gitlab-rails/production.log` and
`nginx/gitlab_access.log`. You can configure common logrotate settings,
configure per-service logrotate settings, and completely disable logrotate
with `/etc/gitlab/gitlab.rb`.
### Configuring common logrotate settings
Settings common to all **logrotate** services can be set in the
`/etc/gitlab/gitlab.rb` file. These settings correspond to configuration options
in the logrotate configuration files for each service. See the logrotate man
page (`man logrotate`) for details.
```ruby
logging['logrotate_frequency'] = "daily" # rotate logs daily
logging['logrotate_maxsize'] = nil # logs will be rotated when they grow bigger than size specified for `maxsize`, even before the specified time interval (daily, weekly, monthly, or yearly)
logging['logrotate_size'] = nil # do not rotate by size by default
logging['logrotate_rotate'] = 30 # keep 30 rotated logs
logging['logrotate_compress'] = "compress" # see 'man logrotate'
logging['logrotate_method'] = "copytruncate" # see 'man logrotate'
logging['logrotate_postrotate'] = nil # no postrotate command by default
logging['logrotate_dateformat'] = nil # use date extensions for rotated files rather than numbers e.g. a value of "-%Y-%m-%d" would give rotated files like production.log-2016-03-09.gz
```
### Configuring individual service logrotate settings
You can customize logrotate settings for each individual service by using
`/etc/gitlab/gitlab.rb`. For example, to customize logrotate frequency and size
for the `nginx` service, use:
```ruby
nginx['logrotate_frequency'] = nil
nginx['logrotate_size'] = "200M"
```
### Disabling logrotate
You can also disable the built-in logrotate service with the following setting
in `/etc/gitlab/gitlab.rb`:
```ruby
logrotate['enable'] = false
```
### Logrotate `notifempty` setting
The logrotate service runs with a non-configurable default of `notifempty`, resolving
the following issues:
- Empty logs being rotated unnecessarily, and often many empty logs being stored.
- One-off logs that are useful for long term troubleshooting being deleted after 30 days, such as database migration logs.
### Logrotate one-off and empty log handling
Logs are now rotated and recreated by **logrotate** as needed, and one-off logs
are only rotated when they change. With this setting in place, some tidying can be done:
- Empty one-off logs such as `gitlab-rails/gitlab-rails-db-migrate*.log` can be deleted.
- Empty logs which were rotated and compressed by older versions of GitLab. These
empty logs are usually 20 bytes in size.
### Run logrotate manually
Logrotate is a scheduled job but it can also be triggered on-demand.
To manually trigger GitLab log rotation with `logrotate`, use the following command:
```shell
/opt/gitlab/embedded/sbin/logrotate -fv -s /var/opt/gitlab/logrotate/logrotate.status /var/opt/gitlab/logrotate/logrotate.conf
```
### Increase how often logrotate is triggered
The logrotate script triggers every 50 minutes and waits for 10 minutes before attempting to rotate the logs.
To modify these values:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
logrotate['pre_sleep'] = 600 # sleep 10 minutes before rotating after start-up
logrotate['post_sleep'] = 3000 # wait 50 minutes after rotating
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## UDP log forwarding
DETAILS:
**Tier:** Premium, Ultimate
**Offering:** Self-managed
Linux package installations can utilize the UDP logging feature in svlogd as well as sending non-svlogd logs to a syslog-compatible remote system using UDP.
To configure a Linux package installation to send syslog-protocol messages via UDP, use the following settings:
```ruby
logging['udp_log_shipping_host'] = '1.2.3.4' # Your syslog server
# logging['udp_log_shipping_hostname'] = nil # Optional, defaults the system hostname
logging['udp_log_shipping_port'] = 1514 # Optional, defaults to 514 (syslog)
```
NOTE:
Setting `udp_log_shipping_host` will [add a `svlogd_prefix`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/logging.rb)
for the specified hostname and service for each of the [runit-managed](../architecture/index.md#runit) services.
Example log messages:
```plaintext
Jun 26 06:33:46 ubuntu1204-test production.log: Started GET "/root/my-project/import" for 127.0.0.1 at 2014-06-26 06:33:46 -0700
Jun 26 06:33:46 ubuntu1204-test production.log: Processing by ProjectsController#import as HTML
Jun 26 06:33:46 ubuntu1204-test production.log: Parameters: {"id"=>"root/my-project"}
Jun 26 06:33:46 ubuntu1204-test production.log: Completed 200 OK in 122ms (Views: 71.9ms | ActiveRecord: 12.2ms)
Jun 26 06:33:46 ubuntu1204-test gitlab_access.log: 172.16.228.1 - - [26/Jun/2014:06:33:46 -0700] "GET /root/my-project/import HTTP/1.1" 200 5775 "https://172.16.228.169/root/my-project/import" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
2014-06-26_13:33:46.49866 ubuntu1204-test sidekiq: 2014-06-26T13:33:46Z 18107 TID-7nbj0 Sidekiq::Extensions::DelayedMailer JID-bbfb118dd1db20f6c39f5b50 INFO: start
2014-06-26_13:33:46.52608 ubuntu1204-test sidekiq: 2014-06-26T13:33:46Z 18107 TID-7muoc RepositoryImportWorker JID-57ee926c3655fcfa062338ae INFO: start
```
## Using a custom NGINX log format
By default the NGINX access logs will use a version of the 'combined' NGINX
format, designed to hide potentially sensitive information embedded in query strings.
If you want to use a custom log format string you can specify it
in `/etc/gitlab/gitlab.rb` - see
[the NGINX documentation](http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format)
for format details.
```ruby
nginx['log_format'] = 'my format string $foo $bar'
mattermost_nginx['log_format'] = 'my format string $foo $bar'
```
## JSON logging
Structured logs can be exported via JSON to be parsed by Elasticsearch,
Splunk, or another log management system.
The JSON format is enabled by default for all services that support it.
NOTE:
PostgreSQL does not support JSON logging without an
external plugin. However, it does support logging in CSV format:
```ruby
postgresql['log_destination'] = 'csvlog'
postgresql['logging_collector'] = 'on'
```
A restart of the database is required for this to take effect. For more
details, see the
[PostgreSQL documentation](https://www.postgresql.org/docs/12/runtime-config-logging.html).
## Text logging
Customers with established log ingestion systems may not wish to use the JSON
log format. Text formatting can be configured by setting the following
in `/etc/gitlab/gitlab.rb` and then running `gitlab-ctl reconfigure` afterward:
```ruby
gitaly['configuration'] = {
logging: {
format: ""
}
}
gitlab_shell['log_format'] = 'text'
gitlab_workhorse['log_format'] = 'text'
registry['log_formatter'] = 'text'
sidekiq['log_format'] = 'text'
gitlab_pages['log_format'] = 'text'
```
NOTE:
There are a few variations in attribute names for the log format depending on the service involved (for example, Container Registry uses `log_formatter`, Gitaly and Praefect both use `logging_format`). See [Issue #4280](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4280) for more details.
## rbtrace
GitLab ships with [`rbtrace`](https://github.com/tmm1/rbtrace), which
allows you to trace Ruby code, view all running threads, take memory dumps,
and more. However, this is not enabled by default. To enable it, define the
`ENABLE_RBTRACE` variable to the environment:
```ruby
gitlab_rails['env'] = {"ENABLE_RBTRACE" => "1"}
```
Then reconfigure the system and restart Puma and Sidekiq. To run this
in a Linux package installation, run as root:
```ruby
/opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/rbtrace
```
## Configuring log level/verbosity
You can configure the minimum log levels (verbosity) for GitLab Rails,
Container Registry, GitLab Shell and Gitaly:
1. Edit `/etc/gitlab/gitlab.rb` and set the log levels:
```ruby
gitlab_rails['env'] = {
"GITLAB_LOG_LEVEL" => "WARN",
}
registry['log_level'] = 'info'
gitlab_shell['log_level'] = 'INFO'
gitaly['configuration'] = {
logging: {
level: "warn"
}
}
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
NOTE:
You [cannot edit](https://gitlab.com/groups/gitlab-org/-/epics/6034)
the `log_level` for certain GitLab logs, for example
`production_json.log`, `graphql_json.log`, and so on.
See also [Override default log level](https://docs.gitlab.com/ee/administration/logs/#override-default-log-level).
## Setting a custom log group
GitLab supports assigning a custom group to the configured [log directories](#configure-default-log-directories)
A global `logging['log_group']` setting in your `/etc/gitlab/gitlab.rb` file can
be configured as well as per-service `log_group` settings such as `gitaly['log_group']`.
You will need to run `sudo gitlab-ctl reconfigure` to configure your instance
when adding `log_group` settings.
Setting a global or per-service `log_group` will:
- Change the permissions on the per-service log directories (or all log directories
if using the global setting) to `0750` to allow the configured group members to
read the contents of the log directory.
- Configure [runit](#runit-logs) to write and rotate logs using the specified
`log_group` : either per-service or for all runit-managed services.
### Custom log group limitations
Logs for services not managed by runit (e.g. the `gitlab-rails` logs in
`/var/log/gitlab/gitlab-rails`) will not inherit the configured `log_group` setting.
The group must already exist on the host. Linux package installations don't create the group
when running `sudo gitlab-ctl reconfigure`.
doc/settings/memory_constrained_envs.md 0 → 100644
View file @ edb06ab2
---
stage: Data Stores
group: Cloud Connector
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Running GitLab in a memory-constrained environment
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
GitLab requires a significant amount of memory when running with all features enabled.
There are use-cases such as running GitLab on smaller installations where not all functionality
is required. Examples include:
- Running GitLab for personal use or very small teams.
- Using a small instance on a cloud provider for cost savings.
- Using resource-constrained devices like the Raspberry PI.
With some adjustments, GitLab can run comfortably on much lower specifications than described in
[minimum requirements](https://docs.gitlab.com/ee/install/requirements.html) or the
[reference architectures](https://docs.gitlab.com/ee/administration/reference_architectures/).
The following sections contain advice that allows GitLab to run in environments
that do not meet the minimum requirements. While most GitLab parts should be
functional with these settings in place, you may experience unexpected degradation
of both product functionality and performance. You should be able to run GitLab
with up to 5 developers with individual Git projects no larger than 100 MB.
## Minimum requirements for constrained environments
The minimum expected specs with which GitLab can be run are:
- Linux-based system (ideally Debian-based or RedHat-based)
- 4 CPU cores of ARM7/ARM64 or 1 CPU core of AMD64 architecture
- Minimum 2 GB of RAM + 1 GB of SWAP, optimally 2.5 GB of RAM + 1 GB of swap
- 20 GB of available storage
- A storage with a good random I/O performance with an order of preference:
- [SSD](https://en.wikipedia.org/wiki/Solid-state_drive)
- [eMMC](https://magazine.odroid.com/article/emmc-memory-modules-a-simple-guide/)
- [HDD](https://en.wikipedia.org/wiki/Hard_disk_drive)
- [high-performance A1-type SD card](https://www.sdcard.org/developers/sd-standard-overview/application-performance-class/)
Of the above list, the single-core performance of the CPU
and the random I/O performance of the storage have the highest impact.
Storage is especially relevant since in a constrained environment we expect some
amount of memory swapping to happen which puts more pressure on a used disk.
A common problem for the limited performance of small platforms is very slow disk storage,
which leads to a system-wide bottleneck.
With these minimal settings, the system should use swap during regular operation.
Since not all components are used at the same time, it should provide acceptable performance.
## Validate the performance of your system
There are number of tools available that allow you to validate the performance of your Linux-based system.
One of the projects that can aid with checking the performance of your system is [sbc-bench](https://github.com/ThomasKaiser/sbc-bench).
It describes all caveats of system testing and the impact of different behaviors on the performance of your system,
which is especially important when running GitLab in an embedded system. It can be used as a way to validate if the performance
of your system is good enough to run GitLab on a constrained environment.
These systems provide adequate performance to run a small installations of GitLab:
- [Raspberry PI 4 2GB](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/).
- [DigitalOcean Basic 2GB with SSD](https://www.digitalocean.com/pricing).
- [Scaleway DEV1-S 2GB/20GB](https://www.scaleway.com/en/pricing/).
- [GCS e2-small](https://cloud.google.com/compute/docs/machine-resource).
## Configure Swap
Before you install GitLab you need swap to be configured. Swap is a dedicated space on disk that
is used when physical RAM is full. When a Linux system runs out of RAM,
inactive pages are moved from RAM to the swap space.
Swap usage is often considered a problem as it can increase latency. However,
due to how GitLab functions, much of the memory that is allocated is not frequently accessed. Using
swap allows the application to run and function normally, and use swap only from time to time.
A general guideline is to configure swap to be around 50% of the available memory. For memory constrained
environments, it is advised to configure at least 1GB of swap for the system. There are a number of guides
on how to do it:
- [How to Add Swap Space on Ubuntu 20.04](https://linuxize.com/post/how-to-add-swap-space-on-ubuntu-20-04/)
- [How to Add Swap Space on CentOS 7](https://linuxize.com/post/how-to-add-swap-space-on-centos-7/)
Once configured, you should validate that swap is properly enabled:
```shell
free -h
total used free shared buff/cache available
Mem: 1.9Gi 115Mi 1.4Gi 0.0Ki 475Mi 1.6Gi
Swap: 1.0Gi 0B 1.0Gi
```
You might also configure how often the system will use the swap space with adjusting `/proc/sys/vm/swappiness`.
Swappiness ranges between `0` and `100`. The default value is `60`. A lower value reduces Linux's preference to free anonymous memory pages and write them to swap, but it increases its preference for doing the same with file-backed pages:
1. Configure it in the current session:
```shell
sudo sysctl vm.swappiness=10
```
1. Edit `/etc/sysctl.conf` to make it permanent:
```shell
vm.swappiness=10
```
## Install GitLab
In a memory-constrained environment, you should consider which GitLab distribution is right for you.
[GitLab Enterprise Edition (EE)](https://about.gitlab.com/install/) comes with significantly more features
than [GitLab Community Edition (CE)](https://about.gitlab.com/install/?version=ce), but all these additional features
increase compute and memory requirements.
When memory consumption is the primary concern, install GitLab CE. You can
always [upgrade to GitLab EE](https://docs.gitlab.com/ee/update/package/convert_to_ee.html) later.
## Optimize Puma
WARNING:
This is an experimental [Alpha feature](https://docs.gitlab.com/ee/policy/experiment-beta-support.html#alpha-features) and subject to change without notice. The feature
is not ready for production use. If you want to use this feature, we recommend testing
with non-production data first. See the [known issues](https://docs.gitlab.com/ee/administration/operations/puma.html#puma-single-mode-known-issues)
for additional details.
GitLab by default runs with a configuration that is designed to handle many concurrent connections.
For small installations, which do not require high throughput,
consider [disabling](https://docs.gitlab.com/ee/administration/operations/puma.html#memory-constrained-environments)
Puma [Clustered mode](https://github.com/puma/puma#clustered-mode).
As the result, only a single Puma process would serve the application.
In `/etc/gitlab/gitlab.rb`:
```ruby
puma['worker_processes'] = 0
```
We observed 100-400MB of memory usage reduction configuring Puma this way.
## Optimize Sidekiq
Sidekiq is a background processing daemon. When configured with GitLab by default
it runs with a concurrency mode of `20`. This does impact how much memory it can
allocate at a given time. It is advised to configure it to use a significantly
smaller value of `5` or `10` (preferred).
In `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['concurrency'] = 10
```
## Optimize Gitaly
Gitaly is a storage service that allows efficient access to Git-based repositories.
It is advised to configure a maximum concurrency and memory limits enforced by Gitaly.
In `/etc/gitlab/gitlab.rb`:
```ruby
gitaly['configuration'] = {
concurrency: [
{
'rpc' => "/gitaly.SmartHTTPService/PostReceivePack",
'max_per_repo' => 3,
}, {
'rpc' => "/gitaly.SSHService/SSHUploadPack",
'max_per_repo' => 3,
},
],
cgroups: {
repositories: {
count: 2,
},
mountpoint: '/sys/fs/cgroup',
hierarchy_root: 'gitaly',
memory_bytes: 500000,
cpu_shares: 512,
},
}
gitaly['env'] = {
'GITALY_COMMAND_SPAWN_MAX_PARALLEL' => '2'
}
```
## Disable monitoring
GitLab enables all services by default to provide a complete DevOps solution without any additional configuration.
Some of the default services, like monitoring, are not essential for GitLab to function and can be disabled to save memory.
In `/etc/gitlab/gitlab.rb`:
```ruby
prometheus_monitoring['enable'] = false
```
We observed 200MB of memory usage reduction configuring GitLab this way.
## Configure how GitLab handles memory
GitLab consists of many components (written in Ruby and Go),
with GitLab Rails being the biggest one and consuming the most of memory.
GitLab Rails uses [jemalloc](https://github.com/jemalloc/jemalloc) as a memory
allocator. [jemalloc](https://github.com/jemalloc/jemalloc) preallocates memory in
bigger chunks that are also being held for longer periods in order to improve performance.
At the expense of some performance loss, you can configure GitLab to free memory right after
it is no longer needed instead of holding it for a longer periods.
In `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['env'] = {
'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000'
}
gitaly['env'] = {
'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000'
}
```
We observed much more stable memory usage during the execution of the application.
## Disable additional in-application monitoring
GitLab uses internal data structures to measure different aspects of itself.
These features are no longer needed if monitoring is disabled.
To disable these features you need to go to Admin Area of GitLab
and disable the Prometheus Metrics feature:
1. On the left sidebar, at the bottom, select **Admin Area**.
1. Select **Settings > Metrics and profiling**.
1. Expand **Metrics - Prometheus**.
1. Disable **Enable Prometheus Metrics**.
1. Select **Save changes**.
## Configuration with all the changes
1. If you apply everything described so far, your `/etc/gitlab/gitlab.rb` file
should contain the following configuration:
```ruby
puma['worker_processes'] = 0
sidekiq['concurrency'] = 10
prometheus_monitoring['enable'] = false
gitlab_rails['env'] = {
'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000'
}
gitaly['configuration'] = {
concurrency: [
{
'rpc' => "/gitaly.SmartHTTPService/PostReceivePack",
'max_per_repo' => 3,
}, {
'rpc' => "/gitaly.SSHService/SSHUploadPack",
'max_per_repo' => 3,
},
],
cgroups: {
repositories: {
count: 2,
},
mountpoint: '/sys/fs/cgroup',
hierarchy_root: 'gitaly',
memory_bytes: 500000,
cpu_shares: 512,
},
}
gitaly['env'] = {
'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000',
'GITALY_COMMAND_SPAWN_MAX_PARALLEL' => '2'
}
```
1. After you make all these changes, reconfigure GitLab to use the new settings:
```shell
sudo gitlab-ctl reconfigure
```
This operation could take a while, since GitLab did not work
with memory conservative settings up-to this point.
## Performance results
After applying the above configuration, you can expect the following memory usage:
```plaintext
total used free shared buff/cache available
Mem: 1.9Gi 1.7Gi 151Mi 31Mi 132Mi 102Mi
Swap: 1.0Gi 153Mi 870Mi
```
doc/settings/microsoft_graph_mailer.md 0 → 100644
View file @ edb06ab2
---
stage: Manage
group: Authentication and Authorization
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#designated-technical-writers
---
# Microsoft Graph Mailer settings
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8259) in GitLab 15.5.
Prerequisites:
- To use the Microsoft Graph API to send mails, you must first
[create an application](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app)
in the Azure Active Directory, and add the `Mail.Send`
[application permission](https://learn.microsoft.com/en-us/graph/permissions-reference).
- Set the application permissions to **App-only access**. Make sure the permissions are not set to **Delegated**.
If you would rather send application emails using [Microsoft Graph API](https://learn.microsoft.com/en-us/graph/api/user-sendmail?view=graph-rest-1.0&tabs=http)
with [OAuth 2.0 client credentials flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow),
add the following configuration information to `/etc/gitlab/gitlab.rb` and run `gitlab-ctl reconfigure`.
```ruby
# The originating email address for outgoing mail
gitlab_rails['gitlab_email_from'] = '<YOUR_ACCOUNT_EMAIL>'
# The reply-to email address
gitlab_rails['gitlab_email_reply_to'] = '<YOUR_ACCOUNT_EMAIL>'
gitlab_rails['microsoft_graph_mailer_enabled'] = true
# The unique identifier for the user. To use Microsoft Graph on behalf of the user.
gitlab_rails['microsoft_graph_mailer_user_id'] = "<YOUR_USER_ID>"
# The directory tenant the application plans to operate against, in GUID or domain-name format.
gitlab_rails['microsoft_graph_mailer_tenant'] = "<YOUR_TENANT_ID>"
# The application ID that's assigned to your app. You can find this information in the portal where you registered your app.
gitlab_rails['microsoft_graph_mailer_client_id'] = "<YOUR_CLIENT_ID>"
# The client secret that you generated for your app in the app registration portal.
gitlab_rails['microsoft_graph_mailer_client_secret'] = "<YOUR_CLIENT_SECRET_ID>"
gitlab_rails['microsoft_graph_mailer_azure_ad_endpoint'] = "https://login.microsoftonline.com"
gitlab_rails['microsoft_graph_mailer_graph_endpoint'] = "https://graph.microsoft.com"
```
## Troubleshooting
### `ErrorSendAsDenied`
The full error message is:
```plaintext
"ErrorSendAsDenied","message":"The user account which was used to submit this request does not have the right to send mail on behalf of the specified sending account., Cannot submit message."
```
To resolve this error:
1. Verify your API permissions are correct by reviewing the [application permission](https://learn.microsoft.com/en-us/graph/permissions-reference).
1. Set the following fields to the email address for the account you're using:
- `gitlab_rails['gitlab_email_from']`.
- `gitlab_rails['gitlab_email_reply_to']`.
Other than permissions, this error is sometimes caused because the server does not allow the default `gitlab_email_from` value to be used. You should set the value to the email address for the account you're authenticating with.
### `Tail logs`
For troubleshooting, use the `tail logs` command to view live GitLab log updates:
```ruby
# Tail all logs for the application
sudo gitlab-ctl tail
# Tail logs for an application sub-directory
sudo gitlab-ctl tail gitlab-rails
# Tail logs for an individual file in the application
sudo gitlab-ctl tail nginx/gitlab_error.log
```
To stop any of these commands, press <kbd>Control</kbd>+<kbd>C</kbd>.
doc/settings/nginx.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# NGINX settings
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
## Service-specific NGINX settings
Users can configure NGINX settings differently for different services via
`gitlab.rb`. Settings for the GitLab Rails application can be configured using the
`nginx['<some setting>']` keys. There are similar keys for other services like
`pages_nginx`, `mattermost_nginx` and `registry_nginx`. All the configurations
available for `nginx` are also available for these `<service_nginx>` settings and
share the same default values as GitLab NGINX.
To operate NGINX for isolated services like Mattermost, you should use `gitlab_rails['enable'] = false` instead of `nginx['enable'] = false`.
For more information, see [Running GitLab Mattermost on its own server](https://docs.gitlab.com/ee/integration/mattermost/#running-gitlab-mattermost-on-its-own-server).
If modifying via `gitlab.rb`, users have to configure NGINX setting for each
service separately. Settings given via `nginx['foo']` WILL NOT be replicated to
service specific NGINX configuration (as `registry_nginx['foo']` or
`mattermost_nginx['foo']`, etc.). For example, to configure HTTP to HTTPS
redirection for GitLab, Mattermost and Registry, the following settings should
be added to `gitlab.rb`:
```ruby
nginx['redirect_http_to_https'] = true
registry_nginx['redirect_http_to_https'] = true
mattermost_nginx['redirect_http_to_https'] = true
```
NOTE:
Modifying NGINX configuration should be done with care as incorrect
or incompatible configuration may yield to unavailability of service.
## Enable HTTPS
By default, Linux package installations do not use HTTPS. If you want to enable HTTPS for
`gitlab.example.com`, you can:
- [Use Let's Encrypt for free, automated HTTPS](ssl/index.md#enable-the-lets-encrypt-integration).
- [Manually configure HTTPS with your own certificates](ssl/index.md#configure-https-manually).
NOTE:
If you use a proxy, load balancer or some other external device to terminate SSL for the GitLab host name,
see [External, proxy, and load balancer SSL termination](ssl/index.md#configure-a-reverse-proxy-or-load-balancer-ssl-termination).
## Change the default proxy headers
By default, when you specify `external_url`, a Linux package installation sets a few
NGINX proxy headers that are assumed to be sane in most environments.
For example, a Linux package installation sets:
```plaintext
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
```
if you have specified `https` schema in the `external_url`.
However, if you have a situation where your GitLab is in a more complex setup
like behind a reverse proxy, you will need to tweak the proxy headers in order
to avoid errors like `The change you wanted was rejected` or
`Can't verify CSRF token authenticity Completed 422 Unprocessable`.
This can be achieved by overriding the default headers, eg. specify
in `/etc/gitlab/gitlab.rb`:
```ruby
nginx['proxy_set_headers'] = {
"X-Forwarded-Proto" => "http",
"CUSTOM_HEADER" => "VALUE"
}
```
Save the file and [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure)
for the changes to take effect.
This way you can specify any header supported by NGINX you require.
## Configuring GitLab `trusted_proxies` and the NGINX `real_ip` module
By default, NGINX and GitLab will log the IP address of the connected client.
If your GitLab is behind a reverse proxy, you may not want the IP address of
the proxy to show up as the client address.
You can have NGINX look for a different address to use by adding your reverse
proxy to the `real_ip_trusted_addresses` list:
```ruby
# Each address is added to the the NGINX config as 'set_real_ip_from <address>;'
nginx['real_ip_trusted_addresses'] = [ '192.168.1.0/24', '192.168.2.1', '2001:0db8::/32' ]
# other real_ip config options
nginx['real_ip_header'] = 'X-Forwarded-For'
nginx['real_ip_recursive'] = 'on'
```
Description of the options:
- <http://nginx.org/en/docs/http/ngx_http_realip_module.html>
By default, Linux package installations use the IP addresses in `real_ip_trusted_addresses`
as GitLab trusted proxies, which will keep users from being listed as signed
in from those IPs.
Save the file and [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure)
for the changes to take effect.
## Configuring the PROXY protocol
If you want to use a proxy like HAProxy in front of GitLab using the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt),
you need to enable this setting. Do not forget to set the `real_ip_trusted_addresses` also as needed:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
# Enable termination of ProxyProtocol by NGINX
nginx['proxy_protocol'] = true
# Configure trusted upstream proxies. Required if `proxy_protocol` is enabled.
nginx['real_ip_trusted_addresses'] = [ "127.0.0.0/8", "IP_OF_THE_PROXY/32"]
```
1. Save the file and
[reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure)
for the changes to take effect.
Once enabled, NGINX only accepts PROXY protocol traffic on these listeners.
Ensure to also adjust any other environments you might have, like monitoring checks.
## Using a non-bundled web-server
By default, the Linux package installs GitLab with bundled NGINX.
Linux package installations allow webserver access through the `gitlab-www` user, which resides
in the group with the same name. To allow an external webserver access to
GitLab, the external webserver user needs to be added to the `gitlab-www` group.
To use another web server like Apache or an existing NGINX installation you
will have to perform the following steps:
1. **Disable bundled NGINX**
In `/etc/gitlab/gitlab.rb` set:
```ruby
nginx['enable'] = false
```
1. **Set the username of the non-bundled web-server user**
By default, Linux package installations have no default setting for the external webserver
user, you have to specify it in the configuration. For Debian/Ubuntu the
default user is `www-data` for both Apache/NGINX whereas for RHEL/CentOS
the NGINX user is `nginx`.
Make sure you have first installed Apache/NGINX so the webserver user is created, otherwise a Linux package installation
fails while reconfiguring.
Let's say for example that the webserver user is `www-data`.
In `/etc/gitlab/gitlab.rb` set:
```ruby
web_server['external_users'] = ['www-data']
```
This setting is an array so you can specify more than one user to be added to `gitlab-www` group.
Run `sudo gitlab-ctl reconfigure` for the change to take effect.
If you are using SELinux and your web server runs under a restricted SELinux profile you may have to [loosen the restrictions on your web server](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache#selinux-modifications).
Make sure that the webserver user has the correct permissions on all directories used by external web-server, otherwise you will receive `failed (XX: Permission denied) while reading upstream` errors.
1. **Add the non-bundled web-server to the list of trusted proxies**
Normally, Linux package installations default the list of trusted proxies to what was
configured in the `real_ip` module for the bundled NGINX.
For non-bundled web-servers the list needs to be configured directly, and should
include the IP address of your web-server if it is not on the same machine as GitLab.
Otherwise, users will be shown as being signed in from your web-server's IP address.
```ruby
gitlab_rails['trusted_proxies'] = [ '192.168.1.0/24', '192.168.2.1', '2001:0db8::/32' ]
```
1. **(Optional) Set the right GitLab Workhorse settings if using Apache**
Apache cannot connect to a UNIX socket but instead needs to connect to a
TCP Port. To allow GitLab Workhorse to listen on TCP (by default port 8181)
edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_workhorse['listen_network'] = "tcp"
gitlab_workhorse['listen_addr'] = "127.0.0.1:8181"
```
Run `sudo gitlab-ctl reconfigure` for the change to take effect.
1. **Download the correct web server configuration**
Go to [GitLab repository](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/support/nginx) and download
the required configuration. Select the correct configuration file depending whether you are serving GitLab with
SSL or not. You might need to make some changes, such as:
- The value of `YOUR_SERVER_FQDN` set to your FQDN.
- If you use SSL, the location of your SSL keys.
- The location of your log files.
## Setting the NGINX listen address or addresses
By default NGINX will accept incoming connections on all local IPv4 addresses.
You can change the list of addresses in `/etc/gitlab/gitlab.rb`.
```ruby
# Listen on all IPv4 and IPv6 addresses
nginx['listen_addresses'] = ["0.0.0.0", "[::]"]
registry_nginx['listen_addresses'] = ['*', '[::]']
mattermost_nginx['listen_addresses'] = ['*', '[::]']
pages_nginx['listen_addresses'] = ['*', '[::]']
```
## Setting the NGINX listen port
By default NGINX will listen on the port specified in `external_url` or
implicitly use the right port (80 for HTTP, 443 for HTTPS). If you are running
GitLab behind a reverse proxy, you may want to override the listen port to
something else. For example, to use port 8081:
```ruby
nginx['listen_port'] = 8081
```
## Verbosity level of NGINX logs
By default NGINX will log at the `error` verbosity level. You may log at a different level
by changing the log level. For example, to enable `debug` logging:
```ruby
nginx['error_log_level'] = "debug"
```
Valid values can be found from the [NGINX documentation](https://nginx.org/en/docs/ngx_core_module.html#error_log).
## Setting the Referrer-Policy header
By default, GitLab sets the `Referrer-Policy` header to `strict-origin-when-cross-origin` on all responses.
This makes the client send the full URL as referrer when making a same-origin request but only send the
origin when making cross-origin requests.
To set this header to a different value:
```ruby
nginx['referrer_policy'] = 'same-origin'
```
You can also disable this header to make the client use its default setting:
```ruby
nginx['referrer_policy'] = false
```
Note that setting this to `origin` or `no-referrer` would break some features in GitLab that require the full referrer URL.
- <https://www.w3.org/TR/referrer-policy/>
## Disabling Gzip compression
By default, GitLab enables Gzip compression for text data over 10240 bytes. To
disable this behavior:
```ruby
nginx['gzip_enabled'] = false
```
NOTE:
The `gzip` setting only works for the main GitLab application and not for the other services.
## Disabling proxy request buffering
Request buffering can be disabled selectively on specific locations by changing `request_buffering_off_path_regex`.
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['request_buffering_off_path_regex'] = "/api/v\\d/jobs/\\d+/artifacts$|/import/gitlab_project$|\\.git/git-receive-pack$|\\.git/gitlab-lfs/objects|\\.git/info/lfs/objects/batch$"
```
1. Reconfigure GitLab, and [HUP](https://nginx.org/en/docs/control.html)
NGINX to cause it to reload with the updated configuration gracefully:
```shell
sudo gitlab-ctl reconfigure
sudo gitlab-ctl hup nginx
```
## Configure `robots.txt`
To configure [`robots.txt`](https://www.robotstxt.org/robotstxt.html) for your instance, specify a custom `robots.txt` file by adding a [custom NGINX configuration](#inserting-custom-nginx-settings-into-the-gitlab-server-block):
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['custom_gitlab_server_config'] = "\nlocation =/robots.txt { alias /path/to/custom/robots.txt; }\n"
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Inserting custom NGINX settings into the GitLab server block
Please keep in mind that these custom settings may create conflicts if the
same settings are defined in your `gitlab.rb` file.
If you need to add custom settings into the NGINX `server` block for GitLab for
some reason you can use the following setting.
```ruby
# Example: block raw file downloads from a specific repository
nginx['custom_gitlab_server_config'] = "location ^~ /foo-namespace/bar-project/raw/ {\n deny all;\n}\n"
```
Run `gitlab-ctl reconfigure` to rewrite the NGINX configuration and restart
NGINX.
This inserts the defined string into the end of the `server` block of
`/var/opt/gitlab/nginx/conf/gitlab-http.conf`.
### Notes
- If you're adding a new location, you might need to include
```conf
proxy_cache off;
proxy_http_version 1.1;
proxy_pass http://gitlab-workhorse;
```
in the string or in the included NGINX configuration. Without these, any sub-location
will return a 404. See
[GitLab CE Issue #30619](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30619).
- You cannot add the root `/` location or the `/assets` location as those already
exist in `gitlab-http.conf`.
## Inserting custom settings into the NGINX configuration
If you need to add custom settings into the NGINX configuration, for example to include
existing server blocks, you can use the following setting.
```ruby
# Example: include a directory to scan for additional config files
nginx['custom_nginx_config'] = "include /etc/gitlab/nginx/sites-enabled/*.conf;"
```
You should create custom server blocks in the `/etc/gitlab/nginx/sites-available` directory. To enable them, symlink them into the
`/etc/gitlab/nginx/sites-enabled` directory:
1. Create the `/etc/gitlab/nginx/sites-enabled` directory.
1. Run the following command:
```shell
sudo ln -s /etc/gitlab/nginx/sites-available/example.conf /etc/gitlab/nginx/sites-enabled/example.conf
```
You can add domains for server blocks [as an alternative name](ssl/index.md#add-alternative-domains-to-the-certificate)
to the generated Let's Encrypt SSL certificate.
Run `gitlab-ctl reconfigure` to rewrite the NGINX configuration and restart
NGINX. You must reload NGINX (`gitlab-ctl hup nginx`) or restart NGINX (`gitlab-ctl restart nginx`) whenever you make changes to the custom server blocks.
This inserts the defined string into the end of the `http` block of
`/var/opt/gitlab/nginx/conf/nginx.conf`.
Custom NGINX settings inside the `/etc/gitlab/` directory are backed up to `/etc/gitlab/config_backup/`
during an upgrade and when `sudo gitlab-ctl backup-etc` is manually executed.
## Custom error pages
You can use `custom_error_pages` to modify text on the default GitLab error page.
This can be used for any valid HTTP error code; e.g 404, 502.
As an example the following would modify the default 404 error page.
```ruby
nginx['custom_error_pages'] = {
'404' => {
'title' => 'Example title',
'header' => 'Example header',
'message' => 'Example message'
}
}
```
This would result in the 404 error page below.
![custom 404 error page](img/error_page_example.png)
Run `gitlab-ctl reconfigure` to rewrite the NGINX configuration and restart
NGINX.
## Using an existing Passenger/NGINX installation
In some cases you may want to host GitLab using an existing Passenger/NGINX
installation but still have the convenience of updating and installing using
the Linux packages.
NOTE:
When disabling NGINX, you won't be able to access other services included in a Linux package installation such as
Mattermost unless you manually add them in `nginx.conf`.
### Configuration
First, you'll need to set up your `/etc/gitlab/gitlab.rb` to disable the built-in
NGINX and Puma:
```ruby
# Define the external url
external_url 'http://git.example.com'
# Disable the built-in nginx
nginx['enable'] = false
# Disable the built-in puma
puma['enable'] = false
# Set the internal API URL
gitlab_rails['internal_api_url'] = 'http://git.example.com'
# Define the web server process user (ubuntu/nginx)
web_server['external_users'] = ['www-data']
```
Make sure you run `sudo gitlab-ctl reconfigure` for the changes to take effect.
### Vhost (server block)
In your custom Passenger/NGINX installation, create the following site configuration file:
```plaintext
upstream gitlab-workhorse {
server unix://var/opt/gitlab/gitlab-workhorse/sockets/socket fail_timeout=0;
}
server {
listen *:80;
server_name git.example.com;
server_tokens off;
root /opt/gitlab/embedded/service/gitlab-rails/public;
client_max_body_size 250m;
access_log /var/log/gitlab/nginx/gitlab_access.log;
error_log /var/log/gitlab/nginx/gitlab_error.log;
# Ensure Passenger uses the bundled Ruby version
passenger_ruby /opt/gitlab/embedded/bin/ruby;
# Correct the $PATH variable to included packaged executables
passenger_env_var PATH "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/usr/local/bin:/usr/bin:/bin";
# Make sure Passenger runs as the correct user and group to
# prevent permission issues
passenger_user git;
passenger_group git;
# Enable Passenger & keep at least one instance running at all times
passenger_enabled on;
passenger_min_instances 1;
location ~ ^/[\w\.-]+/[\w\.-]+/(info/refs|git-upload-pack|git-receive-pack)$ {
# 'Error' 418 is a hack to re-use the @gitlab-workhorse block
error_page 418 = @gitlab-workhorse;
return 418;
}
location ~ ^/[\w\.-]+/[\w\.-]+/repository/archive {
# 'Error' 418 is a hack to re-use the @gitlab-workhorse block
error_page 418 = @gitlab-workhorse;
return 418;
}
location ~ ^/api/v3/projects/.*/repository/archive {
# 'Error' 418 is a hack to re-use the @gitlab-workhorse block
error_page 418 = @gitlab-workhorse;
return 418;
}
# Build artifacts should be submitted to this location
location ~ ^/[\w\.-]+/[\w\.-]+/builds/download {
client_max_body_size 0;
# 'Error' 418 is a hack to re-use the @gitlab-workhorse block
error_page 418 = @gitlab-workhorse;
return 418;
}
# Build artifacts should be submitted to this location
location ~ /ci/api/v1/builds/[0-9]+/artifacts {
client_max_body_size 0;
# 'Error' 418 is a hack to re-use the @gitlab-workhorse block
error_page 418 = @gitlab-workhorse;
return 418;
}
# Build artifacts should be submitted to this location
location ~ /api/v4/jobs/[0-9]+/artifacts {
client_max_body_size 0;
# 'Error' 418 is a hack to re-use the @gitlab-workhorse block
error_page 418 = @gitlab-workhorse;
return 418;
}
# For protocol upgrades from HTTP/1.0 to HTTP/1.1 we need to provide Host header if its missing
if ($http_host = "") {
# use one of values defined in server_name
set $http_host_with_default "git.example.com";
}
if ($http_host != "") {
set $http_host_with_default $http_host;
}
location @gitlab-workhorse {
## https://github.com/gitlabhq/gitlabhq/issues/694
## Some requests take more than 30 seconds.
proxy_read_timeout 3600;
proxy_connect_timeout 300;
proxy_redirect off;
# Do not buffer Git HTTP responses
proxy_buffering off;
proxy_set_header Host $http_host_with_default;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_pass http://gitlab-workhorse;
## The following settings only work with NGINX 1.7.11 or newer
#
## Pass chunked request bodies to gitlab-workhorse as-is
# proxy_request_buffering off;
# proxy_http_version 1.1;
}
## Enable gzip compression as per rails guide:
## http://guides.rubyonrails.org/asset_pipeline.html#gzip-compression
## WARNING: If you are using relative urls remove the block below
## See config/application.rb under "Relative url support" for the list of
## other files that need to be changed for relative url support
location ~ ^/(assets)/ {
root /opt/gitlab/embedded/service/gitlab-rails/public;
gzip_static on; # to serve pre-gzipped version
expires max;
add_header Cache-Control public;
}
error_page 502 /502.html;
}
```
Don't forget to update `git.example.com` in the above example to be your server URL.
If you wind up with a 403 forbidden, it's possible that you haven't enabled passenger in `/etc/nginx/nginx.conf`,
to do so simply uncomment:
```plaintext
# include /etc/nginx/passenger.conf;
```
Then run `sudo service nginx reload`.
## Enabling/Disabling `nginx_status`
By default you will have an NGINX health-check endpoint configured at `127.0.0.1:8060/nginx_status` to monitor your NGINX server status.
### The following information will be displayed
```plaintext
Active connections: 1
server accepts handled requests
18 18 36
Reading: 0 Writing: 1 Waiting: 0
```
- Active connections – Open connections in total.
- 3 figures are shown.
- All accepted connections.
- All handled connections.
- Total number of handled requests.
- Reading: NGINX reads request headers
- Writing: NGINX reads request bodies, processes requests, or writes responses to a client
- Waiting: Keep-alive connections. This number depends on the keepalive-timeout.
### Configuration options
Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['status'] = {
"listen_addresses" => ["127.0.0.1"],
"fqdn" => "dev.example.com",
"port" => 9999,
"options" => {
"access_log" => "on", # Disable logs for stats
"allow" => "127.0.0.1", # Only allow access from localhost
"deny" => "all" # Deny access to anyone else
}
}
```
If you don't find this service useful for your current infrastructure you can disable it with:
```ruby
nginx['status'] = {
'enable' => false
}
```
Make sure you run `sudo gitlab-ctl reconfigure` for the changes to take effect.
#### Warning
To ensure that user uploads are accessible your NGINX user (usually `www-data`)
should be added to the `gitlab-www` group. This can be done using the following command:
```shell
sudo usermod -aG gitlab-www www-data
```
## Templates
Other than the Passenger configuration in place of Puma and the lack of HTTPS
(although this could be enabled) these files are mostly identical to:
- [Bundled GitLab NGINX configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb)
Don't forget to restart NGINX to load the new configuration (on Debian-based
systems `sudo service nginx restart`).
## Troubleshooting
### 400 Bad Request: too many Host headers
Make sure you don't have the `proxy_set_header` configuration in
`nginx['custom_gitlab_server_config']` settings and instead use the
['proxy_set_headers'](ssl/index.md#configure-a-reverse-proxy-or-load-balancer-ssl-termination) configuration in your `gitlab.rb` file.
### `javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure`
Linux package installations don't support TLSv1 protocol by default.
This can cause connection issues with some older Java based IDE clients when interacting with
your GitLab instance.
We strongly urge you to upgrade ciphers on your server, similar to what was mentioned
in [this user comment](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/624#note_299061).
If it is not possible to make this server change, you can default back to the old
behavior by changing the values in your `/etc/gitlab/gitlab.rb`:
```ruby
nginx['ssl_protocols'] = "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"
```
### Mismatch between private key and certificate
If you see `x509 certificate routines:X509_check_private_key:key values mismatch)` in the NGINX logs (`/var/log/gitlab/nginx/current` by default for Omnibus), there is a mismatch between your private key and certificate.
To fix this, you will need to match the correct private key with your certificate.
To ensure you have the correct key and certificate, you can ensure that the modulus of the private key and certificate match:
```shell
/opt/gitlab/embedded/bin/openssl rsa -in /etc/gitlab/ssl/gitlab.example.com.key -noout -modulus | /opt/gitlab/embedded/bin/openssl sha256
/opt/gitlab/embedded/bin/openssl x509 -in /etc/gitlab/ssl/gitlab.example.com.crt -noout -modulus| /opt/gitlab/embedded/bin/openssl sha256
```
Once you verify that they match, you will need to reconfigure and reload NGINX:
```shell
sudo gitlab-ctl reconfigure
sudo gitlab-ctl hup nginx
```
### Request Entity Too Large
If you see `Request Entity Too Large` in the [NGINX logs](https://docs.gitlab.com/ee/administration/logs/index.html#nginx-logs),
you will need to increase the [Client Max Body Size](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size).
You may encounter this error if you have increased the [Max import size](https://docs.gitlab.com/ee/administration/settings/account_and_limit_settings.html#max-import-size).
In a Kubernetes-based GitLab installation, this setting is
[named differently](https://docs.gitlab.com/charts/charts/gitlab/webservice/#proxybodysize).
To increase the `client_max_body_size`, you will need to set the value in your `/etc/gitlab/gitlab.rb`:
```ruby
nginx['client_max_body_size'] = '250m'
```
Make sure you run `sudo gitlab-ctl reconfigure` and run `sudo gitlab-ctl hup nginx` to cause NGINX to
[reload the with the updated configuration](http://nginx.org/en/docs/control.html)
To increase the `client_max_body_size`:
1. Edit `/etc/gitlab/gitlab.rb` and set the preferred value:
```ruby
nginx['client_max_body_size'] = '250m'
```
1. Reconfigure GitLab, and [HUP](https://nginx.org/en/docs/control.html)
NGINX to cause it to reload with the updated configuration gracefully:
```shell
sudo gitlab-ctl reconfigure
sudo gitlab-ctl hup nginx
```
### Security scan is showing a "NGINX HTTP Server Detection" warning
Some security scanners detect issues when they see the `Server: nginx` http header. Most scanners with this alert will
notify as `Low` or `Info` severity. [See Nessus as an example](https://www.tenable.com/plugins/nessus/106375).
We recommend ignoring this warning, as the benefit of removing the header is low, and its presence
[helps support the NGINX project in usage statistics](https://trac.nginx.org/nginx/ticket/1644). We do provide a way to turn off the
header with `hide_server_tokens`:
1. Edit `/etc/gitlab/gitlab.rb` and set the value:
```ruby
nginx['hide_server_tokens'] = 'on'
```
1. Reconfigure GitLab, and [hup](https://nginx.org/en/docs/control.html)
NGINX to cause it to reload the with the updated configuration gracefully:
```shell
sudo gitlab-ctl reconfigure
sudo gitlab-ctl hup nginx
```
### `Branch 'branch_name' was not found in this project's repository` when Web IDE and external NGINX are used
If you get this error, check your NGINX configuration file if you have a trailing slash in `proxy_pass` and remove it:
1. Edit your NGINX configuration file:
```plaintext
proxy_pass https://1.2.3.4;
```
1. Restart NGINX:
```shell
sudo systemctl restart nginx
```
### GitLab is presenting 502 errors and `worker_connections are not enough` in logs
If you get a `worker_connections are not enough` error in the logs, configure the NGINX worker connections to a higher value:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
gitlab['nginx']['worker_connections'] = 10240
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
The default value is [10240 connections](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/86f98401113eb843658b63d45d58be8b706216f3/files/gitlab-cookbooks/gitlab/attributes/default.rb#L750).
doc/settings/praefect.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Gitaly Cluster
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
[Gitaly Cluster](https://docs.gitlab.com/ee/administration/gitaly/praefect.html) provides
fault-tolerant storage for repositories. It uses Praefect as a router and transaction manager for
Gitaly.
## Enable Gitaly Cluster
By default, Gitaly Cluster is not enabled. For information on enabling Gitaly Cluster, see
the Gitaly Cluster [setup instructions](https://docs.gitlab.com/ee/administration/gitaly/praefect.html#setup-instructions).
## Update GitLab when Gitaly Cluster is enabled
For information on updating GitLab with Gitaly Cluster enabled, see the
[specific instructions](https://docs.gitlab.com/ee/update/zero_downtime.html#gitaly-cluster).
doc/settings/prometheus.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Prometheus settings
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
## Remote read/write
Prometheus supports reading and writing to remote services.
To configure a remote read or write service, you can include the following in `gitlab.rb`.
```ruby
prometheus['remote_write'] = [
{
url: 'https://some-remote-write-service.example.com',
basic_auth: {
password: 'remote write secret password'
}
}
]
prometheus['remote_read'] = [
{
url: 'https://some-remote-write-service.example.com'
}
]
```
For more documentation on the options available, see the [remote write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cremote_write%3E) and [remote read](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cremote_read%3E) sections of the official documentation.
## Rules files
Prometheus allows for [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) and [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) rules.
Linux package installations include some [default rules files](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/files/gitlab-cookbooks/monitoring/templates/rules)
that are stored in `/var/opt/gitlab/prometheus/rules/`.
To override the default rules, you can change the default list in `gitlab.rb.`.
No rules:
```ruby
prometheus['rules_files'] = []
```
Custom list:
```ruby
prometheus['rules_files'] = ['/path/to/rules/*.rules', '/path/to/single/file.rules']
```
## External labels
To set [external labels](https://prometheus.io/docs/prometheus/latest/configuration/configuration/):
```ruby
prometheus['external_labels'] = {
'region' => 'us-west-2',
'source' => 'omnibus',
}
```
No external labels are set by default.
## node_exporter
The node_exporter provides system level metrics.
Additional metrics collectors are enabled by default. For example, `mountstats` is used to collect metrics about NFS mounts.
To disable the `mountstats` collector, adjust `gitlab.rb` with the following setting and run `gitlab-ctl reconfigure`:
```ruby
node_exporter['flags'] = {
'collector.mountstats' => false,
}
```
For more information on available collectors, see the [upstream documentation](https://github.com/prometheus/node_exporter#collectors).
## Alertmanager options
You can set [global options](https://prometheus.io/docs/alerting/latest/configuration/)
for the [Alertmanager](https://prometheus.io/docs/alerting/latest/configuration/).
For example, the following `gitlab.rb` configuration overrides the hostname Alertmanager
uses to identify itself to the SMTP server:
```ruby
alertmanager['global'] = {
'smtp_hello' => 'example.org'
}
```
### Additional receivers and routes
In this example, we implement a new receiver for VictorOps.
1. Edit `/etc/gitlab/gitlab.rb` to add a new receiver and define a [route](https://prometheus.io/docs/alerting/latest/configuration/#route):
```ruby
alertmanager['receivers'] = [
{
'name' => 'victorOps-receiver',
'victorops_configs' => [
{
'routing_key' => 'Sample_route',
'api_key' => '558e7ebc-XXXX-XXXX-XXXX-XXXXXXXXXXXX',
'entity_display_name' => '{{ .CommonAnnotations.summary }}',
'message_type' => '{{ .CommonLabels.severity }}',
'state_message' => 'Alert: {{ .CommonLabels.alertname }}. Summary:{{ .CommonAnnotations.summary }}. RawData: {{ .CommonLabels }}',
'http_config' => {
proxy_url: 'http://internet.proxy.com:3128'
}
} #, { Next receiver }
]
}
]
alertmanager['routes'] = [
{
'receiver' => 'victorOps-receiver',
'group_wait' => '30s',
'group_interval' => '5m',
'repeat_interval' => '3h',
'matchers' => [ 'severity = high' ]
} #, { Next route }
]
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
Alertmanager will now route `severity = high` alerts to `victorops-receiver`.
Read more about VictorOps options for Alertmanager at the [VictorOps documentation](https://help.victorops.com/knowledge-base/victorops-prometheus-integration/).
doc/settings/redis.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Configuring Redis
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
## Using an alternate local Redis instance
Linux package installations include Redis by default. To direct the GitLab
application to your own *locally* running Redis instance:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
# Disable the bundled Redis
redis['enable'] = false
# Redis via TCP
gitlab_rails['redis_host'] = '127.0.0.1'
gitlab_rails['redis_port'] = 6379
# OR Redis via Unix domain sockets
gitlab_rails['redis_socket'] = '/tmp/redis.sock' # defaults to /var/opt/gitlab/redis/redis.socket
# Password to Authenticate to alternate local Redis if required
gitlab_rails['redis_password'] = '<redis_password>'
```
1. Reconfigure GitLab for the changes to take effect:
```shell
sudo gitlab-ctl reconfigure
```
## Making the bundled Redis reachable via TCP
Use the following settings if you want to make the Redis instance
managed by the Linux package reachable via TCP:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
redis['port'] = 6379
redis['bind'] = '127.0.0.1'
redis['password'] = 'redis-password-goes-here'
```
1. Save the file and reconfigure GitLab for the changes to take effect:
```shell
sudo gitlab-ctl reconfigure
```
## Setting up a Redis-only server using the Linux package
If you'd like to set up Redis in a separate server than the GitLab application,
you can use the
[bundled Redis from a Linux package installation](https://docs.gitlab.com/ee/administration/redis/standalone.html).
## Running with multiple Redis instances
See <https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html#running-multiple-redis-clusters>.
## Redis Sentinel
See <https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html>.
## Using Redis in a failover setup
See <https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html>.
## Using Google Cloud Memorystore
Google Cloud Memorystore [does not support the Redis `CLIENT` command](https://cloud.google.com/memorystore/docs/redis/product-constraints#blocked_redis_commands).
By default, Sidekiq will attempt to set the `CLIENT` for debugging
purposes. This can be disabled via the following configuration setting:
```ruby
gitlab_rails['redis_enable_client'] = false
```
## Increasing the number of Redis connections beyond the default
By default Redis will only accept 10,000 client connections. If you need
more that 10,000 connections set the `maxclients` attribute to suit your needs.
Be advised that adjusting the `maxclients` attribute means that you will also need
to take into account your systems settings for `fs.file-max` (for example `sysctl -w fs.file-max=20000`)
```ruby
redis['maxclients'] = 20000
```
## Tuning the TCP stack for Redis
The following settings are to enable a more performant Redis server instance. `tcp_timeout` is
a value set in seconds that the Redis server waits before terminating an idle TCP connection.
The `tcp_keepalive` is a tunable setting in seconds to TCP ACKs to clients in absence of
communication.
```ruby
redis['tcp_timeout'] = "60"
redis['tcp_keepalive'] = "300"
```
## Announce IP from hostname
Currently the only way to enable hostnames in Redis is by setting `redis['announce_ip']`. However,
this would need to be set uniquely per Redis instance. `announce_ip_from_hostname` is a boolean that allows us to turn this on or off.
It fetches the hostname dynamically, inferring the hostname from `hostname -f` command.
```ruby
redis['announce_ip_from_hostname'] = true
```
## Setting the Redis Cache instance as an LRU
Using multiple Redis instances allows you to configure Redis as a
[Least Recently Used cache](https://redis.io/docs/latest/operate/rs/databases/memory-performance/eviction-policy/). Note you should only
do this for the Redis cache, rate-limiting, and repository cache instances; the Redis queues, shared
state instances, and tracechunks instances should never be configured as an LRU,
since they contain data (e.g. Sidekiq jobs) that is expected to be persistent.
To cap memory usage at 32GB, you can use:
```ruby
redis['maxmemory'] = "32gb"
redis['maxmemory_policy'] = "allkeys-lru"
redis['maxmemory_samples'] = 5
```
## Using Secure Sockets Layer (SSL)
You can configure Redis to run behind SSL.
### Running Redis server behind SSL
1. To run Redis server behind SSL, you can use the following settings in
`/etc/gitlab/gitlab.rb`. See the TLS/SSL section of
[`redis.conf.erb`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/redis/templates/default/redis.conf.erb)
to learn about the possible values:
```ruby
redis['tls_port']
redis['tls_cert_file']
redis['tls_key_file']
```
1. After specifying the required values, reconfigure GitLab for the changes to take
effect:
```shell
sudo gitlab-ctl reconfigure
```
NOTE:
Some `redis-cli` binaries are not built with support for directly connecting to a Redis server over TLS.
If your `redis-cli` doesn't support the `--tls` flag, you will have to use something like
[`stunnel`](https://redis.io/blog/stunnel-secure-redis-ssl/) to connect to the
Redis server using `redis-cli` for any debugging purposes.
### Make GitLab client connect to Redis server over SSL
To activate GitLab client support for SSL:
1. Add the following line to `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['redis_ssl'] = true
```
1. Reconfigure GitLab for the changes to take effect:
```shell
sudo gitlab-ctl reconfigure
```
## SSL certificates
If you're using custom SSL certificates for Redis, be sure to add them
to the [trusted certificates](../settings/ssl/index.md#install-custom-public-certificates).
## Renamed commands
By default, the `KEYS` command is disabled as a security measure.
If you'd like to obfuscate or disable this command, or other commands, edit the `redis['rename_commands']` setting in `/etc/gitlab/gitlab.rb` to look like:
```ruby
redis['rename_commands'] = {
'KEYS': '',
'OTHER_COMMAND': 'VALUE'
}
```
- `OTHER_COMMAND` is the command you want to modify
- `VALUE` should be one of:
1. A new command name.
1. `''`, which completely disables the command.
To disable this functionality:
1. Set `redis['rename_commands'] = {}` in your `/etc/gitlab/gitlab.rb` file
1. Run `sudo gitlab-ctl reconfigure`
## Lazy freeing
Redis 4 introduced [lazy freeing](http://antirez.com/news/93). This can improve performance when freeing large values.
This setting defaults to `false`. To enable it, you can use:
```ruby
redis['lazyfree_lazy_eviction'] = true
redis['lazyfree_lazy_expire'] = true
redis['lazyfree_lazy_server_del'] = true
redis['replica_lazy_flush'] = true
```
## Threaded I/O
Redis 6 introduced threaded I/O. This allow writes to scale across multiple cores.
This setting is disabled by default. To enable it, you can use:
```ruby
redis['io_threads'] = 4
redis['io_threads_do_reads'] = true
```
## Provide sensitive configuration to Redis clients without plain text storage
For more information, see the example in [configuration documentation](../settings/configuration.md#provide-redis-password-to-redis-server-and-client-components).
## Troubleshooting
### `x509: certificate signed by unknown authority`
This error message suggests that the SSL certificates have not been
properly added to the list of trusted certificates for the server. To
check whether this is an issue:
1. Check Workhorse logs in `/var/log/gitlab/gitlab-workhorse/current`.
1. If you see messages that look like:
```plaintext
2018-11-14_05:52:16.71123 time="2018-11-14T05:52:16Z" level=info msg="redis: dialing" address="redis-server:6379" scheme=rediss
2018-11-14_05:52:16.74397 time="2018-11-14T05:52:16Z" level=error msg="unknown error" error="keywatcher: x509: certificate signed by unknown authority"
```
The first line should show `rediss` as the scheme with the address
of the Redis server. The second line indicates the certificate is
not properly trusted on this server. See the [previous section](#ssl-certificates).
1. Verify that the SSL certificate is working via [these troubleshooting steps](../settings/ssl/ssl_troubleshooting.md#custom-certificates-missing-or-skipped).
### NOAUTH Authentication required
A Redis server may require a password sent via an `AUTH` message before
commands are accepted. A `NOAUTH Authentication required` error message
suggests the client is not sending a password. GitLab logs may help
troubleshoot this error:
1. Check Workhorse logs in `/var/log/gitlab/gitlab-workhorse/current`.
1. If you see messages that look like:
```plaintext
2018-11-14_06:18:43.81636 time="2018-11-14T06:18:43Z" level=info msg="redis: dialing" address="redis-server:6379" scheme=rediss
2018-11-14_06:18:43.86929 time="2018-11-14T06:18:43Z" level=error msg="unknown error" error="keywatcher: pubsub receive: NOAUTH Authentication required."
```
1. Check that the Redis client password specified in `/etc/gitlab/gitlab.rb` is correct:
```ruby
gitlab_rails['redis_password'] = 'your-password-here'
```
1. If you are using the Linux package-provided Redis server, check that the server has the same password:
```ruby
redis['password'] = 'your-password-here'
```
### Redis connection reset (ECONNRESET)
If you see `Redis::ConnectionError: Connection lost (ECONNRESET)` in the
GitLab Rails logs (`/var/log/gitlab-rails/production.log`), this might
indicate that the server is expecting SSL but the client is not
configured to use it.
1. Check that the server is actually listening to the port via SSL.
For example:
```shell
/opt/gitlab/embedded/bin/openssl s_client -connect redis-server:6379
```
1. Check `/var/opt/gitlab/gitlab-rails/etc/resque.yml`. You
should see something like:
```yaml
production:
url: rediss://:mypassword@redis-server:6379/
```
1. If `redis://` is present instead of `rediss://`, the `redis_ssl`
parameter may not have been configured properly, or the reconfigure
step may not have been run.
### Connecting to Redis via the CLI
When connecting to Redis for troubleshooting you can use:
- Redis via Unix domain sockets:
```shell
sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket
```
- Redis via TCP:
```shell
sudo /opt/gitlab/embedded/bin/redis-cli -h 127.0.0.1 -p 6379
```
- Password to authenticate to Redis if required:
```shell
sudo /opt/gitlab/embedded/bin/redis-cli -h 127.0.0.1 -p 6379 -a <password>
```
doc/settings/rpi.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Running on a Raspberry Pi
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
In order to run GitLab Community Edition on a Raspberry Pi, you need the newest
Pi 4 with at least 4GB of RAM for best results. You might be able to run GitLab
on lower resources, like a Pi 2 or newer, but it is not recommended. We do not
package for older Pis, as their CPU and RAM are insufficient.
The only supported architecture is `armhf`. For `arm64` support, see
[this epic](https://gitlab.com/groups/gitlab-org/-/epics/2370).
## Configure Swap
Even with a newer Pi, the first setting you will want to change is to ensure
the device has enough memory available by expanding the swap space to 4GB.
On Raspbian, swap can be configured in `/etc/dphys-swapfile`.
See [the manpage](https://manpages.ubuntu.com/manpages/lunar/en/man8/dphys-swapfile.8.html) for available settings.
## Install GitLab
The recommended and supported way to install GitLab is by using the GitLab
official repository.
Only the [official Raspberry Pi distribution](https://www.raspberrypi.com/software/) is
supported.
### Install GitLab via the official repository
Visit the [installation page](https://about.gitlab.com/install/), choose
Raspberry Pi OS, and follow the instructions to install GitLab.
### Manually download GitLab
If your distribution of choice is other than Raspbian, but Debian-based, you
can [manually download](https://docs.gitlab.com/ee/update/package/#upgrade-using-a-manually-downloaded-package)
GitLab and install it.
## Reduce running processes
If you find that your Pi is struggling to run GitLab, you can reduce
some running processes:
1. Open `/etc/gitlab/gitlab.rb` and change the following settings:
```ruby
# Reduce the number of running workers to the minimum in order to reduce memory usage
puma['worker_processes'] = 2
sidekiq['concurrency'] = 9
# Turn off monitoring to reduce idle cpu and disk usage
prometheus_monitoring['enable'] = false
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Additional recommendations
You can boost GitLab performance with a few settings.
### Use a proper harddrive
GitLab will perform best if you mount `/var/opt/gitlab` and the swapfile from a
hard drive rather than the SD card. You can attach an external hard drive to the
Pi using the USB interface.
### Use external services
You can improve the GitLab performance on the Pi by connecting GitLab to
external [database](database.md#using-a-non-packaged-postgresql-database-management-server)
and [Redis](https://docs.gitlab.com/ee/administration/redis/standalone.html) instances.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment