Commit edb06ab2 authored by 徐豪's avatar 徐豪
Browse files

init

parents

Too many changes to show.

To preserve performance only 532 of 532+ files are displayed.
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# SMTP settings
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
If you would rather send application email via an SMTP server instead of via
Sendmail or Postfix, add the following configuration information to
`/etc/gitlab/gitlab.rb` and run `gitlab-ctl reconfigure`.
WARNING:
Your `smtp_password` should not contain any String delimiters used in
Ruby or YAML (f.e. `'`) to avoid unexpected behavior during the processing of
config settings.
There are [example configurations](#example-configurations) at the end of this page.
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.server"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "smtp user"
gitlab_rails['smtp_password'] = "smtp password"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
# If your SMTP server does not like the default 'From: gitlab@localhost' you
# can change the 'From' with this setting.
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@example.com'
# If your SMTP server is using a self signed certificate or a certificate which
# is signed by a CA which is not trusted by default, you can specify a custom ca file.
# Please note that the certificates from /etc/gitlab/trusted-certs/ are
# not used for the verification of the SMTP server certificate.
gitlab_rails['smtp_ca_file'] = '/path/to/your/cacert.pem'
```
## SMTP connection pooling
You can enable SMTP connection pooling with the following setting:
```ruby
gitlab_rails['smtp_pool'] = true
```
This allows Sidekiq workers to reuse SMTP connections for multiple jobs. The maximum number of connections in the pool follows the [maximum concurrency configuration for Sidekiq](https://docs.gitlab.com/ee/administration/sidekiq/extra_sidekiq_processes.html#concurrency).
## Using encrypted credentials
Instead of storing the SMTP credentials in the configuration files as plain text, you can optionally
use an encrypted file for the SMTP credentials. To use this feature, you first need to enable
[GitLab encrypted configuration](https://docs.gitlab.com/ee/administration/encrypted_configuration.html).
The encrypted configuration for SMTP exists in an encrypted YAML file. By default the file will be created at
`/var/opt/gitlab/gitlab-rails/shared/encrypted_configuration/smtp.yaml.enc`. This location is configurable in the GitLab configuration.
The unencrypted contents of the file should be a subset of the settings from your `smtp_*'` settings in the `gitlab_rails`
configuration block.
The supported configuration items for the encrypted file are:
- `user_name`
- `password`
The encrypted contents can be configured with the [SMTP secret edit Rake command](https://docs.gitlab.com/ee/administration/raketasks/smtp.html).
**Configuration**
If initially your SMTP configuration looked like:
1. In `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.server"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "smtp user"
gitlab_rails['smtp_password'] = "smtp password"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
```
1. Edit the encrypted secret:
```shell
sudo gitlab-rake gitlab:smtp:secret:edit EDITOR=vim
```
1. The unencrypted contents of the SMTP secret should be entered like:
```yaml
user_name: 'smtp user'
password: 'smtp password'
```
1. Edit `/etc/gitlab/gitlab.rb` and remove the settings for `smtp_user_name` and `smtp_password`.
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Example configurations
### SMTP on localhost
This configuration, which simply enables SMTP and otherwise uses the default settings, can be used for an MTA running on localhost that does not provide a `sendmail` interface or that provides a `sendmail` interface that is incompatible with GitLab, such as Exim.
```ruby
gitlab_rails['smtp_enable'] = true
```
### SMTP without SSL
By default SSL is enabled for SMTP. If your SMTP server does not support communication over SSL use following settings:
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'localhost'
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_domain'] = 'localhost'
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_ssl'] = false
gitlab_rails['smtp_force_ssl'] = false
```
### Gmail
NOTE:
Gmail has [strict sending limits](https://support.google.com/a/answer/166852)
that can impair functionality as your organization grows. We strongly recommend using a
transactional service like [SendGrid](https://sendgrid.com/en-us) or [Mailgun](https://www.mailgun.com/)
for teams using SMTP configuration.
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.gmail.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "my.email@gmail.com"
gitlab_rails['smtp_password'] = "my-gmail-password"
gitlab_rails['smtp_domain'] = "smtp.gmail.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'peer' # Can be: 'none', 'peer', 'client_once', 'fail_if_no_peer_cert', see http://api.rubyonrails.org/classes/ActionMailer/Base.html
```
_Don't forget to change `my.email@gmail.com` to your email address and `my-gmail-password` to your own password._
_If you encounter authentication errors, ensure you have [allowed less secure apps to access the account](https://support.google.com/accounts/answer/6010255)
or try [turning on 2-step validation](https://support.google.com/accounts/answer/185839)
and using [an application password](https://support.google.com/mail/answer/185833)._
### Google SMTP relay
You can route outgoing non-Gmail messages through Google [using Google's SMTP relay service](https://support.google.com/a/answer/2956491?hl=en).
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp-relay.gmail.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_domain'] = "yourdomain.com"
gitlab_rails['gitlab_email_from'] = 'username@yourdomain.com'
gitlab_rails['gitlab_email_reply_to'] = 'username@yourdomain.com'
```
### Mailgun
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.mailgun.org"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_user_name'] = "postmaster@mg.gitlab.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "mg.gitlab.com"
```
### Amazon Simple Email Service (AWS SES)
- Using STARTTLS
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "email-smtp.region-1.amazonaws.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "IAMmailerKey"
gitlab_rails['smtp_password'] = "IAMmailerSecret"
gitlab_rails['smtp_domain'] = "yourdomain.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
```
Make sure to permit egress through port 587 in your ACL and security group.
- Using TLS Wrapper
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "email-smtp.region-1.amazonaws.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "IAMmailerKey"
gitlab_rails['smtp_password'] = "IAMmailerSecret"
gitlab_rails['smtp_domain'] = "yourdomain.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_ssl'] = true
gitlab_rails['smtp_force_ssl'] = true
```
Make sure to permit egress through port 465 in your ACL and security group.
### Mandrill
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.mandrillapp.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "MandrillUsername"
gitlab_rails['smtp_password'] = "MandrillApiKey" # https://mandrillapp.com/settings
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
```
### SMTP.com
You can use the [SMTP.com](https://www.smtp.com/) email service. [Retrieve your sender login and password](https://knowledge.smtp.com/s/article/My-Account)
from your account.
To improve delivery by authorizing `SMTP.com` to send emails on behalf of your domain, you should:
- Specify `from` and `reply_to` addresses using your GitLab domain name.
- [Set up SPF and DKIM for the domain](https://knowledge.smtp.com/s/article/Email-authentication-SPF-DKIM-DMARC).
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'send.smtp.com'
gitlab_rails['smtp_port'] = 25 # If your outgoing port 25 is blocked, try 2525, 2082
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_authentication'] = 'plain'
gitlab_rails['smtp_user_name'] = 'your_sender_login'
gitlab_rails['smtp_password'] = 'your_sender_password'
gitlab_rails['smtp_domain'] = 'your.gitlab.domain.com'
gitlab_rails['gitlab_email_from'] = 'user@your.gitlab.domain.com'
gitlab_rails['gitlab_email_reply_to'] = 'user@your.gitlab.domain.com'
```
Check the [SMTP.com Knowledge Base](https://knowledge.smtp.com/s/) for further assistance.
### SparkPost
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.sparkpostmail.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "SMTP_Injection"
gitlab_rails['smtp_password'] = "SparkPost_API_KEY" # https://app.sparkpost.com/account/credentials
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
```
### Gandi
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.gandi.net"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_user_name'] = "your.email@domain.com"
gitlab_rails['smtp_password'] = "your.password"
gitlab_rails['smtp_domain'] = "domain.com"
```
### Zoho Mail
This configuration was tested on Zoho Mail with a custom domain.
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.zoho.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_user_name'] = "gitlab@mydomain.com"
gitlab_rails['smtp_password'] = "mypassword"
gitlab_rails['smtp_domain'] = "smtp.zoho.com"
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@example.com'
```
### SiteAge, LLC Zimbra Mail
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'mail.siteage.net'
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = 'gitlab@domain.com'
gitlab_rails['smtp_password'] = 'password'
gitlab_rails['smtp_authentication'] = 'login'
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['gitlab_email_from'] = "gitlab@domain.com"
gitlab_rails['smtp_tls'] = true
```
### OVH
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "ssl0.ovh.net"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@domain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "ssl0.ovh.net"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
```
### Outlook
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp-mail.outlook.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username@outlook.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "smtp-mail.outlook.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
```
### Office365
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.office365.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username@yourdomain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "yourdomain.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
gitlab_rails['gitlab_email_from'] = 'username@yourdomain.com'
```
### Office365 relay
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "your mx endpoint"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_domain'] = "yourdomain.com"
gitlab_rails['gitlab_email_from'] = 'username@yourdomain.com'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@yourdomain.com'
```
### Online.net
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtpauth.online.net"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@domain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "online.net"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
```
### Amen.fr / Securemail.pro
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp-fr.securemail.pro"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@domain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_tls'] = true
```
### 1&1
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.1and1.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "my.email@domain.com"
gitlab_rails['smtp_password'] = "1and1-email-password"
gitlab_rails['smtp_domain'] = "domain.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
```
### yahoo
```ruby
gitlab_rails['gitlab_email_from'] = 'user@yahoo.com'
gitlab_rails['gitlab_email_reply_to'] = 'user@yahoo.com'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.mail.yahoo.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "user@yahoo.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
```
### QQ exmail
QQ exmail (腾讯企业邮箱)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.exmail.qq.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "xxxx@xx.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['gitlab_email_from'] = 'xxxx@xx.com'
gitlab_rails['smtp_domain'] = "exmail.qq.com"
```
### NetEase Free Enterprise Email
NetEase Free Enterprise Email (网易免费企业邮)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.ym.163.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "xxxx@xx.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['gitlab_email_from'] = 'xxxx@xx.com'
gitlab_rails['smtp_domain'] = "smtp.ym.163.com"
```
### SendGrid with username/password authentication
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.sendgrid.net"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "a_sendgrid_crendential"
gitlab_rails['smtp_password'] = "a_sendgrid_password"
gitlab_rails['smtp_domain'] = "smtp.sendgrid.net"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
```
### SendGrid with API Key authentication
If you don't want to supply a username/password, you can use an [API key](https://www.twilio.com/docs/sendgrid/for-developers/sending-email/getting-started-smtp):
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.sendgrid.net"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "apikey"
gitlab_rails['smtp_password'] = "the_api_key_you_created"
gitlab_rails['smtp_domain'] = "smtp.sendgrid.net"
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
# If use Single Sender Verification You must configure from. If not fail
# 550 The from address does not match a verified Sender Identity. Mail cannot be sent until this error is resolved.
# Visit https://sendgrid.com/docs/for-developers/sending-email/sender-identity/ to see the Sender Identity requirements
gitlab_rails['gitlab_email_from'] = 'email@sender_owner_api'
gitlab_rails['gitlab_email_reply_to'] = 'email@sender_owner_reply_api'
```
Note that `smtp_user_name` must literally be set to `"apikey"`.
The API Key you created must be entered in `smtp_password`.
### Brevo
This configuration was tested with the Brevo [SMTP relay service](https://www.brevo.com/free-smtp-server/). To grab the
relevant account credentials via the URLs commented into this example, [log in to your Brevo account](https://login.brevo.com).
For further details, refer to the Brevo [help page](https://help.brevo.com/hc/en-us/articles/209462765-What-is-Brevo-SMTP).
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp-relay.sendinblue.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "<username@example.com>" # https://app.brevo.com/settings/keys/smtp
gitlab_rails['smtp_password'] = "<password>" # https://app.brevo.com/settings/keys/smtp
gitlab_rails['smtp_domain'] = "<example.com>"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['gitlab_email_from'] = '<gitlab@example.com>'
gitlab_rails['gitlab_email_reply_to'] = '<noreply@example.com>'
```
### SMTP2GO
This configuration was tested using [SMTP2GO](https://www.smtp2go.com/). To get the relevant account credentials using the URLs commented in this example, [sign in to your SMTP2GO account](https://app.smtp2go.com/login/).
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.smtp2go.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "<username>" # https://app.smtp2go.com/settings/users
gitlab_rails['smtp_password'] = "<password>" # https://app.smtp2go.com/settings/users
gitlab_rails['smtp_domain'] = "<example.com>" # https://app.smtp2go.com/settings/sender_domains
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
```
### Yandex
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.yandex.ru"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "login"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "yourdomain_or_yandex.ru"
gitlab_rails['gitlab_email_from'] = 'login_or_login@yandex.ru'
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
```
### UD Media
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.udXX.udmedia.de" # Replace XX, see smtp server information: https://www.udmedia.de/login/mail/
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "login"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
```
### Microsoft Exchange (no authentication)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "example.com"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_enable_starttls_auto'] = true
```
### Microsoft Exchange (with authentication)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.example.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username@domain.com"
gitlab_rails['smtp_password'] = 'password'
gitlab_rails['smtp_domain'] = "mail.example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
```
### Strato.de
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.strato.de"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@stratodomain.de"
gitlab_rails['smtp_password'] = "strato_email_password"
gitlab_rails['smtp_domain'] = "stratodomain.de"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
```
### Rackspace
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "secure.emailsrvr.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@domain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "domain.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
gitlab_rails['gitlab_email_from'] = 'username@domain.com'
gitlab_rails['gitlab_email_reply_to'] = 'username@domain.com'
```
### DomainFactory (df.eu)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "sslout.df.eu"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@domain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "domain.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
```
### Infomaniak (infomaniak.com)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.infomaniak.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "mail.infomaniak.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
```
### GoDaddy (TLS)
- European servers: smtpout.europe.secureserver.net
- Asian servers: smtpout.asia.secureserver.net
- Global (US) servers: smtpout.secureserver.net
```ruby
gitlab_rails['gitlab_email_from'] = 'username@domain.com'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtpout.secureserver.net"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@domain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "domain.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
```
### GoDaddy (No TLS)
See GoDaddy (TLS) entry above for mail server list.
```ruby
gitlab_rails['gitlab_email_from'] = 'username@domain.com'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtpout.secureserver.net"
gitlab_rails['smtp_port'] = 80
gitlab_rails['smtp_user_name'] = "username@domain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "domain.com"
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = false
```
### OpenSRS (hostedemail.com)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.hostedemail.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@domain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "domain.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
gitlab_rails['gitlab_email_from'] = 'username@domain.com'
gitlab_rails['gitlab_email_reply_to'] = 'username@domain.com'
```
### Aruba (aruba.it)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtps.aruba.it"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "user@yourdomain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "yourdomain.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_ssl'] = true
```
### Alibaba Cloud Direct Mail (No TLS)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtpdm-ap-southeast-1.aliyun.com" # Set to the Direct Mail service region in use, refer to: https://www.alibabacloud.com/help/en/directmail/latest/smtp-service-address
gitlab_rails['smtp_port'] = 80
gitlab_rails['smtp_user_name'] = "<username@example.com>" # Direct Mail sender address
gitlab_rails['smtp_password'] = "<password>" # Set Direct Mail password
gitlab_rails['smtp_domain'] = "<example.com>" # Email domain configured in Direct Mail
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
gitlab_rails['gitlab_email_from'] = "<username@example.com>" # Email domain configured in Direct Mail
```
### Alibaba Cloud Direct Mail (TLS)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtpdm-ap-southeast-1.aliyun.com" # Set to the Direct Mail service region in use, refer to: https://www.alibabacloud.com/help/en/directmail/latest/smtp-service-address
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "<username@example.com>" # Direct Mail sender address
gitlab_rails['smtp_password'] = "<password>" # Set Direct Mail password
gitlab_rails['smtp_domain'] = "<example.com>" # Email domain configured in Direct Mail
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['gitlab_email_from'] = "<username@example.com>" # Email domain configured in Direct Mail
```
### Aliyun Direct Mail
Aliyun Direct Mail (阿里云邮件推送)
```ruby
gitlab_rails['gitlab_email_from'] = 'username@your domain'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtpdm.aliyun.com"
gitlab_rails['smtp_port'] = 80
gitlab_rails['smtp_user_name'] = "username@your domain"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "your domain"
gitlab_rails['smtp_authentication'] = "login"
```
### Aliyun Enterprise Mail with TLS
Aliyun Enterprise Mail with TLS (阿里企业邮箱)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.qiye.aliyun.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@your domain"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "your domain"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
```
### FastMail
FastMail requires an
[App Password](https://www.fastmail.help/hc/en-us/articles/360058752854-App-passwords)
even when two-step verification is not enabled.
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.fastmail.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "account@fastmail.com"
gitlab_rails['smtp_password'] = "app-specific-password"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
```
### Dinahosting
```ruby
gitlab_rails['gitlab_email_from'] = 'username@example.com'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "example-com.correoseguro.dinaserver.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username-example-com"
gitlab_rails['smtp_password'] = "mypassword"
gitlab_rails['smtp_domain'] = "example-com.correoseguro.dinaserver.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
```
### GMX Mail
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.gmx.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "my-gitlab@gmx.com"
gitlab_rails['smtp_password'] = "Pa5svv()rD"
gitlab_rails['smtp_domain'] = "mail.gmx.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
### Email Settings
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'my-gitlab@gmx.com'
gitlab_rails['gitlab_email_display_name'] = 'My GitLab'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@gmx.com'
```
### Hetzner
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.your-server.de"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "user@example.com"
gitlab_rails['smtp_password'] = "mypassword"
gitlab_rails['smtp_domain'] = "mail.your-server.de"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['gitlab_email_from'] = "example@example.com"
```
### Snel.com
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtprelay.snel.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = false
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['gitlab_email_from'] = "example@example.com"
gitlab_rails['gitlab_email_reply_to'] = "example@example.com"
```
### JangoSMTP
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "express-relay.jangosmtp.net"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_user_name'] = "your.username"
gitlab_rails['smtp_password'] = "your.password"
gitlab_rails['smtp_domain'] = "domain.com"
gitlab_rails['gitlab_email_from'] = 'gitlab@domain.com'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@domain.com'
```
### Mailjet
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "in-v3.mailjet.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "mailjet-api-key"
gitlab_rails['smtp_password'] = "mailjet-secret-key"
gitlab_rails['smtp_domain'] = "in-v3.mailjet.com"
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['gitlab_email_from'] = 'gitlab@domain.com'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@domain.com'
```
### Mailcow
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.example.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "yourmail@example.com"
gitlab_rails['smtp_password'] = "yourpassword"
gitlab_rails['smtp_domain'] = "smtp.example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
```
### ALL-INKL.COM
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "<userserver>.kasserver.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "<username>"
gitlab_rails['smtp_password'] = "<password>"
gitlab_rails['smtp_domain'] = "<your.domain>"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_tls'] = true
```
### webgo.de
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "sXX.goserver.host" # or serverXX.webgo24.de
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "webXXXpX"
gitlab_rails['smtp_password'] = "Your Password"
gitlab_rails['smtp_domain'] = "sXX.goserver.host" # or serverXX.webgo24.de
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['gitlab_email_from'] = 'Your Mail Address'
gitlab_rails['gitlab_email_reply_to'] = 'Your Mail Address'
```
### mxhichina.com
```ruby
gitlab_rails['gitlab_email_from'] = "username@company.com"
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.mxhichina.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@company.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "mxhichina.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
```
### Postmark
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.postmarkapp.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "your_api_token"
gitlab_rails['smtp_password'] = "your_api_token"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@example.com'
```
### easyDNS (outbound mail)
Check if it's available/enabled and configuration settings in the [control panel](https://cp.easydns.com/manage/domains/mail/outbound/).
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mailout.easydns.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_force_ssl'] = true
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_user_name'] = "example.com"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_password'] = "password_you_set"
gitlab_rails['gitlab_email_from'] = 'no-reply@git.example.com'
```
### Campaign Monitor
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.api.createsend.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "your_api_token" # Menu > Transactional > Send with SMTP > SMTP tokens > Token
gitlab_rails['smtp_password'] = "your_api_token" # Same as gitlab_rails['smtp_user_name'] value
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@example.com'
```
### Freehostia
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mbox.freehostia.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@example.com"
gitlab_rails['smtp_password'] = "password_you_set"
gitlab_rails['smtp_domain'] = "mbox.freehostia.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
```
### Mailbox.org
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.mailbox.org"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "username@example.com"
gitlab_rails['smtp_password'] = "password_you_set"
gitlab_rails['smtp_domain'] = "smtp.mailbox.org"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
```
### Mittwald CM Service (mittwald.de)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.agenturserver.de"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username@example.com"
gitlab_rails['smtp_password'] = "password_you_set"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['gitlab_email_from'] = "username@example.com"
gitlab_rails['gitlab_email_reply_to'] = "username@example.com"
```
### Unitymedia (.de)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "submit.unitybox.de"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username@unitybox.de"
gitlab_rails['smtp_password'] = "yourPassword"
gitlab_rails['smtp_domain'] = "submit.unitybox.de"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@example.com'
```
### united-domains AG (united-domains.de)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.udag.de"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "example-com-0001"
gitlab_rails['smtp_password'] = "smtppassword"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
gitlab_rails['gitlab_email_display_name'] = 'GitLab - my company'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@example.com'
```
### IONOS by 1&1 (ionos.de)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.ionos.de"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "your-user@your-domain.de"
gitlab_rails['smtp_password'] = "Y0uR_Pass_H3r3"
gitlab_rails['smtp_domain'] = "your-domain.de"
gitlab_rails['smtp_authentication'] = 'login'
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
gitlab_rails['gitlab_email_from'] = 'your-user@your-domain.de'
```
### AWS Workmail
From the [AWS workmail documentation](https://docs.aws.amazon.com/workmail/latest/userguide/using_IMAP.html):
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.server"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "smtp user"
gitlab_rails['smtp_password'] = "smtp password"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
```
### Open Telekom Cloud
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "login-cloud.mms.t-systems-service.com"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_domain'] = "yourdomain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_user_name'] = "username"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['gitlab_email_from'] = 'gitlab@yourdomain'
```
### Uberspace 6
From the [Uberspace Wiki](https://manual.uberspace.de/):
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "<your-host>.uberspace.de"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "<your-user>@<your-domain>"
gitlab_rails['smtp_password'] = "<your-password>"
gitlab_rails['smtp_domain'] = "<your-domain>"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
```
### Tipimail
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'smtp.tipimail.com'
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = 'username'
gitlab_rails['smtp_password'] = 'password'
gitlab_rails['smtp_authentication'] = 'login'
gitlab_rails['smtp_enable_starttls_auto'] = true
```
### Netcup
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = '<your-host>.netcup.net'
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "<your-gitlab-domain>"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
# Netcup is picky about the usage of GitLab's TLD instead of the subdomain (if you use one).
# If this is not set up correctly, the scheduled emails will fail. For example, if
# GitLab's domain name is 'gitlab.example.com', the following setting should be set to
# 'gitlab@example.com'.
gitlab_rails['gitlab_email_from'] = "gitlab@<your-top-level-domain>"
```
### Mail-in-a-Box
```ruby
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'box.example.com'
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username@example.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "<your-gitlab-domain>"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
```
### NIFCLOUD ESS
[SMTP Interface](https://docs.nifcloud.com/ess/spec/smtp.htm).
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.ess.nifcloud.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "SMTP user name"
gitlab_rails['smtp_password'] = "SMTP user password"
gitlab_rails['smtp_domain'] = "smtp.ess.nifcloud.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
gitlab_rails['gitlab_email_from'] = 'username@example.com'
gitlab_rails['gitlab_email_reply_to'] = 'username@example.com'
```
Check the SMTP user name and SMTP user password from the ESS [dashboard](https://docs.nifcloud.com/ess/help/dashboard.htm).
`gitlab_email_from` and `gitlab_email_reply_to` must be ESS authenticated sender email addresses.
### Sina mail
User needs first to enabled SMTP through the mailbox settings via web mail interface and get the authenitication code. Check out more details in the [help page](http://help.sina.com.cn/comquestiondetail/view/1566/) of Sina mail.
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.sina.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username@sina.com"
gitlab_rails['smtp_password'] = "authentication code"
gitlab_rails['smtp_domain'] = "smtp.sina.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['gitlab_email_from'] = 'username@sina.com'
```
### Feishu mail
Check out more details in the [help page](https://www.feishu.cn/hc/en-US/articles/360049068017-admin-allow-members-to-access-feishu-mail-using-third-party-email-clients) of Feishu mail.
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.feishu.cn"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "your-user@your-domain"
gitlab_rails['gitlab_email_from'] = "username@yourdomain.com"
gitlab_rails['smtp_domain'] = "yourdomain.com"
gitlab_rails['smtp_password'] = "authentication code"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = true
```
### Hostpoint
For more information about Hostpoint email visit their [help page](https://support.hostpoint.ch/en/technical/e-mail/frequently-asked-questions/e-mail-settings-at-a-glance#hp-section3)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "asmtp.mail.hostpoint.ch"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username@example.com"
gitlab_rails['smtp_password'] = "authentication code"
gitlab_rails['smtp_domain'] = "asmtp.mail.hostpoint.ch"
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['gitlab_email_from'] = 'username@example.com'
```
### Fastweb (fastweb.it)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.fastwebnet.it"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "your_fastweb_fastmail_username@fastwebnet.it"
gitlab_rails['smtp_password'] = "your_fastweb_fastmail_password"
gitlab_rails['smtp_domain'] = "smtp.fastwebnet.it"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
```
### Scaleway Transactional Email
Read more about [Scaleway's Transactional Email](https://www.scaleway.com/en/docs/managed-services/transactional-email/how-to/generate-api-keys-for-tem-with-iam/).
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.tem.scw.cloud"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "transactional_email_user_name"
gitlab_rails['smtp_password'] = "secret_key_of_api_key"
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
```
### Proton Mail
Proton documentation: [How to set up SMTP to use business applications or devices with Proton Mail](https://proton.me/support/smtp-submission)
```ruby
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.protonmail.ch"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_user_name'] = "<the Proton email address for which you generated the SMTP token>"
gitlab_rails['smtp_password'] = "<the generated SMTP token>"
gitlab_rails['smtp_domain'] = "<your domain>"
gitlab_rails['gitlab_email_from'] = "<the Proton email address for which you generated the SMTP token>"
gitlab_rails['gitlab_email_reply_to'] = "<the Proton email address for which you generated the SMTP token>"
```
### More examples are welcome
If you have figured out an example configuration yourself please send a Merge
Request to save other people time.
## Testing the SMTP configuration
You can verify that GitLab can send emails properly using the Rails console.
On the GitLab server, execute `gitlab-rails console` to enter the console. Then,
you can enter the following command at the console prompt to cause GitLab to
send a test email:
```ruby
Notify.test_email('destination_email@address.com', 'Message Subject', 'Message Body').deliver_now
```
## Troubleshooting
### Outgoing connections to port 25 is blocked on major cloud providers
If you are using a cloud provider to host your GitLab instance and you are using port 25 for your
SMTP server, it is possible that your cloud provider is blocking outgoing connections to port 25.
This prevents GitLab from sending any outgoing mail. You can follow the instructions below to work
around this depending on your cloud provider:
- AWS: [How do I remove the restriction on port 25 from my Amazon EC2 instance or AWS Lambda function?](https://repost.aws/knowledge-center/ec2-port-25-throttle)
- Azure: [Troubleshoot outbound SMTP connectivity problems in Azure](https://learn.microsoft.com/en-us/azure/virtual-network/troubleshoot-outbound-smtp-connectivity)
- GCP: [Sending email from an instance](https://cloud.google.com/compute/docs/tutorials/sending-mail)
### Wrong version number when using SSL/TLS
Many users run into the following error after configuring SMTP:
```plaintext
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: wrong version number)
```
This error is usually due to incorrect settings:
- If your SMTP provider is using port 25 or 587, SMTP connections start
**unencrypted** but can be upgraded via
[STARTTLS](https://en.wikipedia.org/wiki/Opportunistic_TLS). Be sure the
following settings are set:
```ruby
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false # This is the default and can be omitted
gitlab_rails['smtp_ssl'] = false # This is the default and can be omitted
```
- If your SMTP provider is using port 465, SMTP connections start
**encrypted** over TLS. Ensure the following line is present:
```ruby
gitlab_rails['smtp_tls'] = true
```
For more details, read [about the confusion over SMTP ports, TLS, and STARTTLS](https://www.fastmail.help/hc/en-us/articles/360058753834-SSL-TLS-and-STARTTLS).
### Emails not sending when using external Sidekiq
If your instance has [an external Sidekiq](https://docs.gitlab.com/ee/administration/sidekiq/index.html)
configured, the SMTP configuration must be present in `/etc/gitlab/gitlab.rb` on the external Sidekiq server. If
the SMTP configuration is missing, you may notice that emails do not get sent through SMTP as many
GitLab emails are sent via Sidekiq.
### Emails not sending when using Sidekiq routing rules
If you are using Sidekiq [routing rules](https://docs.gitlab.com/ee/administration/sidekiq/processing_specific_job_classes.html#routing-rules), your configuration might be missing the `mailers` queue which is required for outgoing mail.
For more details, review the [example configuration](https://docs.gitlab.com/ee/administration/sidekiq/processing_specific_job_classes.html#detailed-example).
### Email not sent
WARNING:
Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case.
If you have correctly configured an email server, but email is not sent:
1. Run a [Rails console](https://docs.gitlab.com/ee/administration/operations/rails_console.html#starting-a-rails-console-session).
1. Check the `ActionMailer` `delivery_method`. It must match the type of server you're using, either `:smtp` for an SMTP server or `:sendmail`
for Sendmail:
intended. If you configured SMTP, it should say `:smtp`. If you're using
Sendmail, it should say `:sendmail`:
```ruby
irb(main):001:0> ActionMailer::Base.delivery_method
=> :smtp
```
1. If you're using SMTP, check the mail settings:
```ruby
irb(main):002:0> ActionMailer::Base.smtp_settings
=> {:address=>"localhost", :port=>25, :domain=>"localhost.localdomain", :user_name=>nil, :password=>nil, :authentication=>nil, :enable_starttls_auto=>true}
```
In the example above, the SMTP server is configured for the local machine. If this is intended, check your local mail
logs (for example, `/var/log/mail.log`) for more details.
1. Send a test message using the console:
```ruby
irb(main):003:0> Notify.test_email('youremail@email.com', 'Hello World', 'This is a test message').deliver_now
```
If you do not receive an email or see an error message, check your mail server settings.
### Email not sent when using STARTTLS and SMTP TLS
You may encounter the following error if both STARTTLS and SMTP TLS are enabled:
```plaintext
:enable_starttls and :tls are mutually exclusive. Set :tls if you're on an SMTPS connection. Set :enable_starttls if you're on an SMTP connection and using STARTTLS for secure TLS upgrade.
```
This error occurs when both `gitlab_rails['smtp_enable_starttls_auto']` and `gitlab_rails['smtp_tls']` are set to `true`. If using SMTPS, set `gitlab_rails['smtp_enable_starttls_auto']` to `false`. If using SMTP with STARTTLS, set `gitlab_rails['smtp_tls']` to `false`. Run `sudo gitlab-ctl reconfigure` for the change to take effect.
## Disable all outgoing email
NOTE:
This will disable **all** outgoing email from your GitLab instance, including but not limited to notification emails, direct mentions, and password reset emails.
In order to disable **all** outgoing email, you can edit or add the following line to `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['gitlab_email_enabled'] = false
```
Run `sudo gitlab-ctl reconfigure` for the change to take effect.
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Configure SSL for a Linux package installation
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
The Linux package supports several common use cases for SSL configuration.
By default, HTTPS is not enabled. To enable HTTPS, you can:
- Use Let's Encrypt for free, automated HTTPS.
- Manually configure HTTPS with your own certificates.
NOTE:
If you use a proxy, load balancer or some other external device to terminate SSL for the GitLab host name,
see [External, proxy, and load balancer SSL termination](#configure-a-reverse-proxy-or-load-balancer-ssl-termination).
The following table shows which method each GitLab service supports.
| Service | Manual SSL | Let's Encrypt integration |
|-|-|-|
| GitLab instance domain | [Yes](#configure-https-manually) | [Yes](#enable-the-lets-encrypt-integration) |
| Container Registry | [Yes](https://docs.gitlab.com/ee/administration/packages/container_registry.html#configure-container-registry-under-its-own-domain) | [Yes](#enable-the-lets-encrypt-integration) |
| Mattermost | [Yes](https://docs.gitlab.com/ee/integration/mattermost/index.html#running-gitlab-mattermost-with-https) | [Yes](#enable-the-lets-encrypt-integration) |
| GitLab Pages | [Yes](https://docs.gitlab.com/ee/administration/pages/#wildcard-domains-with-tls-support) | No |
## Enable the Let's Encrypt integration
[Let's Encrypt](https://letsencrypt.org) is enabled by default if `external_url`
is set with the HTTPS protocol and no other certificates are configured.
Prerequisites:
- Ports `80` and `443` must be accessible to the public Let's Encrypt servers
that run the validation checks. The validation
[does not work with non-standard ports](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3580).
If the environment is private or air-gapped, certbot (the tool used by Let's Encrypt) provides a
[manual method](https://eff-certbot.readthedocs.io/en/stable/using.html#manual)
to install a Let's Encrypt certificate.
To enable Let's Encrypt:
1. Edit `/etc/gitlab/gitlab.rb` and add or change the following entries:
```ruby
## GitLab instance
external_url "https://gitlab.example.com" # Must use https protocol
letsencrypt['contact_emails'] = ['foo@email.com'] # Optional
## Container Registry (optional), must use https protocol
registry_external_url "https://registry.example.com"
#registry_nginx['ssl_certificate'] = "path/to/cert" # Must be absent or commented out
## Mattermost (optional), must use https protocol
mattermost_external_url "https://mattermost.example.com"
```
- Certificates expire every 90 days. Email addresses you specify for `contact_emails` receive an
alert when the expiration date approaches.
- The GitLab instance is the
primary domain name on the certificate. Additional services
such as the Container Registry are added as alternate domain names to the same
certificate. In the example above, the primary domain is `gitlab.example.com` and
the Container Registry domain is `registry.example.com`. You don't need
to set up wildcard certificates.
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
If Let's Encrypt fails to issue a certificate, see the
[troubleshooting section](ssl_troubleshooting.md#lets-encrypt-fails-on-reconfigure)
for potential solutions.
### Renew the certificates automatically
Default installations schedule renewals after midnight on every 4th day of the month.
The minute is determined by the value in `external_url` to help distribute the load
on the upstream Let's Encrypt servers.
To explicitly set the renewal times:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
# Renew every 7th day of the month at 12:30
letsencrypt['auto_renew_hour'] = "12"
letsencrypt['auto_renew_minute'] = "30"
letsencrypt['auto_renew_day_of_month'] = "*/7"
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
NOTE:
The certificate is renewed only if it expires in 30 days.
For example, if you set it to renew on the 1st of every month at 00:00 and the
certificate expires on the 31st, then the certificate will expire before it's renewed.
Automatic renewals are managed with [go-crond](https://github.com/webdevops/go-crond).
If wanted, one can pass [CLI arguments](https://github.com/webdevops/go-crond#usage) to
go-crond by editing the `/etc/gitlab/gitlab.rb`:
```ruby
crond['flags'] = {
'log.json' = true,
'server.bind' = ':8040'
}
```
To disable the automatic renewal:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
letsencrypt['auto_renew'] = false
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
### Renew the certificates manually
Renew the Let's Encrypt certificates manually using either one of the following commands:
```shell
sudo gitlab-ctl reconfigure
```
```shell
sudo gitlab-ctl renew-le-certs
```
The previous commands only generate a renewal if the certificate is close to expiration.
If encountering an error during renewal, [consider the upstream rate limits](https://letsencrypt.org/docs/rate-limits/).
### Use an ACME server other than Let's Encrypt
You can use an ACME server other than Let's Encrypt, and configure GitLab to
use that to fetch a certificate. Some services that provide their own ACME
server are:
- [ZeroSSL](https://zerossl.com/documentation/acme/)
- [Buypass](https://www.buypass.com/products/tls-ssl-certificates/go-ssl)
- [SSL.com](https://www.ssl.com/guide/ssl-tls-certificate-issuance-and-revocation-with-acme/)
- [`step-ca`](https://smallstep.com/docs/step-ca/index.html)
To configure GitLab to use a custom ACME server:
1. Edit `/etc/gitlab/gitlab.rb` and set the ACME endpoints:
```ruby
external_url 'https://example.com'
letsencrypt['acme_staging_endpoint'] = 'https://ca.internal/acme/acme/directory'
letsencrypt['acme_production_endpoint'] = 'https://ca.internal/acme/acme/directory'
```
If the custom ACME server provides it, use a staging endpoint as well.
Checking the staging endpoint first ensures that the ACME configuration is correct
before submitting the request to ACME production. Do this to avoid ACME
rate-limits while working on your configuration.
The default values are:
```plaintext
https://acme-staging-v02.api.letsencrypt.org/directory
https://acme-v02.api.letsencrypt.org/directory
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
### Add alternative domains to the certificate
You can add alternative domains (or subject alternative names) to the Let's Encrypt certificate.
This can be helpful if you would like to use the [bundled NGINX](../nginx.md) as a
[reverse proxy for other backend applications](../nginx.md#inserting-custom-settings-into-the-nginx-configuration).
The DNS records for the alternative domains must point to the GitLab instance.
To add alternative domains to your Let's Encrypt certificate:
1. Edit `/etc/gitlab/gitlab.rb` and add the alternative domains:
```ruby
# Separate multiple domains with commas
letsencrypt['alt_names'] = ['another-application.example.com']
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
The resulting Let's Encrypt certificates generated for the main GitLab application will
include the alternative domains specified. The generated files are located at:
- `/etc/gitlab/ssl/gitlab.example.com.key` for the key.
- `/etc/gitlab/ssl/gitlab.example.com.crt` for the certificate.
## Configure HTTPS manually
WARNING:
The NGINX configuration tells browsers and clients to only communicate with your
GitLab instance over a secure connection for the next 365 days using
[HSTS](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security).
See [Configure the HTTP Strict Transport Security](#configure-the-http-strict-transport-security-hsts)
for more configuration options. If enabling HTTPS, you must provide a
secure connection to your instance for at least the next 24 months.
To enable HTTPS:
1. Edit `/etc/gitlab/gitlab.rb`:
1. Set the `external_url` to your domain. Note the `https` in the URL:
```ruby
external_url "https://gitlab.example.com"
```
1. Disable the Let's Encrypt integration:
```ruby
letsencrypt['enable'] = false
```
GitLab attempts to renew any Let's Encrypt certificate with every reconfigure.
If you plan to use your own manually created certificate you must disable
the Let's Encrypt integration, otherwise the certificate could be overwritten
due to the automatic renewal.
1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there:
```shell
sudo mkdir -p /etc/gitlab/ssl
sudo chmod 755 /etc/gitlab/ssl
sudo cp gitlab.example.com.key gitlab.example.com.crt /etc/gitlab/ssl/
```
In the example, the hostname is `gitlab.example.com`, so the Linux package installation
looks for private key and public certificate files called
`/etc/gitlab/ssl/gitlab.example.com.key` and `/etc/gitlab/ssl/gitlab.example.com.crt`,
respectively. If you want, you can
[use a different location and certificates names](#change-the-default-ssl-certificate-location).
You must use the full certificate chain, in the correct order, to prevent
SSL errors when clients connect: first the server certificate,
then all intermediate certificates, and finally the root CA.
1. Optional. If the `certificate.key` file is password protected, NGINX doesn't ask for
the password when you reconfigure GitLab. In that case, the Linux package installation
fails silently with no error messages.
To specify the password for the key file, store the password in a text file
(for example, `/etc/gitlab/ssl/key_file_password.txt`) and add the following
to `/etc/gitlab/gitlab.rb`:
```ruby
nginx['ssl_password_file'] = '/etc/gitlab/ssl/key_file_password.txt'
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
1. Optional. If you are using a firewall, you may have to open port 443 to allow inbound
HTTPS traffic:
```shell
# UFW example (Debian, Ubuntu)
sudo ufw allow https
# lokkit example (RedHat, CentOS 6)
sudo lokkit -s https
# firewall-cmd (RedHat, Centos 7)
sudo firewall-cmd --permanent --add-service=https
sudo systemctl reload firewalld
```
If you are updating existing certificates, follow a
[different process](#update-the-ssl-certificates).
### Redirect `HTTP` requests to `HTTPS`
By default, when you specify an `external_url` starting with `https`, NGINX
no longer listens for unencrypted HTTP traffic on port 80. To redirect all HTTP
traffic to HTTPS:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['redirect_http_to_https'] = true
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
NOTE:
This behavior is enabled by default when using the [Let's Encrypt integration](#enable-the-lets-encrypt-integration).
### Change the default HTTPS port
If you need to use an HTTPS port other than the default (443), specify it
as part of the `external_url`:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
external_url "https://gitlab.example.com:2443"
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
### Change the default SSL certificate location
If your hostname is `gitlab.example.com`, a Linux package installation
looks for a private key called `/etc/gitlab/ssl/gitlab.example.com.key`
and a public certificate called `/etc/gitlab/ssl/gitlab.example.com.crt`
by default.
To set a different location of the SSL certificates:
1. Create a directory, give it the appropriate permissions, and place the
`.crt` and `.key` files in the directory:
```shell
sudo mkdir -p /mnt/gitlab/ssl
sudo chmod 755 /mnt/gitlab/ssl
sudo cp gitlab.key gitlab.crt /mnt/gitlab/ssl/
```
You must use the full certificate chain, in the correct order, to prevent
SSL errors when clients connect: first the server certificate,
then all intermediate certificates, and finally the root CA.
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['ssl_certificate'] = "/mnt/gitlab/ssl/gitlab.crt"
nginx['ssl_certificate_key'] = "/mnt/gitlab/ssl/gitlab.key"
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
### Update the SSL certificates
If the content of your SSL certificates has been updated, but no configuration
changes have been made to `/etc/gitlab/gitlab.rb`, then reconfiguring GitLab
doesn't affect NGINX. Instead, you must cause NGINX to
[reload the existing configuration and new certificates](http://nginx.org/en/docs/control.html)
gracefully:
```shell
sudo gitlab-ctl hup nginx
sudo gitlab-ctl hup registry
```
## Configure a reverse proxy or load balancer SSL termination
By default, Linux package installations auto-detect whether to use SSL if `external_url`
contains `https://` and configures NGINX for SSL termination.
However, if you configure GitLab to run behind a reverse proxy or an external load balancer,
some environments may want to terminate SSL outside the GitLab application.
To prevent the bundled NGINX from handling SSL termination:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['listen_port'] = 80
nginx['listen_https'] = false
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
The external load balancer may need access to a GitLab endpoint
that returns a `200` status code (for installations requiring login, the root
page returns a `302` redirect to the login page). In that case, it's
recommended to leverage a
[health check endpoint](https://docs.gitlab.com/ee/administration/monitoring/health_check.html).
Other bundled components, like the Container Registry, GitLab Pages, or Mattermost,
use a similar strategy for proxied SSL. Set the particular component's `*_external_url` with `https://` and
prefix the `nginx[...]` configuration with the component name. For example, the
GitLab Container Registry configuration is prefixed with `registry_`:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
registry_external_url 'https://registry.example.com'
registry_nginx['listen_port'] = 80
registry_nginx['listen_https'] = false
```
The same format can be used for Pages (`pages_` prefix) and Mattermost (`mattermost_` prefix).
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
1. Optional. You may need to configure your reverse proxy or load balancer to
forward certain headers (for example `Host`, `X-Forwarded-Ssl`, `X-Forwarded-For`,
`X-Forwarded-Port`) to GitLab (and Mattermost if you use one). If you forget
this step, you may see improper redirections or errors, like
"422 Unprocessable Entity" or "Can't verify CSRF token authenticity".
Some cloud provider services, such as AWS Certificate Manager (ACM), do not allow
the download of certificates. This prevents them from being used to terminate
on the GitLab instance. If SSL is desired between such a cloud service and
GitLab, another certificate must be used on the GitLab instance.
## Use custom SSL ciphers
By default, the Linux package [uses SSL ciphers](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/0482fb343a4434ba3a2523a7fb576d2bbb2a3f5f/files/gitlab-cookbooks/gitlab/attributes/default.rb#L876) that are a combination of testing on
<https://gitlab.com> and various best practices contributed by the GitLab community.
To change the SSL ciphers:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['ssl_ciphers'] = "CIPHER:CIPHER1"
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
To enable the `ssl_dhparam` directive:
1. Generate `dhparams.pem`:
```shell
openssl dhparam -out /etc/gitlab/ssl/dhparams.pem 2048
```
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['ssl_dhparam'] = "/etc/gitlab/ssl/dhparams.pem"
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Configure the HTTP/2 protocol
By default, when you specify that your GitLab instance is reachable
through HTTPS, the [HTTP/2 protocol](https://www.rfc-editor.org/rfc/rfc7540) is
also enabled.
The Linux package sets the required SSL ciphers that are compatible with
the HTTP/2 protocol.
If you specify your own [custom SSL ciphers](#use-custom-ssl-ciphers) and a cipher is
in the [HTTP/2 cipher blacklist](https://www.rfc-editor.org/rfc/rfc7540#appendix-A),
when you try to reach your GitLab instance you are presented with the
`INADEQUATE_SECURITY` error in your browser. In that case, consider removing the
offending ciphers from the cipher list. Changing ciphers is only necessary if
you have a very specific custom setup.
For more information on why you would want to have the HTTP/2 protocol enabled,
check out the
[NGINX HTTP/2 whitepaper](https://assets.wp.nginx.com/wp-content/uploads/2015/09/NGINX_HTTP2_White_Paper_v4.pdf?_ga=1.127086286.212780517.1454411744).
If changing the ciphers is not an option, you can disable the HTTP/2 support:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['http2_enabled'] = false
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
NOTE:
The HTTP/2 setting only works for the main GitLab application and not for the other services,
like GitLab Pages, Container Registry, and Mattermost.
## Enable 2-way SSL client authentication
To require web clients to authenticate with a trusted certificate, you can
enable 2-way SSL:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['ssl_verify_client'] = "on"
nginx['ssl_client_certificate'] = "/etc/pki/tls/certs/root-certs.pem"
```
1. Optional. You can configure how deeply in the certificate chain NGINX should verify
before deciding that the clients don't have a valid certificate (default is `1`).
Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['ssl_verify_depth'] = "2"
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Configure the HTTP Strict Transport Security (HSTS)
NOTE:
The HSTS settings only work for the main GitLab application and not for the other services,
like GitLab Pages, Container Registry, and Mattermost.
HTTP Strict Transport Security (HSTS) is enabled by default and it informs browsers that
they should only contact the website using HTTPS. When a browser visits a
GitLab instance even once, it remembers to no longer attempt insecure connections,
even when the user is explicitly entering a plain HTTP URL (`http://`). Plain
HTTP URLs are automatically redirected by the browser to the `https://` variant.
By default, `max_age` is set for two years, this is how long a browser will
remember to only connect through HTTPS.
To change the max age value:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
nginx['hsts_max_age'] = 63072000
nginx['hsts_include_subdomains'] = false
```
Setting `max_age` to `0` disables HSTS.
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
For more information on HSTS and NGINX, see <https://blog.nginx.org/blog/http-strict-transport-security-hsts-and-nginx>.
## Install custom public certificates
Some environments connect to external resources for various tasks and GitLab
allows these connections to use HTTPS, and supports connections with self-signed certificates.
GitLab has its own ca-cert bundle that you can add certs to by placing the
individual custom certs in the `/etc/gitlab/trusted-certs` directory. They then
get added to the bundle. They are added using openssl's `c_rehash` method, which
only works on a [single certificate](#using-a-custom-certificate-chain).
The Linux package ships with the official
[Mozilla](https://wiki.mozilla.org/CA/Included_Certificates) collection of trusted root
certification authorities which are used to verify certificate authenticity.
NOTE:
For installations that use self-signed certificates, the Linux package
provides a way to manage these certificates. For more technical details how
this works, see the [details](#details-on-how-gitlab-and-ssl-work)
at the bottom of this page.
To install custom public certificates:
1. Generate the **PEM** or **DER** encoded public certificate from your private key certificate.
1. Copy only the public certificate file into the `/etc/gitlab/trusted-certs` directory.
If you have a multi-node installation, make sure to copy the certificate in all nodes.
- When configuring GitLab to use a custom public certificate, by default, GitLab expects to find a certificate named
after your GitLab domain name with a `.crt` extension. For example, if your server address is
`https://gitlab.example.com`, the certificate should be named `gitlab.example.com.crt`.
- If GitLab needs to connect to an external resource that uses a custom public certificate, store the certificate in
the `/etc/gitlab/trusted-certs` directory with a `.crt` extension. You don't have to name the file based on the
domain name of the related external resource, though it helps to use a consistent naming scheme.
To specify a different path and file name, you can
[change the default SSL certificate location](#change-the-default-ssl-certificate-location).
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
### Using a custom certificate chain
Because of a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/1425), if using a custom certificate
chain, the server, intermediate, and root certificates **must** be put into separate files in the `/etc/gitlab/trusted-certs`
directory.
This applies in both cases where GitLab itself, or external resources GitLab must connect to, are using a custom
certificate chain.
For example, for GitLab itself you can use:
- `/etc/gitlab/trusted-certs/example.gitlab.com.crt`
- `/etc/gitlab/trusted-certs/example.gitlab.com_intermediate.crt`
- `/etc/gitlab/trusted-certs/example.gitlab.com_root.crt`
For external resources GitLab must connect to, you can use:
- `/etc/gitlab/trusted-certs/external-service.gitlab.com.crt`
- `/etc/gitlab/trusted-certs/external-service.gitlab.com_intermediate.crt`
- `/etc/gitlab/trusted-certs/external-service.gitlab.com_root.crt`
## Details on how GitLab and SSL work
The Linux package includes its own library of OpenSSL and links all compiled
programs (e.g. Ruby, PostgreSQL, etc.) against this library. This library is
compiled to look for certificates in `/opt/gitlab/embedded/ssl/certs`.
The Linux package manages custom certificates by symlinking any certificate that
gets added to `/etc/gitlab/trusted-certs/` to `/opt/gitlab/embedded/ssl/certs`
using the [c_rehash](https://www.openssl.org/docs/manmaster/man1/c_rehash.html)
tool. For example, let's suppose we add `customcacert.pem` to
`/etc/gitlab/trusted-certs/`:
```shell
$ sudo ls -al /opt/gitlab/embedded/ssl/certs
total 272
drwxr-xr-x 2 root root 4096 Jul 12 04:19 .
drwxr-xr-x 4 root root 4096 Jul 6 04:00 ..
lrwxrwxrwx 1 root root 42 Jul 12 04:19 7f279c95.0 -> /etc/gitlab/trusted-certs/customcacert.pem
-rw-r--r-- 1 root root 263781 Jul 5 17:52 cacert.pem
-rw-r--r-- 1 root root 147 Feb 6 20:48 README
```
Here we see the fingerprint of the certificate is `7f279c95`, which links to
the custom certificate.
What happens when we make an HTTPS request? Let's take a simple Ruby program:
```ruby
#!/opt/gitlab/embedded/bin/ruby
require 'openssl'
require 'net/http'
Net::HTTP.get(URI('https://www.google.com'))
```
This is what happens behind the scenes:
1. The "require `openssl`" line causes the interpreter to load `/opt/gitlab/embedded/lib/ruby/2.3.0/x86_64-linux/openssl.so`.
1. The `Net::HTTP` call then attempts to read the default certificate bundle in `/opt/gitlab/embedded/ssl/certs/cacert.pem`.
1. SSL negotiation occurs.
1. The server sends its SSL certificates.
1. If the certificates that are sent are covered by the bundle, SSL finishes successfully.
1. Otherwise, OpenSSL may validate other certificates by searching for files
that match their fingerprints inside the predefined certificate directory. For
example, if a certificate has the fingerprint `7f279c95`, OpenSSL will attempt
to read `/opt/gitlab/embedded/ssl/certs/7f279c95.0`.
Note that the OpenSSL library supports the definition of `SSL_CERT_FILE` and
`SSL_CERT_DIR` environment variables. The former defines the default
certificate bundle to load, while the latter defines a directory in which to
search for more certificates. These variables should not be necessary if you
have added certificates to the `trusted-certs` directory. However, if for some
reason you need to set them, they can be [defined as environment variables](../environment-variables.md). For example:
```ruby
gitlab_rails['env'] = {"SSL_CERT_FILE" => "/usr/lib/ssl/private/customcacert.pem"}
```
## Troubleshooting
See our [guide for troubleshooting SSL](ssl_troubleshooting.md).
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Troubleshooting SSL
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
This page contains a list of common SSL-related errors and scenarios that you
may encounter while working with GitLab. It should serve as an addition to the
main SSL documentation:
- [Configure SSL for a Linux package installation](index.md).
- [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html).
- [Configure HTTPS manually](index.md#configure-https-manually).
## Useful OpenSSL Debugging Commands
Sometimes it's helpful to get a better picture of the SSL certificate chain by viewing it directly
at the source. These commands are part of the standard OpenSSL library of tools for diagnostics and
debugging.
NOTE:
GitLab includes its own [custom-compiled version of OpenSSL](index.md#details-on-how-gitlab-and-ssl-work)
that all GitLab libraries are linked against. It's important to run the following commands using
this OpenSSL version.
- Perform a test connection to the host over HTTPS. Replace `HOSTNAME` with your GitLab URL
(excluding HTTPS), and replace `port` with the port that serves HTTPS connections (usually 443):
```shell
echo | /opt/gitlab/embedded/bin/openssl s_client -connect HOSTNAME:port
```
The `echo` command sends a null request to the server, causing it to close the connection rather
than wait for additional input. You can use the same command to test remote hosts (for example, a
server hosting an external repository), by replacing `HOSTNAME:port` with the remote host's domain
and port number.
This command's output shows you the certificate chain, any public certificates the server
presents, along with validation or connection errors if they occur. This makes for a quick check
for any immediate issues with your SSL settings.
- View a certificate's details in text form using `x509`. Be sure to replace
`/path/to/certificate.crt` with the certificate's path:
```shell
/opt/gitlab/embedded/bin/openssl x509 -in /path/to/certificate.crt -text -noout
```
For example, GitLab automatically fetches and places certificates acquired from Let's Encrypt at
`/etc/gitlab/ssl/hostname.crt`. You can use the `x509` command with that path to quickly display
the certificate's information (for example, the hostname, issuer, validity period, and more).
If there's a problem with the certificate, [an error occurs](#custom-certificates-missing-or-skipped).
- Fetch a certificate from a server and decode it. This combines both of the above commands to fetch
the server's SSL certificate and decode it to text:
```shell
echo | /opt/gitlab/embedded/bin/openssl s_client -connect HOSTNAME:port | /opt/gitlab/embedded/bin/openssl x509 -text -noout
```
## Common SSL errors
1. `SSL certificate problem: unable to get local issuer certificate`
This error indicates the client cannot get the root CA. To fix this, you can either [trust the root CA](index.md#install-custom-public-certificates) of the server you are trying to connect to on the client or [modify the certificate](index.md#configure-https-manually) to present the full chained certificate on the server you are trying to connect to.
NOTE:
It is recommended to use the full certificate chain in order to prevent SSL errors when clients connect. The full certificate chain order should consist of the server certificate first, followed by all intermediate certificates, with the root CA last.
1. `unable to verify the first certificate`
This error indicates that an incomplete certificate chain is being presented by the server. To fix this error, you will need to [replace server's certificate with the full chained certificate](index.md#configure-https-manually). The full certificate chain order should consist of the server certificate first, followed by all intermediate certificates, with the root CA last.
NOTE:
If you get this error while running the system OpenSSL utility instead of the `/opt/gitlab/embedded/bin/openssl` utility, make sure you update your CA certificates at the OS level to fix it.
1. `certificate signed by unknown authority`
This error indicates that the client does not trust the certificate or CA. To fix this error, the client connecting to server will need to [trust the certificate or CA](index.md#install-custom-public-certificates).
1. `SSL certificate problem: self signed certificate in certificate chain`
This error indicates that the client does not trust the certificate or CA. To fix this error, the client connecting to server will need to [trust the certificate or CA](index.md#install-custom-public-certificates).
1. `x509: certificate relies on legacy Common Name field, use SANs instead`
This error indicates that [SANs](http://wiki.cacert.org/FAQ/subjectAltName) (subjectAltName) must be configured in the certificate. For more information, see [this issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28841).
## Reconfigure Fails Due to Certificates
```shell
ERROR: Not a certificate: /opt/gitlab/embedded/ssl/certs/FILE. Move it from /opt/gitlab/embedded/ssl/certs to a different location and reconfigure again.
```
Check `/opt/gitlab/embedded/ssl/certs` and remove any files other than `README.md` that aren't valid X.509 certificates.
NOTE:
Running `gitlab-ctl reconfigure` constructs symlinks named from the subject hashes
of your custom public certificates and places them in `/opt/gitlab/embedded/ssl/certs/`.
Broken symlinks in `/opt/gitlab/embedded/ssl/certs/` will be automatically removed.
Files other than `cacert.pem` and `README.md` stored in
`/opt/gitlab/embedded/ssl/certs/` will be moved into the `/etc/gitlab/trusted-certs/`.
## Custom Certificates Missing or Skipped
GitLab versions ***8.9.0***, ***8.9.1***, and ***8.9.2*** all mistakenly used the
`/etc/gitlab/ssl/trusted-certs/` directory. This directory is safe to remove if it
is empty. If it still contains custom certificates then move them to `/etc/gitlab/trusted-certs/`
and run `gitlab-ctl reconfigure`.
If no symlinks are created in `/opt/gitlab/embedded/ssl/certs/` and you see
the message "Skipping `cert.pem`" after running `gitlab-ctl reconfigure`, that
means there may be one of four issues:
1. The file in `/etc/gitlab/trusted-certs/` is a symlink
1. The file is not a valid PEM- or DER-encoded certificate
1. Perl is not installed on the operating system which is needed for c_rehash to properly symlink certificates
1. The certificate contains the string `TRUSTED`
Test the certificate's validity using the commands below:
```shell
/opt/gitlab/embedded/bin/openssl x509 -in /etc/gitlab/trusted-certs/example.pem -text -noout
/opt/gitlab/embedded/bin/openssl x509 -inform DER -in /etc/gitlab/trusted-certs/example.der -text -noout
```
Invalid certificate files produce the following outputs:
- ```shell
unable to load certificate
140663131141784:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
```
- ```shell
cannot load certificate
PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
```
In either of those cases, and if your certificates begin and end with anything other than the following:
```shell
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
```
Then they are not compatible with GitLab. You should separate them into the certificate components (server, intermediate, root), and convert them to the compatible PEM format.
To test if `c_rehash` is not symlinking the certificate due to a missing perl interpreter:
```shell
$ /opt/gitlab/embedded/bin/c_rehash /etc/gitlab/trusted-certs
bash: /opt/gitlab/embedded/bin/c_rehash: /usr/bin/perl: bad interpreter: No such file or directory
```
If you see this message, you will need to install perl with your distribution's package manager.
If you inspect the certificate itself, then look for the string `TRUSTED`:
```plaintext
-----BEGIN TRUSTED CERTIFICATE-----
...
-----END TRUSTED CERTIFICATE-----
```
If it does, like the example above, then try removing the string `TRUSTED` and running `gitlab-ctl reconfigure` again.
## Custom certificates not detected
If after running `gitlab-ctl reconfigure`:
1. no symlinks are created in `/opt/gitlab/embedded/ssl/certs/`;
1. you have placed custom certificates in `/etc/gitlab/trusted-certs/`; and
1. you do not see any skipped or symlinked custom certificate messages
You may be encountering an issue where a Linux package installation thinks that the custom
certificates have already been added.
To resolve, delete the trusted certificates directory hash:
```shell
rm /var/opt/gitlab/trusted-certs-directory-hash
```
Then run `gitlab-ctl reconfigure` again. The reconfigure should now detect and symlink
your custom certificates.
## Let's Encrypt Certificate signed by unknown authority
The initial implementation of Let's Encrypt integration only used the certificate, not the full certificate chain.
Starting in 10.5.4, the full certificate chain will be used. For installs which are already using a certificate, the switchover will not happen until the renewal logic indicates the certificate is near expiration. To force it sooner, run the following
```shell
rm /etc/gitlab/ssl/HOSTNAME*
gitlab-ctl reconfigure
```
Where HOSTNAME is the hostname of the certificate.
## Let's Encrypt fails on reconfigure
NOTE:
You can test your domain using the [Let's Debug](https://letsdebug.net/)
diagnostic tool. It can help you figure out why you can't issue a Let's Encrypt
certificate.
When you reconfigure, there are common scenarios under which Let's Encrypt may fail:
- Let's Encrypt may fail if your server isn't able to reach the Let's Encrypt verification servers or vice versa:
```shell
letsencrypt_certificate[gitlab.domain.com] (letsencrypt::http_authorization line 3) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [gitlab.domain.com] Validation failed for domain gitlab.domain.com
```
If you run into issues reconfiguring GitLab due to Let's Encrypt [make sure you have ports 80 and 443 open and accessible](index.md#enable-the-lets-encrypt-integration).
- Your domain's Certification Authority Authorization (CAA) record does not allow Let's Encrypt to issue a certificate for your domain. Look for the following error in the reconfigure output:
```shell
letsencrypt_certificate[gitlab.domain.net] (letsencrypt::http_authorization line 5) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for gitlab.domain.net] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [gitlab.domain.com] Validation failed, unable to request certificate
```
- If you're using a test domain such as `gitlab.example.com`, without a certificate, you'll see the `unable to request certificate` error shown above. In that case, disable Let's Encrypt by setting `letsencrypt['enable'] = false` in `/etc/gitlab/gitlab.rb`.
- [Let's Encrypt enforces rate limits](https://letsencrypt.org/docs/rate-limits/),
which is at the top-level domain. In case you're using your cloud provider's
hostname as the `external_url`, for example `*.cloudapp.azure.com`, Let's
Encrypt would enforce limits to `azure.com`, which could make the certificate
creation incomplete.
In that case, you can try renewing the Let's Encrypt certificates manually:
```shell
sudo gitlab-ctl renew-le-certs
```
## Using an internal CA certificate with GitLab
After configuring a GitLab instance with an internal CA certificate, you might
not be able to access it by using various CLI tools. You may experience the
following issues:
- `curl` fails:
```shell
curl "https://gitlab.domain.tld"
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
```
- Testing by using the [rails console](https://docs.gitlab.com/ee/administration/operations/rails_console.html#starting-a-rails-console-session)
also fails:
```ruby
uri = URI.parse("https://gitlab.domain.tld")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = 1
response = http.request(Net::HTTP::Get.new(uri.request_uri))
...
Traceback (most recent call last):
1: from (irb):5
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate))
```
- The error `SSL certificate problem: unable to get local issuer certificate`
is displayed when setting up a [mirror](https://docs.gitlab.com/ee/user/project/repository/mirror/index.html)
from this GitLab instance.
- `openssl` works when specifying the path to the certificate:
```shell
/opt/gitlab/embedded/bin/openssl s_client -CAfile /root/my-cert.crt -connect gitlab.domain.tld:443
```
If you have the previously described issues, add your certificate to
`/etc/gitlab/trusted-certs`, and then run `sudo gitlab-ctl reconfigure`.
## X.509 key values mismatch error
After configuring your instance with a certificate bundle, NGINX may display
the following error message:
`SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch`
This error message means that the server certificate and key you have provided
don't match. You can confirm this by running the following command and then
comparing the output:
```shell
openssl rsa -noout -modulus -in path/to/your/.key | openssl md5
openssl x509 -noout -modulus -in path/to/your/.crt | openssl md5
```
The following is an example of an md5 output between a matching key and
certificate. Note the matching md5 hashes:
```shell
$ openssl rsa -noout -modulus -in private.key | openssl md5
4f49b61b25225abeb7542b29ae20e98c
$ openssl x509 -noout -modulus -in public.crt | openssl md5
4f49b61b25225abeb7542b29ae20e98c
```
This is an opposing output with a non-matching key and certificate which shows
different md5 hashes:
```shell
$ openssl rsa -noout -modulus -in private.key | openssl md5
d418865077299af27707b1d1fa83cd99
$ openssl x509 -noout -modulus -in public.crt | openssl md5
4f49b61b25225abeb7542b29ae20e98c
```
If the two outputs differ like the previous example, there's a mismatch between
the certificate and key. Contact the provider of the SSL certificate for
further support.
## Using GitLab Runner with a GitLab instance configured with internal CA certificate or self-signed certificate
Besides getting the errors mentioned in
[Using an internal CA certificate with GitLab](ssl_troubleshooting.md#using-an-internal-ca-certificate-with-gitlab),
your CI pipelines may get stuck in `Pending` status. In the runner logs you may
see the following error message:
```shell
Dec 6 02:43:17 runner-host01 gitlab-runner[15131]: #033[0;33mWARNING: Checking for jobs... failed
#033[0;m #033[0;33mrunner#033[0;m=Bfkz1fyb #033[0;33mstatus#033[0;m=couldn't execute POST against
https://gitlab.domain.tld/api/v4/jobs/request: Post https://gitlab.domain.tld/api/v4/jobs/request:
x509: certificate signed by unknown authority
```
Follow the details in [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html).
## Mirroring a remote GitLab repository that uses a self-signed SSL certificate
When configuring a local GitLab instance to [mirror a repository](https://docs.gitlab.com/ee/user/project/repository/mirror/index.html)
from a remote GitLab instance that uses a self-signed certificate, you may see
the `SSL certificate problem: self signed certificate` error message in the
user interface.
The cause of the issue can be confirmed by checking if:
- `curl` fails:
```shell
$ curl "https://gitlab.domain.tld"
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
```
- Testing by using the Rails console also fails:
```ruby
uri = URI.parse("https://gitlab.domain.tld")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = 1
response = http.request(Net::HTTP::Get.new(uri.request_uri))
...
Traceback (most recent call last):
1: from (irb):5
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate))
```
To fix this problem:
- Add the self-signed certificate from the remote GitLab instance to the
`/etc/gitlab/trusted-certs` directory on the local GitLab instance, and then
run `sudo gitlab-ctl reconfigure` as per the instructions for
[installing custom public certificates](index.md#install-custom-public-certificates).
- If your local GitLab instance was installed using the Helm Charts, you can
[add your self-signed certificate to your GitLab instance](https://docs.gitlab.com/runner/install/kubernetes.html#providing-a-custom-certificate-for-accessing-gitlab).
You may also get another error message when trying to mirror a repository from
a remote GitLab instance that uses a self-signed certificate:
```shell
2:Fetching remote upstream failed: fatal: unable to access &amp;#39;https://gitlab.domain.tld/root/test-repo/&amp;#39;:
SSL: unable to obtain common name from peer certificate
```
In this case, the problem can be related to the certificate itself:
1. Validate that your self-signed certificate isn't missing a common name. If it
is, regenerate a valid certificate
1. Add the certificate to `/etc/gitlab/trusted-certs`.
1. Run `sudo gitlab-ctl reconfigure`.
## Unable to perform Git operations due to an internal or self-signed certificate
If your GitLab instance is using a self-signed certificate, or if the
certificate is signed by an internal certificate authority (CA), you might
experience the following errors when attempting to perform Git operations:
```shell
$ git clone https://gitlab.domain.tld/group/project.git
Cloning into 'project'...
fatal: unable to access 'https://gitlab.domain.tld/group/project.git/': SSL certificate problem: self signed certificate
```
```shell
$ git clone https://gitlab.domain.tld/group/project.git
Cloning into 'project'...
fatal: unable to access 'https://gitlab.domain.tld/group/project.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
```
To fix this problem:
- If possible, use SSH remotes for all Git operations. This is considered more
secure and convenient to use.
- If you must use HTTPS remotes, you can try the following:
- Copy the self-signed certificate or the internal root CA certificate to a
local directory (for example, `~/.ssl`) and configure Git to trust your
certificate:
```shell
git config --global http.sslCAInfo ~/.ssl/gitlab.domain.tld.crt
```
- Disable SSL verification in your Git client. This is intended as a
temporary measure, as it could be considered a security risk.
```shell
git config --global http.sslVerify false
```
## SSL_connect wrong version number
A misconfiguration may result in:
- `gitlab-rails/exceptions_json.log` entries containing:
```plaintext
"exception.class":"Excon::Error::Socket","exception.message":"SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError)",
"exception.class":"Excon::Error::Socket","exception.message":"SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError)",
```
- `gitlab-workhorse/current` containing:
```plaintext
http: server gave HTTP response to HTTPS client
http: server gave HTTP response to HTTPS client
```
- `gitlab-rails/sidekiq.log` or `sidekiq/current` containing:
```plaintext
message: SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError)
message: SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError)
```
Some of these errors come from the Excon Ruby gem, and could be generated in
circumstances where GitLab is configured to initiate an HTTPS session to a
remote server that is serving only HTTP.
One scenario is that you're using [object storage](https://docs.gitlab.com/ee/administration/object_storage.html), which
isn't served under HTTPS. GitLab is misconfigured and attempts a TLS handshake,
but the object storage responds with plain HTTP.
## `schannel: SEC_E_UNTRUSTED_ROOT`
If you're on Windows and get the following error:
```plaintext
Fatal: unable to access 'https://gitlab.domain.tld/group/project.git': schannel: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted."
```
You must specify that Git should use OpenSSL:
```shell
git config --system http.sslbackend openssl
```
Alternatively, you can ignore SSL verification by running:
WARNING:
Proceed with caution when [ignoring SSL](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpsslVerify)
due to the potential security issues associated with disabling this option at global level. Use this option _only_ when troubleshooting, and reinstate SSL verification immediately after.
```shell
git config --global http.sslVerify false
```
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Troubleshooting Omnibus GitLab installation issues
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
Use this page to learn about common issues users can encounter when installing Omnibus GitLab packages.
## Hash Sum mismatch when downloading packages
`apt-get install` outputs something like:
```plaintext
E: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/pool/trusty/main/g/gitlab-ce/gitlab-ce_8.1.0-ce.0_amd64.deb Hash Sum mismatch
```
Run the following to fix this:
```shell
sudo rm -rf /var/lib/apt/lists/partial/*
sudo apt-get update
sudo apt-get clean
```
See [Joe Damato's from Packagecloud comment](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/628#note_1824330) and [his blog article](https://blog.packagecloud.io/apt-hash-sum-mismatch/) for more context.
Another workaround is to download the package manually by selecting the correct package from the [CE packages](https://packages.gitlab.com/gitlab/gitlab-ce) or [EE packages](https://packages.gitlab.com/gitlab/gitlab-ee) repository:
```shell
curl -LJO "https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/trusty/gitlab-ce_8.1.0-ce.0_amd64.deb/download"
dpkg -i gitlab-ce_8.1.0-ce.0_amd64.deb
```
## Installation on openSUSE and SLES platforms warns about unknown key signature
Omnibus GitLab packages are [signed with GPG keys](update/package_signatures.md) in
addition to the package repositories providing signed metadata. This ensures
authenticity and integrity of the packages that are distributed to the users.
However, the package manager used in openSUSE and SLES operating systems may
sometime raise false warnings with these signatures, similar to
```plaintext
File 'repomd.xml' from repository 'gitlab_gitlab-ce' is signed with an unknown key '14219A96E15E78F4'. Continue? [yes/no] (no):
File 'repomd.xml' from repository 'gitlab_gitlab-ce' is signed with an unknown key '14219A96E15E78F4'. Continue? [yes/no] (no): yes
```
This is a known bug with zypper where zypper ignores the `gpgkey` keyword in the
repository configuration file. With later versions of Packagecloud, there may be
improvements regarding this, but currently users have to manually agree to
package installation.
So, in openSUSE or SLES systems, if such a warning is displayed, it is safe to
continue installation.
## apt/yum complains about GPG signatures
You already have GitLab repositories configured, and ran `apt-get update`,
`apt-get install` or `yum install`, and saw errors like the following:
```plaintext
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 3F01618A51312F3F
```
or
```plaintext
https://packages.gitlab.com/gitlab/gitlab-ee/el/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for gitlab-ee
```
This is because on April 2020, GitLab changed the GPG keys used to sign
metadata of the apt and yum repositories available through the
[Packagecloud instance](https://packages.gitlab.com). If you see this error, it
generally means you do not have the public keys currently used to sign
repository metadata in your keyring. To fix this error, follow the
[steps to fetch the new key](update/package_signatures.md#fetching-new-keys-after-2020-04-06).
## Reconfigure shows an error: `NoMethodError - undefined method '[]=' for nil:NilClass`
You ran `sudo gitlab-ctl reconfigure` or package upgrade triggered the
reconfigure which produced error similar to:
```plaintext
================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb
================================================================================
NoMethodError
-------------
undefined method '[]=' for nil:NilClass
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:21:in 'from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in 'from_file'
Relevant File Content:
```
This error is thrown when `/etc/gitlab/gitlab.rb` configuration file contains
configuration that is invalid or unsupported. Double check that there are no
typos or that the configuration file does not contain obsolete configuration.
You can check the latest available configuration by using `sudo gitlab-ctl diff-config` or check the latest [`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template).
## GitLab is unreachable in my browser
Try [specifying](settings/configuration.md#configure-the-external-url-for-gitlab) an `external_url` in
`/etc/gitlab/gitlab.rb`. Also check your firewall settings; port 80 (HTTP) or
443 (HTTPS) might be closed on your GitLab server.
Note that specifying the `external_url` for GitLab, or any other bundled service (Registry and
Mattermost) doesn't follow the `key=value` format that other parts of `gitlab.rb` follow. Make sure
that you have them set in the following format:
```ruby
external_url "https://gitlab.example.com"
registry_external_url "https://registry.example.com"
mattermost_external_url "https://mattermost.example.com"
```
NOTE:
Don't add the equal sign (`=`) between `external_url` and the value.
## Emails are not being delivered
To test email delivery you can create a new GitLab account for an email that is
not used in your GitLab instance yet.
If necessary, you can modify the 'From' field of the emails sent by GitLab with
the following setting in `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
```
Run `sudo gitlab-ctl reconfigure` for the change to take effect.
## TCP ports for GitLab services are already taken
By default, Puma listens at TCP address 127.0.0.1:8080. NGINX
listens on port 80 (HTTP) and/or 443 (HTTPS) on all interfaces.
The ports for Redis, PostgreSQL and Puma can be overridden in
`/etc/gitlab/gitlab.rb` as follows:
```ruby
redis['port'] = 1234
postgresql['port'] = 2345
puma['port'] = 3456
```
For NGINX port changes please see [Setting the NGINX listen port](settings/nginx.md#setting-the-nginx-listen-port).
## Git user does not have SSH access
### SELinux-enabled systems
On SELinux-enabled systems the Git user's `.ssh` directory or its contents can
get their security context messed up. You can fix this by running `sudo
gitlab-ctl reconfigure`, which sets the `gitlab_shell_t` security context on
`/var/opt/gitlab/.ssh`.
To improve this behavior, we set the context permanently using
`semanage`. The runtime dependency `policycoreutils-python` has been added to the
RPM package for RHEL based operating systems in order to ensure the `semanage`
command is available.
#### Diagnose and resolve SELinux issues
Omnibus GitLab detects default path changes in `/etc/gitlab/gitlab.rb` and should apply
the correct file contexts.
NOTE:
From GitLab 16.10 forward, administrators can try `gitlab-ctl apply-sepolicy`
to automatically fix SELinux issues. Consult
`gitlab-ctl apply-sepolicy --help` for runtime options.
For installations using custom data path configuration,
the administrator may have to manually resolve SELinux issues.
Data paths may be altered via `gitlab.rb`, however, a common scenario forces the
use of `symlink` paths. Administrators should be cautious, because `symlink` paths are
not supported for all scenarios, such as [Gitaly data paths](settings/configuration.md#store-git-data-in-an-alternative-directory).
For example, if `/data/gitlab` replaced `/var/opt/gitlab` as the base data directory, the following fixes the security context:
```shell
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/.ssh/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/.ssh/authorized_keys
sudo restorecon -Rv /data/gitlab/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/gitlab-shell/config.yml
sudo restorecon -Rv /data/gitlab/gitlab-shell/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/gitlab-rails/etc/gitlab_shell_secret
sudo restorecon -Rv /data/gitlab/gitlab-rails/
sudo semanage fcontext --list | grep /data/gitlab/
```
After the policies are applied, you can verify the SSH access is working
by getting the welcome message:
```shell
ssh -T git@gitlab-hostname
```
### All systems
The Git user is created, by default, with a locked password, shown by `'!'` in
/etc/shadow. Unless "UsePam yes" is enabled, the OpenSSH daemon prevents the
Git user from authenticating even with SSH keys. An alternative secure solution
is to unlock the password by replacing `'!'` with `'*'` in `/etc/shadow`. The Git
user is still unable to change the password because it runs in a restricted
shell and the `passwd` command for non-superusers requires entering the current
password prior to a new password. The user cannot enter a password that matches
`'*'`, which means the account continues to not have a password.
Keep in mind that the Git user must have access to the system so please review
your security settings at `/etc/security/access.conf` and make sure the Git user
is not blocked.
## PostgreSQL error `FATAL: could not create shared memory segment: Cannot allocate memory`
The packaged PostgreSQL instance tries to allocate 25% of total memory as
shared memory. On some Linux (virtual) servers, there is less shared memory
available, which prevents PostgreSQL from starting. In
`/var/log/gitlab/postgresql/current`:
```plaintext
1885 2014-08-08_16:28:43.71000 FATAL: could not create shared memory segment: Cannot allocate memory
1886 2014-08-08_16:28:43.71002 DETAIL: Failed system call was shmget(key=5432001, size=1126563840, 03600).
1887 2014-08-08_16:28:43.71003 HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space, or exceeded your kernel's SHMALL parameter. You can either reduce the request size or reconfigure the kernel with larger SHMALL. To reduce the request size (currently 1126563840 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
1888 2014-08-08_16:28:43.71004 The PostgreSQL documentation contains more information about shared memory configuration.
```
You can manually lower the amount of shared memory PostgreSQL tries to allocate
in `/etc/gitlab/gitlab.rb`:
```ruby
postgresql['shared_buffers'] = "100MB"
```
Run `sudo gitlab-ctl reconfigure` for the change to take effect.
## PostgreSQL error `FATAL: could not open shared memory segment "/PostgreSQL.XXXXXXXXXX": Permission denied`
By default, PostgreSQL tries to detect the shared memory type to use. If you don't
have shared memory enabled, you might see this error in `/var/log/gitlab/postgresql/current`.
To fix this, you can disable PostgreSQL's shared memory detection. Set the
following value in `/etc/gitlab/gitlab.rb`:
```ruby
postgresql['dynamic_shared_memory_type'] = 'none'
```
Run `sudo gitlab-ctl reconfigure` for the change to take effect.
## PostgreSQL error `FATAL: remaining connection slots are reserved for non-replication superuser connections`
PostgreSQL has a setting for the maximum number of the concurrent connections
to the database server. If you see this error, it means that your GitLab instance is trying to exceed
this limit on the number of concurrent connections.
To fix this problem, you have two options:
- Either increase the max connections value:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
postgresql['max_connections'] = 600
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
- Or, you can consider [using PgBouncer](https://docs.gitlab.com/ee/administration/postgresql/pgbouncer.html) which is a connection pooler for PostgreSQL.
## Reconfigure complains about the GLIBC version
```shell
$ gitlab-ctl reconfigure
/opt/gitlab/embedded/bin/ruby: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/gitlab/embedded/lib/libruby.so.2.1)
/opt/gitlab/embedded/bin/ruby: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by /opt/gitlab/embedded/lib/libruby.so.2.1)
```
This can happen if the omnibus package you installed was built for a different
OS release than the one on your server. Double-check that you downloaded and
installed the correct Omnibus GitLab package for your operating system.
## Reconfigure fails to create the Git user
This can happen if you run `sudo gitlab-ctl reconfigure` as the Git user.
Switch to another user.
More importantly: do not give sudo rights to the Git user or to any of the
other users used by Omnibus GitLab. Bestowing unnecessary privileges on a
system user weakens the security of your system.
## Failed to modify kernel parameters with sysctl
If sysctl cannot modify the kernel parameters you could possibly get an error with the following stack trace:
```plaintext
* execute[sysctl] action run
================================================================================
Error executing action `run` on resource 'execute[sysctl]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '255'
---- Begin output of /sbin/sysctl -p /etc/sysctl.conf ----
```
This is unlikely to happen with non virtualized machines but on a VPS with virtualization like openVZ, container might not have the required module enabled
or container doesn't have access to kernel parameters.
Try [enabling the module](https://serverfault.com/questions/477718/sysctl-p-etc-sysctl-conf-returns-error) on which sysctl errored out.
There is a reported workaround described in [this issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/361) which requires editing the GitLab' internal recipe by supplying the switch which ignores failures. Ignoring errors can have unexpected side effects on the performance of your GitLab server, so it isn't recommended to do so.
Another variation of this error reports the file system is read-only and shows following stack trace:
```plaintext
* execute[load sysctl conf] action run
[execute] sysctl: setting key "kernel.shmall": Read-only file system
sysctl: setting key "kernel.shmmax": Read-only file system
================================================================================
Error executing action `run` on resource 'execute[load sysctl conf]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '255'
---- Begin output of cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p - ----
STDOUT:
STDERR: sysctl: setting key "kernel.shmall": Read-only file system
sysctl: setting key "kernel.shmmax": Read-only file system
---- End output of cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p - ----
Ran cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p - returned 255
```
This error is also reported to occur in virtual machines only, and the recommended workaround is to set the values in the host. The values needed for GitLab can be found inside the file `/opt/gitlab/embedded/etc/90-omnibus-gitlab.conf` in the virtual machine. After setting these values in `/etc/sysctl.conf` file in the host OS, run `cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p -` on the host. Then try running `gitlab-ctl reconfigure` inside the virtual machine. It should detect that the kernel is already running with the necessary settings, and not raise any errors.
You may have to repeat this process for other lines. For example, reconfigure fails three times, after having added something like this to `/etc/sysctl.conf`:
```plaintext
kernel.shmall = 4194304
kernel.sem = 250 32000 32 262
net.core.somaxconn = 2048
kernel.shmmax = 17179869184
```
You may find it easier to look at the line in the Chef output than to find the file (since the file
is different for each error). See the last line of this snippet.
```plaintext
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf kernel.shmall] action create
- create new file /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf
- update content in file /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf from none to 6d765d
--- /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf 2017-11-28 19:09:46.864364952 +0000
+++ /opt/gitlab/embedded/etc/.chef-90-omnibus-gitlab-kernel.shmall.conf kernel.shmall20171128-13622-sduqoj 2017-11-28 19:09:46.864364952 +0000
@@ -1 +1,2 @@
+kernel.shmall = 4194304
```
## I am unable to install Omnibus GitLab without root access
Occasionally people ask if they can install GitLab without root access.
This is problematic for several reasons.
### Installing the .deb or .rpm
To our knowledge there is no clean way to install Debian or RPM
packages as a non-privileged user. You cannot install Omnibus GitLab
RPM's because the Omnibus build process does not create source RPM's.
### Hassle-free hosting on port 80 and 443
The most common way to deploy GitLab is to have a web server
(NGINX/Apache) running on the same server as GitLab, with the web
server listening on a privileged (below-1024) TCP port. In
Omnibus GitLab we provide this convenience by bundling an
automatically configured NGINX service that needs to run its master
process as root to open ports 80 and 443.
If this is problematic, administrators installing GitLab can disable
the bundled NGINX service, but this puts the burden on them to keep
the NGINX configuration in tune with GitLab during application
updates.
### Isolation between Omnibus services
Bundled services in Omnibus GitLab (GitLab itself, NGINX, PostgreSQL,
Redis, Mattermost) are isolated from each other using Unix user
accounts. Creating and managing these user accounts requires root
access. By default, Omnibus GitLab will create the required Unix
accounts during `gitlab-ctl reconfigure` but that behavior can be
[disabled](settings/configuration.md#disable-user-and-group-account-management).
In principle Omnibus GitLab could do with only 2 user accounts (one
for GitLab and one for Mattermost) if we give each application its own
runit (runsvdir), PostgreSQL and Redis process. But this would be a
major change in the `gitlab-ctl reconfigure` Chef code and it would
probably create major upgrade pain for all existing Omnibus GitLab
installations. (We would probably have to rearrange the directory
structure under `/var/opt/gitlab`.)
### Tweaking the operating system for better performance
During `gitlab-ctl reconfigure` we set and install several sysctl
tweaks to improve PostgreSQL performance and increase connection limits.
This can only be done with root access.
## `gitlab-rake assets:precompile` fails with 'Permission denied'
Some users report that running `gitlab-rake assets:precompile` does not work
with the omnibus packages. The short answer to this is: do not run that
command, it is only for GitLab installations from source.
The GitLab web interface uses CSS and JavaScript files, called 'assets' in Ruby
on Rails-speak. In the [upstream GitLab repository](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/assets)
these files are stored in a developer-friendly way: easy to read and edit. When
you are a normal user of GitLab, you do not want these files to be in the
developer friendly format however because that makes GitLab slow. This is why
part of the GitLab setup process is to convert the assets from a
developer-friendly format to an end-user friendly (compact, fast) format; that
is what the `rake assets:precompile` script is for.
When you install GitLab from source (which was the only way to do it before we
had omnibus packages) you need to convert the assets on your GitLab server
every time you update GitLab. People used to overlook this step and there are
still posts, comments and mails out there on the internet where users recommend
each other to run `rake assets:precompile` (which has now been renamed
`gitlab:assets:compile`). With the omnibus packages things are different: when
we build the package [we compile the assets for you](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1cfe925e0c015df7722bb85eddc0b4a3b59c1211/config/software/gitlab-rails.rb#L74).
When you install GitLab with an omnibus package, the converted assets are
already there! That is why you do not need to run `rake assets:precompile` when
you install GitLab from a package.
When `gitlab-rake assets:precompile` fails with a permission error it fails for
a good reason from a security standpoint: the fact that the assets cannot
easily be rewritten makes it harder for an attacker to use your GitLab server
to serve evil JavaScript code to the visitors of your GitLab server.
If you want to run GitLab with custom JavaScript or CSS code you are probably
better off running GitLab from source, or building your own packages.
If you really know what you are doing,
you can execute `gitlab-rake gitlab:assets:compile` like this:
```shell
sudo NO_PRIVILEGE_DROP=true USE_DB=false gitlab-rake gitlab:assets:clean gitlab:assets:compile
# user and path might be different if you changed the defaults of
# user['username'], user['group'] and gitlab_rails['dir'] in gitlab.rb
sudo chown -R git:git /var/opt/gitlab/gitlab-rails/tmp/cache
```
## 'Short read or OOM loading DB' error
Try [cleaning the old Redis session](https://docs.gitlab.com/ee/administration/operations/index.html).
## Apt error 'The requested URL returned error: 403'
When trying to install GitLab using the apt repo if you receive an error similar to:
```shell
W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/DISTRO/dists/CODENAME/main/source/Sources The requested URL returned error: 403
```
check if there is a repository cacher in front of your server, like for example `apt-cacher-ng`.
Add the following line to apt-cacher-ng config(eg. in `/etc/apt-cacher-ng/acng.conf`):
```shell
PassThroughPattern: (packages\.gitlab\.com|packages-gitlab-com\.s3\.amazonaws\.com|*\.cloudfront\.net)
```
Read more about `apt-cacher-ng` and the reasons why this change is needed [on the packagecloud blog](https://blog.packagecloud.io/using-apt-cacher-ng-with-ssl-tls/).
## Using self signed certificate or custom certificate authorities
If you are installing GitLab in an isolated network with custom certificate authorities or using self-signed certificate make sure that the certificate can be reached by GitLab. Not doing so will cause errors like:
```shell
Faraday::SSLError (SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed)
```
when GitLab tries to connect with the internal services like GitLab Shell.
To fix these errors, see the [Install Custom Public Certificates](settings/ssl/index.md#install-custom-public-certificates) section.
## error: proxyRoundTripper: XXX failed with: "net/http: timeout awaiting response headers"
If GitLab Workhorse doesn't receive an answer from
GitLab within 1 minute (default), it will serve a 502 page.
There are various reasons why the request might timeout, perhaps user
was loading a very large diff or similar.
You can increase the default timeout value by setting the value in `/etc/gitlab/gitlab.rb`:
```ruby
gitlab_workhorse['proxy_headers_timeout'] = "2m0s"
```
Save the file and [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) for the changes to take effect.
## The change you wanted was rejected
Most likely you have GitLab setup in an environment that has proxy in front
of GitLab and the proxy headers set in package by default are incorrect
for your environment.
See [Change the default proxy headers section of NGINX doc](settings/nginx.md#change-the-default-proxy-headers) for details on
how to override the default headers.
## Can't verify CSRF token authenticity Completed 422 Unprocessable
Most likely you have GitLab setup in an environment that has proxy in front
of GitLab and the proxy headers set in package by default are incorrect
for your environment.
See [Change the default proxy headers section of NGINX doc](settings/nginx.md#change-the-default-proxy-headers) for details on
how to override the default headers.
## Extension missing pg_trgm
[GitLab requires](https://docs.gitlab.com/ee/install/requirements.html#postgresql-requirements)
the PostgreSQL extension `pg_trgm`.
If you are using Omnibus GitLab package with the bundled database, the extension
should be automatically enabled when you upgrade.
If you however, are using an external (non-packaged) database, you will need to
enable the extension manually. The reason for this is that Omnibus GitLab
package with external database has no way of confirming if the extension exists,
and it also doesn't have a way of enabling the extension.
To fix this issue, you'll need to first install the `pg_trgm` extension.
The extension is located in the `postgresql-contrib` package. For Debian:
```shell
sudo apt-get install postgresql-contrib
```
Once the extension is installed, access the `psql` as superuser and enable the
extension.
1. Access `psql` as superuser:
```shell
sudo gitlab-psql -d gitlabhq_production
```
1. Enable the extension:
```plaintext
CREATE EXTENSION pg_trgm;
\q
```
1. Now run migrations again:
```shell
sudo gitlab-rake db:migrate
```
---
If using Docker, you first need to access your container, then run the commands
above, and finally restart the container.
1. Access the container:
```shell
docker exec -it gitlab bash
```
1. Run the commands above
1. Restart the container:
```shell
docker restart gitlab
```
## Errno::ENOMEM: Cannot allocate memory during backup or upgrade
[GitLab requires](https://docs.gitlab.com/ee/install/requirements.html#memory)
2GB of available memory to run without errors. Having 2GB of memory installed may
not be enough depending on the resource usage of other processes on your server.
If GitLab runs fine when not upgrading or running a backup, then adding more swap
should solve your problem. If you see the server using swap during normal usage,
you can add more RAM to improve performance.
## NGINX error: 'could not build server_names_hash, you should increase server_names_hash_bucket_size'
If your external URL for GitLab is longer than the default bucket size (64 bytes),
NGINX may stop working and show this error in the logs. To allow larger server
names, double the bucket size in `/etc/gitlab/gitlab.rb`:
```ruby
nginx['server_names_hash_bucket_size'] = 128
```
Run `sudo gitlab-ctl reconfigure` for the change to take effect.
## Reconfigure fails due to "'root' cannot chown" with NFS root_squash
```shell
$ gitlab-ctl reconfigure
================================================================================
Error executing action `run` on resource 'ruby_block[directory resource: /gitlab-data/git-data]'
================================================================================
Errno::EPERM
------------
'root' cannot chown /gitlab-data/git-data. If using NFS mounts you will need to re-export them in 'no_root_squash' mode and try again.
Operation not permitted @ chown_internal - /gitlab-data/git-data
```
This can happen if you have directories mounted using NFS and configured in `root_squash`
mode. Reconfigure is not able to properly set the ownership of your directories. You
will need to switch to using `no_root_squash` in your NFS exports on the NFS server, or
[disable storage directory management](settings/configuration.md#disable-storage-directories-management)
and manage the permissions yourself.
## `gitlab-runsvdir` not starting
This applies to operating systems using systemd (e.g. Ubuntu 18.04+, CentOS, etc.).
`gitlab-runsvdir` starts during the `multi-user.target`
instead of `basic.target`. If you are having trouble starting this service
after upgrading GitLab, you may need to check that your system has properly
booted all the required services for `multi-user.target` via the command:
```shell
systemctl -t target
```
If everything is working properly, the output should show look something like this:
```plaintext
UNIT LOAD ACTIVE SUB DESCRIPTION
basic.target loaded active active Basic System
cloud-config.target loaded active active Cloud-config availability
cloud-init.target loaded active active Cloud-init target
cryptsetup.target loaded active active Encrypted Volumes
getty.target loaded active active Login Prompts
graphical.target loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded active active Local File Systems
multi-user.target loaded active active Multi-User System
network-online.target loaded active active Network is Online
network-pre.target loaded active active Network (Pre)
network.target loaded active active Network
nss-user-lookup.target loaded active active User and Group Name Lookups
paths.target loaded active active Paths
remote-fs-pre.target loaded active active Remote File Systems (Pre)
remote-fs.target loaded active active Remote File Systems
slices.target loaded active active Slices
sockets.target loaded active active Sockets
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
time-sync.target loaded active active System Time Synchronized
timers.target loaded active active Timers
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
22 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
```
Every line should show `loaded active active`. As seen in the line below, if
you see `inactive dead`, this means there may be something wrong:
```plaintext
multi-user.target loaded inactive dead start Multi-User System
```
To examine which jobs may be queued by systemd, run:
```shell
systemctl list-jobs
```
If you see a `running` job, a service may be stuck and thus blocking GitLab
from starting. For example, some users have had trouble with Plymouth not
starting:
```plaintext
1 graphical.target start waiting
107 plymouth-quit-wait.service start running
2 multi-user.target start waiting
169 ureadahead-stop.timer start waiting
121 gitlab-runsvdir.service start waiting
151 system-getty.slice start waiting
31 setvtrgb.service start waiting
122 systemd-update-utmp-runlevel.service start waiting
```
In this case, consider uninstalling Plymouth.
## Init daemon detection in non-Docker container
In Docker containers, GitLab package detects existence of `/.dockerenv` file and
skips automatic detection of an init system. However, in non-Docker containers
(like containerd, cri-o, etc.), that file does not exist and package falls back
to sysvinit, and this can cause issues with installation. To prevent this, users
can explicitly disable init daemon detection by adding the following setting in
`gitlab.rb` file:
```ruby
package['detect_init'] = false
```
If using this configuration, runit service must be started before running
`gitlab-ctl reconfigure`, using the `runsvdir-start` command:
```shell
/opt/gitlab/embedded/bin/runsvdir-start &
```
## `gitlab-ctl reconfigure` hangs while using AWS Cloudformation
The GitLab systemd unit file by default uses `multi-user.target` for both `After`
and `WantedBy` fields. This is done to ensure service runs after `remote-fs` and
`network` targets, and thus GitLab will function properly.
However, this interacts poorly with [cloud-init](https://cloudinit.readthedocs.io/en/latest/)'s
own unit ordering, which is used by AWS Cloudformation.
To fix this, users can make use of `package['systemd_wanted_by']` and
`package['systemd_after']` settings in `gitlab.rb` to specify values needed for
proper ordering and run `sudo gitlab-ctl reconfigure`. After reconfigure has
completed, restart `gitlab-runsvdir` service for changes to take effect.
```shell
sudo systemctl restart gitlab-runsvdir
```
## Errno::EAFNOSUPPORT: Address family not supported by protocol - socket(2)
When starting up GitLab, if an error similar to the following is observed:
```ruby
FATAL: Errno::EAFNOSUPPORT: Address family not supported by protocol - socket(2)
```
Check if the hostnames in use are resolvable and **IPv4**
addresses are returned:
```shell
getent hosts gitlab.example.com
# Example IPv4 output: 192.168.1.1 gitlab.example.com
# Example IPv6 output: 2002:c0a8:0101::c0a8:0101 gitlab.example.com
getent hosts localhost
# Example IPv4 output: 127.0.0.1 localhost
# Example IPv6 output: ::1 localhost
```
If an **IPv6** address format is returned, further check if
**IPv6** protocol support (keyword `ipv6`) is enabled on the
network interface:
```shell
ip addr # or 'ifconfig' on older operating systems
```
When **IPv6** network protocol support is absent or disabled,
but the DNS configuration resolves the hostnames as **IPv6** addresses,
GitLab services will be unable to establish network connections.
This can be resolved by fixing the DNS configurations (or `/etc/hosts`) to
resolve the hosts to an **IPv4** address instead of **IPv6**.
## `URI::InvalidComponentError (bad component(expected host component: my_url.tld)` when `external_url` contains underscores
If you have set `external_url` with underscores (for example `https://my_company.example.com`), you may face the following issues with CI/CD:
- It will not be possible to open project's **Settings > CI/CD** page.
- Runners will not pick up jobs and will fail with an error 500.
If that's the case, [`production.log`](https://docs.gitlab.com/ee/administration/logs/index.html#productionlog) will contain the following error:
```plaintext
Completed 500 Internal Server Error in 50ms (ActiveRecord: 4.9ms | Elasticsearch: 0.0ms | Allocations: 17672)
URI::InvalidComponentError (bad component(expected host component): my_url.tld):
lib/api/helpers/related_resources_helpers.rb:29:in `expose_url'
ee/app/controllers/ee/projects/settings/ci_cd_controller.rb:19:in `show'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'
app/controllers/application_controller.rb:486:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:477:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
```
As a workaround, avoid using underscores in `external_url`. There is an open issue about it: [Setting `external_url` with underscore results in a broken GitLab CI/CD functionality](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6077).
## Upgrade fails with `timeout: run: /opt/gitlab/service/gitaly` error
If the package upgrade fails when running reconfigure with the following error,
check that all Gitaly processes are stopped and then rerun `sudo gitlab-ctl reconfigure`.
```plaintext
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
STDOUT: timeout: run: /opt/gitlab/service/gitaly: (pid 4886) 15030s, got TERM
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1
```
Refer to [issue 341573](https://gitlab.com/gitlab-org/gitlab/-/issues/341573) for more details.
## Reconfigure is stuck when re-installing GitLab
Because of a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7776), you can see the reconfigure process stuck at
`ruby_block[wait for logrotate service socket] action run` after uninstalling GitLab and trying to install it again. This problem occurs when one of the `systemctl` commands are
not executed when [uninstalling GitLab](installation/index.md#uninstall-the-linux-package-omnibus).
To resolve this issue:
- Make sure you followed all the steps when uninstalling GitLab and perform them if necessary.
- Follow the workaround in [issue 7776](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7776).
## Mirroring the GitLab `yum` repository with Pulp or Red Hat Satellite fails
Direct mirroring of the Omnibus GitLab `yum` repositories located at <https://packages.gitlab.com/gitlab/> with [Pulp](https://pulpproject.org/) or
[Red Hat Satellite](https://www.redhat.com/en/technologies/management/satellite) fails
when syncing. Different errors are caused by different software:
- Pulp 2 or Satellite < 6.10 fails with `"Malformed repository: metadata is specified for different set of packages in filelists.xml and in other.xml"` error.
- Satellite 6.10 fails with `"pkgid"` error.
- Pulp 3 or Satellite > 6.10 seems to succeed, but only the repository metadata is synced.
These sync failures are caused by issues with the metadata in the GitLab `yum`
mirror repository. This metadata includes a `filelists.xml.gz` file that
normally includes a list of files for every RPM in the repository. The GitLab
`yum` repository leaves this file mostly empty to work around a size issue that
would be caused if the file was fully populated.
Each GitLab RPM contains an enormous number of files, which when multiplied by
the large number of RPMs in the repository, would result in a huge
`filelists.xml.gz` file if it was fully populated. Because of storage and build
constraints, we create the file but do not populate it. The empty file causes
Pulp and RedHat Satellite (which uses Pulp) repository mirroring of the file to
fail.
Refer to [issue 2766](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2766) for details.
### Work around the issue
To work around the issue:
1. Use an alternative RPM repository mirroring tool like `reposync` or
`createrepo` to make a local copy of the official GitLab `yum` repository. These
tools recreate the repository metadata in the local data, which includes
creating a fully-populated `filelists.xml.gz` file.
1. Point Pulp or Satellite at the local mirror.
### Local mirror example
The following is an example of how to do local mirroring. The example uses:
- [Apache](https://httpd.apache.org/) as the web server for the repository.
- [`reposync`](https://dnf-plugins-core.readthedocs.io/en/latest/reposync.html)
and [`createrepo`](http://createrepo.baseurl.org/) to sync the GitLab
repository to the local mirror. This local mirror can then be used as a source
for Pulp or RedHat Satellite. You can use other tools like
[Cobbler](https://cobbler.github.io/) as well.
In this example:
- The local mirror is running on a `RHEL 8`, `Rocky 8`, or `AlmaLinux 8` system.
- The host name used for the web-server is `mirror.example.com`.
- Pulp 3 syncs from the local mirror.
- Mirroring is of the [GitLab Enterprise Edition repository](https://packages.gitlab.com/gitlab/gitlab-ee).
#### Create and configure an Apache server
The following example shows how to install and configure a basic Apache 2
server to host one or more Yum repository mirrors.
Consult the [Apache](https://httpd.apache.org/) documentation for details on
configuring and securing your web server.
1. Install `httpd`:
```shell
sudo dnf install httpd
```
1. Add a `Directory` stanza to `/etc/httpd/conf/httpd.conf`:
```apache
<Directory “/var/www/html/repos“>
Options All Indexes FollowSymLinks
Require all granted
</Directory>
```
1. Complete the `httpd` configuration:
```shell
sudo rm -f /etc/httpd/conf.d/welcome.conf
sudo mkdir /var/www/html/repos
sudo systemctl enable httpd --now
```
#### Get the mirrored Yum repository URL
1. Install the GitLab repository `yum` configuration file:
```shell
curl "https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh" | sudo bash
sudo dnf config-manager --disable gitlab_gitlab-ee gitlab_gitlab-ee-source
```
1. Get the repository URL:
```shell
sudo dnf config-manager --dump gitlab_gitlab-ee | grep baseurl
baseurl = https://packages.gitlab.com/gitlab/gitlab-ee/el/8/x86_64
```
You use the contents of `baseurl` as the source of the local mirror. For example,
`https://packages.gitlab.com/gitlab/gitlab-ee/el/8/x86_64`.
#### Create the local mirror
1. Install the `createrepo` package:
```shell
sudo dnf install createrepo
```
1. Run `reposync` to copy RPMs to the local mirror:
```shell
sudo dnf reposync --arch x86_64 --repoid=gitlab_gitlab-ee --download-path=/var/www/html/repos --newest-only
```
The `--newest-only` option only downloads the latest RPM. If you omit this
option, all RPMs in the repo (approximately 1 GB each) are downloaded.
1. Run `createrepo` to recreate the repository metadata:
```shell
sudo createrepo -o /var/www/html/repos/gitlab_gitlab-ee /var/www/html/repos/gitlab_gitlab-ee
```
The local mirror repository should now be available at
<http://mirror.example.com/repos/gitlab_gitlab-ee/>.
#### Update the local mirror
Your local mirror should be updated periodically to get new RPMs as new GitLab
versions are released. One way of doing this is using `cron`.
Create `/etc/cron.daily/sync-gitlab-mirror` with the following contents:
```shell
#!/bin/sh
dnf reposync --arch x86_64 --repoid=gitlab_gitlab-ee --download-path=/var/www/html/repos --newest-only --delete
createrepo -o /var/www/html/repos/gitlab_gitlab-ee /var/www/html/repos/gitlab_gitlab-ee
```
The `--delete` option used in the `dnf reposync` command deletes RPMs in the
local mirror that are no longer present in the corresponding GitLab repository.
#### Using the local mirror
1. Create the Pulp `repository` and `remote`:
```shell
pulp rpm repository create --retain-package-versions=1 --name "gitlab-ee"
pulp rpm remote create --name gitlab-ee --url "http://mirror.example.com/repos/gitlab_gitlab-ee/" --policy immediate
pulp rpm repository update --name gitlab-ee --remote gitlab-ee
```
1. Sync the repository:
```shell
pulp rpm repository sync --name gitlab-ee
```
This command must be run periodically to update the local mirror with changes
to the GitLab repository.
After the repository is synced, you can create a publication and distribution to
make it available. See <https://docs.pulpproject.org/pulp_rpm/> for details.
## Error: `E: connection refused to d20rj4el6vkp4c.cloudfront.net 443`
When you install a package hosted on our package repository at `packages.gitlab.com`, your client will receive and follow a redirect to the CloudFront address `d20rj4el6vkp4c.cloudfront.net`. Servers in an air-gapped environment can receive the following errors:
```shell
E: connection refused to d20rj4el6vkp4c.cloudfront.net 443
```
```shell
Failed to connect to d20rj4el6vkp4c.cloudfront.net port 443: Connection refused
```
To resolve this issue, you have three options:
- If you can allowlist by domain, add the endpoint `d20rj4el6vkp4c.cloudfront.net` to your firewall settings.
- If you cannot allowlist by domain, add the [CloudFront IP address ranges](https://d7uri8nf7uskq.cloudfront.net/tools/list-cloudfront-ips) to your firewall settings. You must
keep this list synced with your firewall settings because they can change.
- Manually download the package file and upload it to your server.
## Do I need to increase `net.core.somaxconn` ?
The following may assist in identifying if the value of `net.core.somaxconn`
is set too low:
```shell
$ netstat -ant | grep -c SYN_RECV
4
```
The return value from `netstat -ant | grep -c SYN_RECV` is the number of connections
waiting to be established. If the value is greater than `net.core.somaxconn`:
```shell
$ sysctl net.core.somaxconn
net.core.somaxconn = 1024
```
You may experience timeouts or HTTP 502 errors and is recommended to increase this
value by updating the `puma['somaxconn']` variable in your `gitlab.rb`.
## `exec request failed on channel 0` or `shell request failed on channel 0` errors
When pulling or pushing by using Git over SSH, you might see the following errors:
- `exec request failed on channel 0`
- `shell request failed on channel 0`
These errors can happen if the number of processes from the `git` user is above the limit.
To try and resolve this issue:
1. Increase the `nproc` setting for the `git` user in the `/etc/security/limits.conf` file on the nodes where `gitlab-shell` is running.
Typically, `gitlab-shell` runs on GitLab Rails nodes.
1. Retry the pull or push Git command.
## Hung installation after SSH connection loss
If you're installing GitLab on a remote virtual machine and your SSH connection gets lost,
the installation could hang with a zombie `dpkg` process. To resume the installation:
1. Run `top` to find the process ID of the associated `apt` process, which is the parent of the `dpkg` process.
1. Kill the `apt` process by running `sudo kill <PROCESS_ID>`.
1. Only if doing a fresh install, run `sudo gitlab-ctl cleanse`. This step erases existing data, so must not be used on upgrades.
1. Run `sudo dpkg configure -a`.
1. Edit the `gitlab.rb` file to include the desired external URL and any other configuration that might be missing.
1. Run `sudo gitlab-ctl reconfigure`.
## Redis-related error when reconfiguring GitLab
You might encounter the following error when reconfiguring GitLab:
```plaintext
RuntimeError: redis_service[redis] (redis::enable line 19) had an error: RuntimeError: ruby_block[warn pending redis restart] (redis::enable line 77) had an error: RuntimeError: Execution of the command /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket INFO failed with a non-zero exit code (1)
```
The error message indicates that Redis might have restarted or shut down while trying to establish a connection with `redis-cli`. Given that recipe runs
`gitlab-ctl restart redis` and tries to check the version right away, there might be a race condition that causes the error.
To resolve this problem, run the following command:
```shell
sudo gitlab-ctl reconfigure
```
If that fails, check the output of `gitlab-ctl tail redis` and try to run `redis-cli`.
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Upgrading from a non-Linux package installation to a Linux package installation
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
Upgrading from non-Linux package installations has not been tested by GitLab.
Please be advised that you lose your settings in files such as `gitlab.yml`,
`puma.rb` and `smtp_settings.rb`. You must
[configure those settings in `/etc/gitlab/gitlab.rb`](../index.md#configuring).
Before starting the migration, ensure that you are moving to **exactly the same version** of GitLab.
To convert your installation to using the Linux package:
1. Create a backup from your current installation:
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
```
1. [Install GitLab using a Linux package](https://about.gitlab.com/install/).
1. Copy the backup file to the directory `/var/opt/gitlab/backups/` of the new server.
1. Restore the backup in the new installation ([detailed instructions](https://docs.gitlab.com/ee/administration/backup_restore/restore_gitlab.html#restore-for-linux-package-installations)):
```shell
# This command will overwrite the contents of your GitLab database!
sudo gitlab-backup restore BACKUP=<FILE_NAME>
```
The restore takes a few minutes depending on the size of you database and Git data.
1. Configure the new installation because in Linux package installations, all settings are stored in
`/etc/gitlab/gitlab.rb`. Individual settings need to be manually moved from
files such as `gitlab.yml`, `puma.rb` and `smtp_settings.rb`. See the
[`gitlab.rb` template](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template)
for all available options.
1. To finalize the configuration process, copy the secrets from the old installation
to the new one. GitLab uses secrets to multiple purposes, like database encryption,
session encryption, and so on. In Linux package installations, all secrets are placed in a single
file `/etc/gitlab/gitlab-secrets.json`, whereas in source installations, the
secrets are placed in multiple files:
1. First, you need to restore secrets related to Rails. Copy the values of
`db_key_base`, `secret_key_base` and `otp_key_base` from
`/home/git/gitlab/config/secrets.yml` (self-compiled installation) to the equivalent
ones in `/etc/gitlab/gitlab-secrets.json` (Linux package installation).
1. Then, copy the contents of `/home/git/gitlab-shell/.gitlab_shell_secret`
(GitLab source) to GitLab Shell's `secret_token` in
`/etc/gitlab/gitlab-secrets.json` (Linux package installation). It will look something like:
```json
{
"gitlab_workhorse": {
"secret_token": "..."
},
"gitlab_shell": {
"secret_token": "..."
},
"gitlab_rails": {
"secret_key_base": "...",
"db_key_base": "...",
"otp_key_base": "...",
}
...
}
```
1. Reconfigure GitLab to apply the changes:
```shell
sudo gitlab-ctl reconfigure
```
1. If you migrated `/home/git/gitlab-shell/.gitlab_shell_secret`, you also [need to restart Gitaly](https://gitlab.com/gitlab-org/gitaly/-/issues/3837):
```shell
sudo gitlab-ctl restart gitaly
```
## Upgrading from a non-Linux package PostgreSQL to a Linux package installation using a backup
Upgrade by [creating a backup from the non-Linux package installation](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html)
and [restoring this in the Linux package installation](https://docs.gitlab.com/ee/administration/backup_restore/restore_gitlab.html#restore-for-linux-package-installations).
Ensure you are using **exactly equal versions** of GitLab (for example 6.7.3)
when you do this. You might have to upgrade your non-Linux package installation before
creating the backup to achieve this.
After upgrading make sure that you run the check task:
```shell
sudo gitlab-rake gitlab:check
```
If you receive an error similar to `No such file or directory @ realpath_rec - /home/git`,
run this one liner to fix the Git hooks path:
```shell
find . -lname /home/git/gitlab-shell/hooks -exec sh -c 'ln -snf /opt/gitlab/embedded/service/gitlab-shell/hooks $0' {} \;
```
This assumes that `gitlab-shell` is located in `/home/git`.
## Upgrading from a non-Linux package PostgreSQL to a Linux package installation in-place
It is also possible to upgrade a self-compiled installation to a Linux package installation
in-place. Below we assume you are using PostgreSQL on Ubuntu, and that you
have an Linux package matching your current GitLab version. We also
assume that your source installation of GitLab uses all the default paths and
users.
First, stop and disable GitLab, Redis and NGINX.
```shell
# Ubuntu
sudo service gitlab stop
sudo update-rc.d gitlab disable
sudo service nginx stop
sudo update-rc.d nginx disable
sudo service redis-server stop
sudo update-rc.d redis-server disable
```
If you are using a configuration management system to manage GitLab on your
server, remember to also disable GitLab and its related services there. Also
note that in the following steps, the existing home directory of the Git user
(`/home/git`) will be changed to `/var/opt/gitlab`.
Next, create a `gitlab.rb` file for your new setup:
```shell
sudo mkdir /etc/gitlab
sudo tee -a /etc/gitlab/gitlab.rb <<'EOF'
# Use your own GitLab URL here
external_url 'http://gitlab.example.com'
# We assume your repositories are in /home/git/repositories (default for source installs)
git_data_dirs({ 'default' => { 'path' => '/home/git' } })
# Re-use the PostgreSQL that is already running on your system
postgresql['enable'] = false
# This db_host setting is for Debian PostgreSQL packages
gitlab_rails['db_host'] = '/var/run/postgresql/'
gitlab_rails['db_port'] = 5432
# We assume you called the GitLab DB user 'git'
gitlab_rails['db_username'] = 'git'
EOF
```
Now install the Linux package and reconfigure the installation:
```shell
sudo gitlab-ctl reconfigure
```
You are not done yet! The `gitlab-ctl reconfigure` run has changed the home
directory of the Git user, so OpenSSH can no longer find its authorized_keys
file. Rebuild the keys file with the following command:
```shell
sudo gitlab-rake gitlab:shell:setup
```
You should now have HTTP and SSH access to your GitLab server with the
repositories and users that were there before.
If you can log into the GitLab web interface, the next step is to reboot your
server to make sure none of the old services interfere with the Linux package installation.
If you are using special features such as LDAP you will have to put your
settings in `gitlab.rb`, see the [settings documentation](../settings/index.md).
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
remove_date: '2024-08-21'
redirect_to: 'https://docs.gitlab.com/ee/update/package/index.html#version-specific-changes'
---
This document was moved to [another location](https://docs.gitlab.com/ee/update/package/index.html#version-specific-changes).
<!-- This redirect file can be deleted after 2024-08-21. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
remove_date: '2024-08-21'
redirect_to: 'https://docs.gitlab.com/ee/update/package/index.html#version-specific-changes'
---
This document was moved to [another location](https://docs.gitlab.com/ee/update/package/index.html#version-specific-changes).
<!-- This redirect file can be deleted after 2024-08-21. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
remove_date: '2024-08-21'
redirect_to: 'https://docs.gitlab.com/ee/update/package/index.html#version-specific-changes'
---
This document was moved to [another location](https://docs.gitlab.com/ee/update/package/index.html#version-specific-changes).
<!-- This redirect file can be deleted after 2024-08-21. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
remove_date: '2024-08-21'
redirect_to: 'https://docs.gitlab.com/ee/update/package/index.html#version-specific-changes'
---
This document was moved to [another location](https://docs.gitlab.com/ee/update/package/index.html#version-specific-changes).
<!-- This redirect file can be deleted after 2024-08-21. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
remove_date: '2024-08-21'
redirect_to: 'https://docs.gitlab.com/ee/update/versions/gitlab_14_changes.html'
---
This document was moved to [another location](https://docs.gitlab.com/ee/update/versions/gitlab_14_changes.html).
<!-- This redirect file can be deleted after 2024-08-21. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
remove_date: '2024-08-21'
redirect_to: 'https://docs.gitlab.com/ee/update/versions/gitlab_15_changes.html'
---
This document was moved to [another location](https://docs.gitlab.com/ee/update/versions/gitlab_15_changes.html).
<!-- This redirect file can be deleted after 2024-08-21. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
remove_date: '2024-08-21'
redirect_to: 'https://docs.gitlab.com/ee/update/versions/gitlab_16_changes.html'
---
This document was moved to [another location](https://docs.gitlab.com/ee/update/versions/gitlab_16_changes.html).
<!-- This redirect file can be deleted after 2023-11-08. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Cryptographic details related to `omnibus-gitlab` packages
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
GitLab uses a [packagecloud](https://packages.gitlab.com) instance to share the
different OS packages it offers, and uses various cryptographic methods to
ensure the integrity of these packages. This page serves to provide details
regarding these methods.
## Package repository metadata signing keys
The apt and yum repositories on the GitLab packagecloud instance uses a GPG key to
sign their metadata. This key is automatically installed by the repo setup
script specified in the installation instructions.
### Current key
| Key Attribute | Value |
| ------------- | ---------------------------------------------------- |
| Name | `GitLab B.V.` |
| EMail | `packages@gitlab.com` |
| Comment | `package repository signing key` |
| Fingerprint | `F640 3F65 44A3 8863 DAA0 B6E0 3F01 618A 5131 2F3F` |
| Expiry | `2026-02-27` |
This key is active from **2020-04-06**. Existing users who already have
configured the GitLab apt/yum package repositories will have to fetch and add this
key to their trusted keyring again to continue installing packages from those
repositories without apt/yum complaining about mismatches, which is described
below.
This key's expiration was extended from **2024-03-01** to **2026-02-27**.
If you encounter a complaint of expiration on `2024-03-01`, perform the steps
in [Update keys after expiry extension](#update-keys-after-expiry-extension)
to incorporate the updated public key content.
#### Update keys after expiry extension
For Debian based distributions:
PackageCloud generally made use of `apt-key`, which will be deprecated in the future. Manually installed
or configured repositories from some distributions, such as [TurnKey Linux](https://www.turnkeylinux.org/), are
already using the `signed-by` support within Debian package source lists.
1. Determine if you're using `apt-key` or `signed-by` functionality:
```shell
grep 'deb \[signed-by=' /etc/apt/sources.list.d/gitlab_gitlab-?e.list
```
- If this `grep` returns any lines, you're using `signed-by` functionality. This takes
precedence over any `apt-key` usage.
- If this `grep` returns no lines, you're using `apt-key` functionality.
1. For `signed-by`, the following script (run as root) updates the public keys for GitLab repositories:
```shell
awk '/deb \[signed-by=/{
pubkey = $2;
sub(/\[signed-by=/, "", pubkey);
sub(/\]$/, "", pubkey);
print pubkey
}' /etc/apt/sources.list.d/gitlab_gitlab-?e.list | \
while read line; do
curl -s "https://packages.gitlab.com/gpg.key" | gpg --dearmor > $line
done
```
1. For `apt-key`, the following script (run as root) updates the public keys for GitLab repositories:
```shell
apt-key del 3F01618A51312F3F
curl -s "https://packages.gitlab.com/gpg.key" | apt-key add -
apt-key list 3F01618A51312F3F
```
For RPM based distributions:
There are mild differences between Yum and Dnf, but the underlying configuration is identical.
1. Remove any existing key from the repository keyrings:
```shell
for pubring in /var/cache/dnf/*gitlab*/pubring
do
gpg --homedir $pubring --delete-key F6403F6544A38863DAA0B6E03F01618A51312F3F
done
```
1. Update the repository data/cache, which asks you to confirm keys:
```shell
dnf check-update
```
#### Fetching new keys before 2020-04-06
```shell
# Download the new key
curl "https://gitlab-org.gitlab.io/omnibus-gitlab/gitlab_new_gpg.key" -o /tmp/omnibus_gitlab_gpg.key
# Import the key
## Debian/Ubuntu/Raspbian
sudo apt-key add /tmp/omnibus_gitlab_gpg.key
# CentOS/OpenSUSE/SLES
sudo rpm --import /tmp/omnibus_gitlab_gpg.key
```
#### Fetching new keys after 2020-04-06
To fetch the latest repository signing key, users can run the `curl` command
used to add GitLab repository, as mentioned in the [install page](https://about.gitlab.com/install/),
again. It will fetch the new key and add it to the user's keyring.
Or, users can manually fetch and add the new key using the following commands:
```shell
# Download the new key
curl "https://packages.gitlab.com/gpg.key" -o /tmp/omnibus_gitlab_gpg.key
# Import the key
## Debian/Ubuntu/Raspbian
sudo apt-key add /tmp/omnibus_gitlab_gpg.key
# CentOS/OpenSUSE/SLES
sudo rpm --import /tmp/omnibus_gitlab_gpg.key
```
NOTE:
Make sure that the new key has the necessary permissions to be properly recognized by your OS, which should be `644`. You can set the permissions by running
the following command: `chmod 644 <keyfile>`.
### Previous keys
| Sl. No. | Key ID | Expiry Date |
| ------- | ---------------------------------------------------- | ------------ |
| 1 | `1A4C 919D B987 D435 9396 38B9 1421 9A96 E15E 78F4` | `2020-04-15` |
## Package Signatures
This document will provide methods for verifying the signatures of GitLab produced
packages, both manually and automatically where supported.
### RPM based distributions
The RPM format contains a full implementation of GPG signing functionality, and
thus is fully integrated with the package management systems based upon that
format. There are two methods of verification.
#### Verify GitLab public key is present
To verify a package on an RPM based distribution, we'll need to ensure
that the GitLab, Inc. public key is present in the `rpm` tool's keychain.
```shell
rpm -q gpg-pubkey-f27eab47-60d4a67e --qf '%{name}-%{version}-%{release} --> %{summary}'
```
This will produce either the information on
the public key, or `gpg-pubkey-f27eab47-60d4a67e is not installed`. If the key is
not present, perform the following steps:
```shell
rpm --import https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey/gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg
```
#### Verify if signature check is active
The simplest method of checking if package signature checking is active on an existing install is to compare the content of the repository file in use.
- Check if the repository file exist: `file /etc/yum.repos.d/gitlab_gitlab-ce.repo`
- Check that signature checking is active: `grep gpgcheck /etc/yum.repos.d/gitlab_gitlab-ce.repo` should output
```plaintext
repo_gpgcheck=1
gpgcheck=1
```
or
```plaintext
repo_gpgcheck=1
pkg_gpgcheck=1
```
If the file does not exist, you don't have the repository installed. If the file exists, but the output shows `gpgpcheck=0`, then you will need to edit that value to enable it, as below.
#### Enable Automatic Verification
The `rpm` tool and related package managers (`yum`,`zypper`) directly support the automatic verification of packages without intervention. If you used the automated repository configuration script after signed packages became available, then you will have no additional steps required. If you installed prior to the release of signed packages, you can either make the necessary changes, or re-run the automatic repository configuration script as found on the [Installation](https://about.gitlab.com/install/) page.
##### Yum (RedHat, CentOS)
1. Enable GPG checking of the packages
```shell
sudo sed -i'' 's/^gpgcheck=0/gpgcheck=1/' /etc/yum.repos.d/gitlab_gitlab-ce.repo
```
1. Add the package signing public key to the `gpgkey` list:
Edit `/etc/yum.repos.d/gitlab_gitlab-ce.repo`, changing `gpgkey` to read:
```plaintext
gpgkey=https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey
https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey/gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg
```
1. Tell `yum` to refresh the cache for the repository
```shell
sudo yum -q makecache -y --disablerepo='*' --enablerepo='gitlab_gitlab-ce'
```
##### Zypper (SuSE/SLES)
1. Enable GPG checking of the packages
```shell
sudo sed -i'' 's/pkg_gpgcheck=0/pkg_gpgcheck=1/' /etc/zypp/repos.d/gitlab_gitlab-ce.repo
```
1. Add the package signing public key to the `gpgkey` list:
Edit `/etc/zypp/repos.d/gitlab_gitlab-ce.repo`, changing `gpgkey` to read:
```plaintext
gpgkey=https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey
https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey/gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg
```
1. Tell `zypper` to refresh the repository and import the keys
```shell
sudo zypper --gpg-auto-import-keys refresh gitlab_gitlab-ce
```
#### Manual Verification
Once the public key is confirmed present, an RPM package can be manually verified with `rpm --checksig gitlab-xxx.rpm`.
### DEB based distributions
The DEB format does not officially contain a default and included method for signing packages. At GitLab, we have chosen to implement the standard for `debsig` which is well documented, while not enabled by default on most distributions.
#### Manual Verification
Manual verification of DEB packages signed with `debsigs` can be performed in two ways: using `debsig-verify` after configuring the necessary `debsigs` policy and keyring, or manually checking the contained `_gpgorigin` file with GnuPG.
##### Manually verify with GnuPG
The `debsig-verify` package has a [slew of dependencies](https://packages.debian.org/sid/devel/debsig-verify) that a user may not wish to install. To verify the `debsigs` based signature without installing `debsig-verify` and dependencies, a user can complete the following manual steps:
1. Download and import the package signing public key
```shell
curl -JLO "https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey/gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg"
gpg --import gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg
```
1. Extract the signature file (`_gpgorigin`)
```shell
ar x gitlab-ce-xxx.deb _gpgorigin
```
1. Verify the signature matches the content
```shell
ar p gitlab-xxx.deb debian-binary control.tar.gz data.tar.gz | gpg --verify _gpgorigin -
```
The output of the final command should appear as such:
```shell
$ ar p gitlab-xxx.deb debian-binary control.tar.gz data.tar.gz | gpg --verify _gpgorigin -
gpg: Signature made Tue Aug 01 22:21:11 2017 UTC
gpg: using RSA key DBEF89774DDB9EB37D9FC3A03CFCF9BAF27EAB47
gpg: issuer "support@gitlab.com"
gpg: Good signature from "GitLab, Inc. <support@gitlab.com>" [unknown]
Primary key fingerprint: DBEF 8977 4DDB 9EB3 7D9F C3A0 3CFC F9BA F27E AB47
```
##### Configuring debsigs
Configuring a policy and keyring for `debsigs` can be complicated, so GitLab provides `gitlab-debsigs.sh` as a scripted method of configuration.
To use this script, you will need to download the public key and the script.
```shell
curl -JLO "https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey/gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg"
curl -JLO "https://gitlab.com/gitlab-org/omnibus-gitlab/raw/master/scripts/gitlab-debsigs.sh"
chmod +x gitlab-debsigs.sh
sudo ./gitlab-debsigs.sh gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg
```
##### Verify with `debsig-verify`
To make use of `debsig-verify`, perform the steps in [Configuring debsigs](#configuring-debsigs) and install the `debsig-verify` package.
`debsig-verify gitlab-xxx.deb`
FROM ubuntu:22.04
MAINTAINER GitLab Inc. <support@gitlab.com>
SHELL ["/bin/sh", "-c"]
# Default to supporting utf-8
ENV LANG=C.UTF-8
# Explicitly set supported locales
COPY locale.gen /etc/locale.gen
# Install required packages
# Note: libatomic1 is only required for arm64, but it is small enough to not
# bother about the conditional inclusion logic
RUN apt-get update -q \
&& DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
busybox \
ca-certificates \
locales \
openssh-server \
tzdata \
wget \
perl \
libperl5.34 \
libatomic1 \
&& locale-gen \
&& cp -a /usr/lib/locale/locale-archive /tmp/locale-archive \
&& DEBIAN_FRONTEND=noninteractive apt-get purge -yq locales \
&& mv /tmp/locale-archive /usr/lib/locale/locale-archive \
&& rm -rf /var/lib/apt/lists/*
# Use BusyBox
ENV EDITOR /bin/vi
RUN busybox --install \
&& { \
echo '#!/bin/sh'; \
echo '/bin/vi "$@"'; \
} > /usr/local/bin/busybox-editor \
&& chmod +x /usr/local/bin/busybox-editor \
&& update-alternatives --install /usr/bin/editor editor /usr/local/bin/busybox-editor 1
# Remove MOTD
RUN rm -rf /etc/update-motd.d /etc/motd /etc/motd.dynamic
RUN ln -fs /dev/null /run/motd.dynamic
# Legacy code to be removed on 17.0. See: https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/7035
ENV GITLAB_ALLOW_SHA1_RSA=false
ARG TARGETARCH
# Copy assets
COPY RELEASE /
COPY assets/ /assets/
# as gitlab-ci checks out with mode 666 we need to set permissions of the files we copied into the
# container to a secure value. Issue #5956
RUN chmod -R og-w /assets RELEASE ; \
/assets/setup
# Allow to access embedded tools
ENV PATH /opt/gitlab/embedded/bin:/opt/gitlab/bin:/assets:$PATH
# Resolve error: TERM environment variable not set.
ENV TERM xterm
# Expose web & ssh
EXPOSE 443 80 22
# Define data volumes
VOLUME ["/etc/gitlab", "/var/opt/gitlab", "/var/log/gitlab"]
# Wrapper to handle signal, trigger runit and reconfigure GitLab
CMD ["/assets/wrapper"]
HEALTHCHECK --interval=60s --timeout=30s --retries=5 \
CMD /opt/gitlab/bin/gitlab-healthcheck --fail --max-time 10
The latest Docker guide can be found here: [GitLab Docker images](https://docs.gitlab.com/ee/install/docker.html).
#!/bin/bash
echo "Downloading package as artifact - ${DOWNLOAD_URL}"
wget --quiet --header "JOB-TOKEN: ${CI_JOB_TOKEN}" ${DOWNLOAD_URL} -O /tmp/gitlab.deb
results=$?
if [ ${results} -ne 0 ]; then
>&2 echo "There was an error downloading ${DOWNLOAD_URL}. Please check the output for more information"
exit ${results}
fi
# Initialize the env config for each service in case we need to add them in via the wrapper
unless ENV['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'].nil?
[gitlab_workhorse['env'], gitlab_pages['env'], registry['env'], gitlab_rails['env']].each do |new_env|
new_env['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'] = ENV['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI']
new_env['AWS_DEFAULT_REGION'] = ENV['AWS_DEFAULT_REGION']
new_env['AWS_REGION'] = ENV['AWS_REGION']
new_env['ECS_CONTAINER_METADATA_URI_V4'] = ENV['ECS_CONTAINER_METADATA_URI_V4'] unless ENV['ECS_CONTAINER_METADATA_URI_V4'].nil?
new_env['ECS_CONTAINER_METADATA_URI'] = ENV['ECS_CONTAINER_METADATA_URI']
end
end
# Docker options
## Prevent Postgres from trying to allocate 25% of total memory
postgresql['shared_buffers'] = '1MB'
# Disable Prometheus node_exporter inside Docker.
node_exporter['enable'] = false
# Manage accounts with docker
manage_accounts['enable'] = false
# Get hostname from shell
host = `hostname`.strip
external_url "http://#{host}"
# Explicitly disable init detection since we are running on a container
package['detect_init'] = false
# Explicitly disable attempt to update kernel parameters
package['modify_kernel_parameters'] = false
# Load custom config from environment variable: GITLAB_OMNIBUS_CONFIG
# Disabling the cop since rubocop considers using eval to be security risk but
# we don't have an easy way out, atleast yet.
eval ENV["GITLAB_OMNIBUS_CONFIG"].to_s # rubocop:disable Security/Eval
# Load configuration stored in /etc/gitlab/gitlab.rb
from_file("/etc/gitlab/gitlab.rb")
#!/bin/bash
set -e
source /etc/lsb-release
source /RELEASE
# Remove sensitive content from RELEASE file. We can't remove the file because
# we are using it in assets/wrapper.
sed -i "/DOWNLOAD_URL/d;/CI_JOB_TOKEN/d;" /RELEASE
# Install GitLab
if [[ "${TARGETARCH}" == "amd64" ]]; then
export DOWNLOAD_URL=${DOWNLOAD_URL_amd64}
elif [[ "${TARGETARCH}" == "arm64" ]]; then
export DOWNLOAD_URL=${DOWNLOAD_URL_arm64}
else
echo "Unknown TARGETARCH: DOWNLOAD_URL not set"
fi
DOWNLOAD_URL=${DOWNLOAD_URL} CI_JOB_TOKEN=${CI_JOB_TOKEN} /assets/download-package && dpkg -i /tmp/gitlab.deb
rm -rf /tmp/gitlab.deb /var/lib/apt/lists/*
unset DOWNLOAD_URL_amd64
unset DOWNLOAD_URL_arm64
unset DOWNLOAD_URL
unset CI_JOB_TOKEN
# Create sshd daemon
mkdir -p /opt/gitlab/sv/sshd/supervise /opt/gitlab/sv/sshd/log/supervise
mkfifo /opt/gitlab/sv/sshd/supervise/ok /opt/gitlab/sv/sshd/log/supervise/ok
printf "#!/bin/sh\nexec 2>&1\numask 077\nexec /usr/sbin/sshd -D -f /assets/sshd_config -e" > /opt/gitlab/sv/sshd/run
printf "#!/bin/sh\nexec svlogd -tt /var/log/gitlab/sshd" > /opt/gitlab/sv/sshd/log/run
chmod a+x /opt/gitlab/sv/sshd/run /opt/gitlab/sv/sshd/log/run
# Remove current gitlab.rb file
rm -f /etc/gitlab/gitlab.rb
# Patch omnibus package
sed -i "s/external_url 'GENERATED_EXTERNAL_URL'/# external_url 'GENERATED_EXTERNAL_URL'/" /opt/gitlab/etc/gitlab.rb.template
sed -i "s/\/etc\/gitlab\/gitlab.rb/\/assets\/gitlab.rb/" /opt/gitlab/embedded/cookbooks/gitlab/recipes/show_config.rb
sed -i "s/\/etc\/gitlab\/gitlab.rb/\/assets\/gitlab.rb/" /opt/gitlab/embedded/cookbooks/gitlab/recipes/config.rb
# Set install type to docker
echo 'gitlab-docker' > /opt/gitlab/embedded/service/gitlab-rails/INSTALLATION_TYPE
# Create groups
groupadd -g 998 git
groupadd -g 999 gitlab-www
groupadd -g 997 gitlab-redis
groupadd -g 996 gitlab-psql
groupadd -g 994 mattermost
groupadd -g 993 registry
groupadd -g 992 gitlab-prometheus
groupadd -g 991 gitlab-consul
groupadd -g 990 gitlab-backup
# Create accounts
## The git account is created with * as crypted password as ssh treats the account as locked if it has a !
## Issue #5891 https://gitlab.com/gitlab-org/omnibus-gitlab
useradd -m -u 998 -g git -p '*' -m -s /bin/sh -d /var/opt/gitlab git
useradd -m -u 999 -g gitlab-www -m -s /bin/false -d /var/opt/gitlab/nginx gitlab-www
useradd -m -u 997 -g gitlab-redis -m -s /bin/false -d /var/opt/gitlab/redis gitlab-redis
useradd -m -u 996 -g gitlab-psql -m -s /bin/sh -d /var/opt/gitlab/postgresql gitlab-psql
useradd -m -u 994 -g mattermost -m -s /bin/sh -d /var/opt/gitlab/mattermost mattermost
useradd -m -u 993 -g registry -m -s /bin/sh -d /var/opt/gitlab/registry registry
useradd -m -u 992 -g gitlab-prometheus -m -s /bin/sh -d /var/opt/gitlab/prometheus gitlab-prometheus
useradd -m -u 991 -g gitlab-consul -m -s /bin/sh -d /var/opt/gitlab/consul gitlab-consul
useradd -m -u 990 -g gitlab-backup -m -s /bin/sh -d /var/opt/gitlab/backups gitlab-backup
# The gitlab-backup user needs access to these groups for backup/restore purposes
usermod -a -G gitlab-psql,registry,git gitlab-backup
Port 22
ChallengeResponseAuthentication no
HostKey /etc/gitlab/ssh_host_rsa_key
HostKey /etc/gitlab/ssh_host_ecdsa_key
HostKey /etc/gitlab/ssh_host_ed25519_key
Protocol 2
PermitRootLogin no
PasswordAuthentication no
MaxStartups 100:30:200
AllowUsers git
PrintMotd no
PrintLastLog no
PubkeyAuthentication yes
AuthorizedKeysFile %h/.ssh/authorized_keys /gitlab-data/ssh/authorized_keys
AuthorizedKeysCommand /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-authorized-keys-check git %u %k
AuthorizedKeysCommandUser git
# With "UsePAM yes" the "!" is seen as a password disabled account and not fully locked so ssh public key login works
# Please make sure that the account is created without passwordlogin ("*" in /etc/shadow) or configure pam.
# Issue #5891 https://gitlab.com/gitlab-org/omnibus-gitlab
UsePAM no
# Disabling use DNS in ssh since it tends to slow connecting
UseDNS no
# Enable the use of Git protocol v2
AcceptEnv GIT_PROTOCOL
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment