Skip to content

GitLab

Commit edb06ab2 authored by 徐豪's avatar 徐豪
Browse files

init

parents
Hide whitespace changes
Inline Side-by-side
Showing with 2737 additions and 0 deletions

Too many changes to show.

To preserve performance only 532 of 532+ files are displayed.
Plain diff Email patch
doc/development/gitlab-ctl-commands.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Adding new `gitlab-ctl` commands
New `gitlab-ctl` commands can be added to two directory : `files/gitlab-ctl-commands` and `files/gitlab-ctl-commands-ee`. Chef stores `gitlab-ctl` commands in `/opt/gitlab/embedded/service/omnibus-ctl`.
There are two main functions you will use to add commands (<https://github.com/chef/omnibus-ctl>):
## `add_command_under_category(string, string, string, int, ruby_block)`
This method will add a new command to your `gitlab-ctl` under a category, useful for grouping similar commands together logically in help output.
Input arguments:
1. Name of the command.
1. Category of the command. It should be string consisting of only characters and "-". If the category does not exist, it will be added. Default categories are "general" and "service-management" (if the latter is enabled).
1. Description. This will be outputted below the command name when the help command is run.
1. Arity. Always set to `2`, due to [a bug in `omnibus-ctl`](https://github.com/chef/omnibus-ctl#add_command_under_categorystring-string-string-int-ruby_block).
1. Ruby block. Ruby code to be executed when your command is run (arguments to that command will be passed into the block).
## `add_command(string, string, int, ruby_block)`
This method will add a new command to your `gitlab-ctl` without a category. It will be displayed above all categories when the help command is called.
Input arguments are the same as `add_command_under_category` except 2 doesn't exist.
doc/development/index.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Contribute to Omnibus GitLab development
## Getting started
1. [Understanding the Omnibus GitLab Architecture](../architecture/index.md)
1. [Setting up a local development environment](setup.md)
## How everyone can contribute to Omnibus GitLab
- [How to make common enhancements](contribute-to-omnibus-gitlab.md#common-enhancement-tasks)
- [How to perform common maintenance tasks](contribute-to-omnibus-gitlab.md#common-maintenance-tasks)
- [Building and testing enhancements](contribute-to-omnibus-gitlab.md#build-and-test-your-enhancement)
- [Submitting an enhancement merge request for review](contribute-to-omnibus-gitlab.md#submit-your-enhancement-for-review)
- [Development for s390x architecture](s390x.md)
## Omnibus GitLab continuous integration
- [CI variables](ci-variables.md)
- [Repositories, branches and CI pipelines](pipelines.md)
- [Release process](../release/index.md)
- [Version format of packages, Docker images and other artifacts](package_version_format.md)
- [Handling broken master pipelines](broken_master.md)
- [Handling vulnerabilities](vulnerabilities.md)
## Technical notes
- [AWS AMis and Marketplace Listing](aws_amis_and_marketplace_listings.md)
- [Behind the scenes of `gitlab-ctl reconfigure`](reconfigure_in_detail.md)
doc/development/maintainership.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Maintainership
## Scoped Maintainership
`omnibus-gitlab` is a comparatively complex project that handles both software
builds and deployment.
The build aspect covers the build toolchain, CI infrastructure, and dependency
management. The deployment aspect covers the installation and configuration of
GitLab at the user site along with upgrade tasks.
Because the two aspects are very different both in nature and the technical
stack involved, trainee-maintainers must spend a large amount of time to gain
both sets of competencies. Scoped maintainership decreases the time required
to onboard a new maintainer through separation of these core responsibilities.
### Build scope
The build scope covers all the parts of the codebase that build the artifacts
used to ship Omnibus GitLab.
This includes:
1. Omnibus project configurations and component software definitions.
1. Patches used in software definitions.
1. Libraries and Rake tasks used for build, release, and other maintenance
activities.
1. CI configuration used for build, release, and other maintenance activities.
1. Infrastructure management required for the above.
Ideally, a build-scoped maintainer of `omnibus-gitlab` should be
well versed in all the above-mentioned topics. Several of these areas
get infrequent updates, therefore it is not fair to expect trainee-maintainers
to work on all of them. The following checklist provides a guideline to evaluate
the progress of a trainee-maintainer in the build scope.
NOTE:
We do not differentiate between the trainee-maintainer as the author
or reviewer in this list because `omnibus-gitlab` is a relatively stable
and mature project. The majority of merge requests follow established
patterns and only need to pass a set of well known tests that prove
the changes work as expected.
1. Author or review merge requests which update any 3 components from the list below that
support high availability. The trainee-maintainer should gain familiarity with complex
deployment scenarios and how to test them.
1. PostgreSQL
1. Patroni
1. PgBouncer
1. Consul
1. Redis
1. Sentinel
1. Author or review a merge request which updates any of our "runtime" environments, preferably
Ruby/Go. This ensures familiarity with the
[Omnibus Builder](https://gitlab.com/gitlab-org/gitlab-omnibus-builder)
project and how it relates to `omnibus-gitlab`.
1. Ruby
1. Go
1. Python
1. Author or review a Mattermost version update merge request. Ensures familiarity with its update
process and Distribution team's communication process with the Mattermost team.
1. Author or review merge requests which update 5 other components.
1. Author or review 3 merge requests which modify CI configuration.
1. Author or review 3 merge requests which refactor build related code.
1. **OPTIONAL**: Author or review 1 merge request which modifies the `omnibus` project. This
is an optional requirement , because updates to `omnibus` itself are
comparatively rare. It is highly recommended that the Maintainers look through
the commits we have [added on top of the upstream tag](https://gitlab.com/gitlab-org/omnibus/-/compare/9.0.19...9.0.19-stable).
### Deploy scope
TBA
### Architectural changes
Architectural changes to `omnibus-gitlab` project, as well as the `omnibus`
project require sign-off from current full maintainers of the project.
doc/development/managing-postgresql-versions.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Managing PostgreSQL versions
The PostgreSQL Global Development Group typically releases [one major version of PostgreSQL each year](https://www.postgresql.org/support/versioning/), usually in the third quarter. Our goal is to balance supporting and adopting new PostgreSQL features with the downtime and administrative costs of having our users upgrade. GitLab aims to support two versions of PostgreSQL at any given time. This means that prior to adding a new version of PostgreSQL, we will remove the oldest version of PostgreSQL that we support, and bump the minimally required PostgreSQL version up by one major version. PostgreSQL removals are only done in major GitLab releases.
## Software definitions
The software definitions are in:
- `config/software/postgresql_old.rb`
- `config/software/postgresql.rb`
- `config/software/postgresql_new.rb`
## Default version
The version that should be installed by default is controlled by using the 'link bin files' step. The software definition with this step will be used on a new installation.
## Upgrading
The [`gitlab-ctl pg-upgrade` command](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-ctl-commands/pg-upgrade.rb) is used to upgrade from `postgresql_old` or `postgresql` to `postgresql_new`. See our [documentation for how to use it](../settings/database.md#upgrade-packaged-postgresql-server)
### Automatic upgrades
The [`gitlab-ctl pg-upgrade` command](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-ctl-commands/pg-upgrade.rb) is executed during every upgrade. To change the default PostgreSQL version used during the upgrade and revert processes, you must adjust the configuration to specify the desired PostgreSQL version that will be automatically upgraded during installation.
## Removing an older version
When it is time to remove an older version, create an epic with issues to track the following:
1. Remove the old version from Omnibus.
1. Remove the old version from the Helm install.
1. To minimize CI costs, remove the older version from test suites (before doing this, check that .com is not using the old version still).
1. Remove references to that version of PostgreSQL in the GitLab user documentation.
1. Start printing deprecation notices in the release post three GitLab versions before a major PostgreSQL version is removed. If the PostgreSQL version is a version that will be removed within three releases, print deprecation notices in the Admin UI and during the GitLab upgrade process, regardless of whether it is an Omnibus-managed PostgreSQL database or an external database.
For removals, perform the following steps:
1. Run `git rm config/software/postgresql_old.rb`
1. Run `git mv config/software/postgresql{,_old}.rb`
1. Edit `config/software/postgresql_old.rb` and change name from `postgresql` to `postgresql_old`
1. Run `git mv config/software/postgresql{_new,}.rb`
1. Edit `config/software/postgresql.rb` and change name from `postgresql_new` to `postgresql`
## Adding a new version
1. Run `git cp config/software/postgresql{,_new}.rb`
1. Edit `config/software/postgresql_new.rb`. Update:
1. `name` to `postgresql_new`
1. `default_version` to the new version
1. `version` to have the new version, and the `sha256`
1. `major_version` if necessary
1. Add the new PostgreSQL version to the full test suite.
1. Run a nightly test of the `gitlab-org/gitlab` repo against the new version of PostgreSQL.
1. Ensure that the package build includes both versions of PostgreSQL.
1. For Helm installs, update the default PostgreSQL chart version if the default is changing.
1. Update user documentation.
### Testing
1. GitLab runs on the new version of PostgreSQL.
1. Test running GitLab on the new PostgreSQL version at the 10k reference architecture scale and check for performance regressions.
Test upgrades and fresh installs for the following environments:
1. Single node.
1. Install with a separate database node managed by Omnibus.
1. HA database cluster with 3 or more database nodes in the cluster.
1. Geo installations with a single node primary and single node secondary (`postgresql` and `geo-postgresql` on the same secondary node).
1. Geo installations with a HA database cluster on the primary.
1. Geo installations with a separate database and a separate tracking database on the secondary.
1. Helm installs.
1. After testing that upgrades to the newest version work, confirm that `revert-pg-upgrade` successfully downgrades to the previously used version, including on a Geo secondary standalone tracking database.
1. If the default PostgreSQL version changes, test GitLab upgrades with external PostgreSQL databases.
1. Back up and restore.
If the default PostgreSQL version is changing:
1. Auto upgrades on a single node install, separate database node, HA cluster.
1. Auto upgrades where an external PostgreSQL database is being used.
1. Geo installs are not auto upgraded.
If the minimally required version is changing:
1. GitLab upgrade errors out if an old version of Omnibus-managed PostgreSQL is still installed.
If the above tests are manual, we risk missing a breaking change that is introduced after the manual tests have been performed. We should automate as many of these tests as possible.
1. The package build includes both versions of PostgreSQL
1. Running `gitlab-ctl pg-upgrade` works
### The case of `libpq`
Some modules, including `pyscopg2`, depend on PostgreSQL client library, i.e. `libpq`. It should be always linked to the
latest bundled version. By using the latest version we rely on backward compatibility of `libpq`.
## Known issues
Geo uses streaming replication, which requires that the entire secondary database be resynced after a major PostgreSQL upgrade. This can cause hours or days of downtime, and as such, we do not recommend auto upgrades for Geo customers. Starting in 12.10, automatic PostgreSQL upgrades are disabled if Geo is detected.
doc/development/new-services.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Adding a new Service to Omnibus GitLab
In order to add a new service to GitLab, you should follow these steps:
1. [Fetch and compile the software during build](#fetch-and-compile-the-software-during-build)
1. [Add a top-level configuration object for the service](#add-a-top-level-configuration-object-for-the-service)
1. [Include the service in the services list](#include-the-service-in-the-services-list)
1. [Create enable and disable recipes for the service](#create-enable-and-disable-recipes-for-the-service)
1. [Determine and document how log rotation will be handled](#determine-and-document-how-log-rotation-will-be-handled)
Optionally another common task is to add [additional configuration parsing](#additional-configuration-parsing-for-the-service)
for the service.
## Fetch and compile the software during build
You need to add a [new Software Definition](new-software-definition.md) for your
service if it is not already included in the project.
## Add a top-level configuration object for the service
The cookbooks and recipes located in `files/gitlab-cookbooks` are what get run during
`gitlab-ctl reconfigure` in instances where the Omnibus GitLab package has been
installed. This is where we will need to add the settings for a new service.
### Define the default attributes
Pick one of the existing cookbooks to configure your service within, or create a
new cookbook if your service warrants its own.
Within the cookbook there should be an `attributes/default.rb` file. This is where
you want to define the [Default Attributes](../architecture/index.md#default-attributes)
for your service. For a service you should define an `enable` option by default.
```ruby
default['gitlab']['best_service']['enable'] = false
default['gitlab']['best_service']['dir'] = '/var/opt/gitlab/best-service'
default['gitlab']['best_service']['log_directory'] = '/var/log/gitlab/best-service'
```
- `default` is how you define basic cookbook attributes.
- `['gitlab']` contains the cookbook name.
- `['best_service']` is the name of your service.
- `enable`, `dir`, and `log_directory` are our configuration settings.
- `/var/opt/gitlab` is where the working directory and configuration files for the services are placed.
- `/var/log/gitlab` is where logs are written to for the GitLab package.
Define all your settings that you want configurable in the package here. Default
them to `nil` if you need to calculate their defaults based on other settings for
now.
#### Naming convention
A service is referred to mainly in three scenarios:
1. Accessing the Chef attributes corresponding to the service
1. Referencing items such as users, groups, and paths corresponding to the
service
1. Passing the service name to methods which look up on service properties
similar to the following examples:
- "Is the service enabled?"
- "Get the log ownership details corresponding to this service"
- "Generate runit configuration for this service"
For the first case mentioned above, we use underscores to differentiate words in
the service name. For the other two cases, we use hyphens to differentiate words
in the service name. Since the configuration is mainly used as a Ruby object,
using underscores instead of hyphens is more flexible (for example, underscores
make it cleaner to use symbols in configuration hashes).
For example, if we take GitLab Pages, the attributes are available as
`Gitlab['gitlab_pages']` and `node['gitlab_pages']` while the default
directories and paths might look like `/var/log/gitlab/gitlab-pages` and
`/var/opt/gitlab/gitlab-pages`. Similarly, method calls will look like
`service_enabled?("gitlab-pages")`.
### Create a configuration Mash for your service
In order for user to be able to configure your service from `/etc/gitlab/gitlab.rb`
you will need to add a top level Mash for the service.
In `files/gitlab-cookbooks/package/libraries/config/gitlab.rb` you will find the list of
`attribute` methods.
If your service exists within the attributes for the GitLab cookbook, you should
add it as an attribute within the `attribute_block('gitlab')` block. Otherwise,
if your service has its own cookbook, add it above.
```ruby
attribute('best_service')
```
For an EE only attribute, use `ee_attribute` instead.
```ruby
ee_attribute('best_service')
```
### Add service configuration to the settings template
We maintain a [global configuration template](../architecture/index.md#global-gitlab-configuration-template)
where examples of how to configure the services are available, commented out.
This file becomes the `/etc/gitlab/gitlab.rb` on fresh installs of the package.
Once you want to expose your service's configuration to users for them to change, add it
to this file. `files/gitlab-config-template/gitlab.rb.template`
```ruby
### Best Service configuration
# best_service['enable'] = true
# best_service['dir'] = '/var/opt/gitlab/best-service'
# best_service['log_directory'] = '/var/log/gitlab/best-service'
```
The values provided are not meant to reflect the defaults, but are to make it
easier to uncomment to use the service. If that isn't possible you can use
values clearly meant to be replaced like `YOURSECRET` etc. Or use the default
when it makes the most sense.
## Include the service in the services list
In order to allow the service to be easily enable/disabled within the recipes, it
should be added to the [services list](../architecture/index.md#services)
and given appropriate groups.
In the `files/gitlab-cookbooks/package/libraries/config/services.rb` file, add the
service to the appropriate Config class, Base or EE depending on whether the
service is only for GitLab EE.
```ruby
service 'best_service', groups: ['bestest']
```
Specifying groups makes it easier to disable/enable multiple related services as
once.
If none of the existing groups match with what your service does, and you don't
currently need to enable/disable the service using a group. Don't bother adding
at this time.
Some examples of existing groups you may want to use:
- If the service is enabled in omnibus be default, it should add the `DEFAULT_GROUP` group.
- If the service should really not be disabled in almost any scenario, add the `SYSTEM_GROUP`.
- If the service relies on GitLab Rails having been configured, add the `rails` group.
- If the service is a new Prometheus exporter, add the `prometheus` group.
## Create enable and disable recipes for the service
### Enable recipe
The enable recipe should be created as `files/gitlab-cookbooks/<cookbook-name>/recipes/<service-name>.rb`
if it being added to an existing cookbook. If the service has its own cookbook,
the enable recipe can be created as `files/gitlab-cookbooks/<cookbook-name>/recipes/enable.rb`.
In the recipe you will want to create the working directory in `/var/opt/gitlab`
for your service. You will want to ensure the system user that runs your service
is created. Render any configuration files needed for your service into your working
directory.
Near the end of the recipe you will want to make a call to the runit service definition
to define your recipe. In order for this work you will need to have created
a run file in the cookbooks `templates/default` directory. These file names start
with `sv-` followed by the service name, followed by the runit action name.
A service typically needs a `run`, `log-run`, and `log-config`.
`sv-best-service-log-config.erb`:
```ruby
<%= "s#@svlogd_size" if @svlogd_size %>
<%= "n#@svlogd_num" if @svlogd_num %>
<%= "t#@svlogd_timeout" if @svlogd_timeout %>
<%= "!#@svlogd_filter" if @svlogd_filter %>
<%= "u#@svlogd_udp" if @svlogd_udp %>
<%= "p#@svlogd_prefix" if @svlogd_prefix %>
```
`sv-best-service-log-run.erb`:
```ruby
#!/bin/sh
exec chpst -P \
-U root:<%= @options[:log_group] || 'root' %> \
-u root:<%= @options[:log_group] || 'root' %> \
svlogd -tt <%= @options[:log_directory] %>
```
`sv-best-service-run.erb`:
```ruby
#!/bin/sh
exec 2>&1
<%= render("mount_point_check.erb") %>
cd <%= node['gitlab']['best-service']['dir'] %>
exec chpst -P /opt/gitlab/embedded/bin/best-service -config-flags -etc
```
Depending on what you are running, and which user should run it, your run file
should be configured differently. Look in our other `-run.erb` for examples.
Within your recipe, the runit service should be called and started:
```ruby
runit_service "best-service" do
options({
configItem: 'value',
[...]
log_directory: logging_settings[:log_directory],
log_user: logging_settings[:runit_owner],
log_group: logging_settings[:runit_group],
}.merge(params))
log_options logging_settings[:options]
end
if node['gitlab']['bootstrap']['enable']
execute "/opt/gitlab/bin/gitlab-ctl start best-service" do
retries 20
end
end
```
#### Log Directory
The example settings referenced above that include `logging_settings` make use of
the [`LogfilesHelper`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab/libraries/logfiles_helper.rb)
class in order to provide a consistent reference to the configuration settings
for the service log directory, the log group assigned to the log directory, and
the group used for svlogd execution.
To make use of these settings, please include the `LogfilesHelper` class in your
`enable.rb` for your service, for example:
```ruby
[...]
logfiles_helper = LogfilesHelper.new(node)
logging_settings = logfiles_helper.logging_settings('best-service')
[...]
```
Please add `best-service` to the list of services in the `default_logdir_ownership`
class method with the default user/group that should be used for the log directory
user/group. If you don't have a specific user/group need - default to
`{ username: gitlab_user, group: gitlab_group }`
### Disable recipe
The disable recipe should be created as `files/gitlab-cookbooks/<cookbook-name>/recipes/<service-name>_disable.rb`
if it being added to an existing cookbook. If the service has its own cookbook,
the disable recipe can be created as `files/gitlab-cookbooks/<cookbook-name>/recipes/disable.rb`.
The recipe needs to container any cleanup you want to do when you service is disabled,
and have a call to disable the runit service.
```ruby
runit_service "best-service" do
action :disable
end
```
## Determine and document how log rotation will be handled
In Omnibus, [log rotation](https://docs.gitlab.com/ee/administration/logs/index.html#log-rotation) for
a given service can be handled by `logrotate`, `svlogd`, both or neither. The new service should
be included in the [log rotation](https://docs.gitlab.com/ee/administration/logs/index.html#log-rotation)
table with an indication about what is responsible for managing and rotating the logs for that
service. When adding a service to Omnibus GitLab, you should:
- Ensure that log rotation is in place for the new service.
- Open a merge request to have the new service added to
[the log rotation table](https://docs.gitlab.com/ee/administration/logs/index.html#log-rotation).
If a new log is added that is not using `runit` (`svlogd`), the log must be manually
added to the logrotate configuration. The
[Improve logrotate handling](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6182) issue
has more information.
## Additional configuration parsing for the service
If you want to populate certain configuration options based on other options being set
by the user, we add a library for your service to parse variables.
The library should be added as `files/gitlab-cookbooks/<cookbook name>/libraries/<service-name>.rb`
The library should be a module named after your service that has a `parse_variables` method.
```ruby
module BestService
class << self
def parse_variables
# setup some additional configuration based on the values of the user provided configuration
end
end
end
```
We then need to have the GitLab configuration call your parse_variables method.
Go into `files/gitlab-cookbooks/package/libraries/config/gitlab.rb` and update
your attribute to use the library.
```ruby
attribute('best_service').use { BestService }
```
Note that sequence for parsing variables matters. So if your library expects to
be parsed after another service's library, you need to update your attribute with
a `priority` value that comes later. (The default `priority` value is `20`)
```ruby
attribute('expected_service').use { ExpectedService }
attribute('best_service', sequence: 25).use { BestService }
```
doc/development/new-software-definition.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Adding a new Software Definition to Omnibus GitLab
In order to add a new component to GitLab, you should follow these steps:
1. [Fetch and compile the software during build](#fetch-and-compile-the-software-during-build)
1. [Add a dependency for the software definition to another component](#add-a-dependency-for-the-software-definition-to-another-component)
## Fetch and compile the software during build
[Software Definitions](../architecture/index.md#software-definitions), which
can be found in `/config/software`, specify where omnibus should fetch the
software, how to compile it and install it to the required folder. This part of
the project is run when we build the Omnibus package for GitLab.
When adding a component that should be fetched from Git the clone address of the
repositories of the local mirror and upstream should be added to
`/.custom_sources.yml`.
The local mirror should be created in the [omnibus-mirror project](omnibus-mirror.md) by a member of the Distribution team.
See other Software services in the directory for examples on how to include your
software service.
## Handling Licenses
Most software repositories include a license file. Add the license using a patch
file if it is not explicitly included. Software installed using a package manager
such as `gem` or `pip` should also use this method.
[Create patches](creating-patches.md) for licenses added manually and store them
at a directory path with the naming convention
`config/patches/SOFTWARE_NAME/license/VERSION_NUMBER/add-license-file.patch`.
> Licenses can and do change over the lifetime of a project. This method
> intentionally causes builds to fail reminding contributors to verify manually
> installed licenses. If the license has not changed then `git mv` the `VERSION_NUMBER`
> directory containing the patch file to the new `VERSION_NUMBER`.
## Add a dependency for the software definition to another component
Add a `dependency` statement to the definition of the GitLab project found in
`/config/projects/gitlab.rb`, unless there is a more specific component it makes
sense to be a dependency of (eg `config/software/gitlab-rails.rb` for a
component only needed by `gitlab-rails`)
Omnibus will build dependency components first, and then other ones in the order
of their presence in `/config/projects/gitlab.rb`. So, when a software component
A is marked as a dependency of another software B, A will be built towards the
beginning of the process. In cases where A is a component that changes frequently, cache gets invalidated often causing every subsequent component to be
rebuilt, increasing overall build time. A workaround is to ensure A gets built
immediately before B, avoiding cache invalidation.
1. Add the software A to `/config/projects/gitlab.rb` immediately before
software B
Since A and B are now top-level dependencies of the project, omnibus will
build them in the order of their presence in `/config/projects/gitlab.rb`.
1. In the software definition of B, add a line similar to the following
```ruby
dependency '<name of software A>' unless project.dependencies.include?('<name of software A>')
```
Above ensures that whenever the project is not built from
`/config/projects/gitlab.rb`, A is marked as a dependency of B and is built
before B. There will be no effect on builds from `/config/projects/gitlab.rb`
however.
## Validating changes to a single software dependency
It can be useful to only build one piece of software, rather than rebuild the whole package each time. For instance,
when adding a new software definition. Using this method, you can quickly rebuild an omnibus package containing only
the software and its dependencies. Once you've confirmed the software builds on its own, you can add it to the Omnibus GitLab
build and confirm it there. To use this:
1. [Setup your development environment](setup.md)
1. Copy the [simple.rb](examples/simple.rb) file into your projects
```shell
cp doc/development/examples/simple.rb config/projects/
```
1. Change the `dependency` in `config/projects/simple.rb` to match the software you are testing
1. Build the simple project by running
```shell
bundle exec omnibus build simple
```
doc/development/omnibus-mirror.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# omnibus-mirror
In order to minimize the dependency on external resources during our build and release process, we maintain a mirror of various software dependencies within GitLab resources.
The system consists of two groups, one on GitLab.com, and the other on `dev.gitlab.org`
1. [GitLab.com projects](https://gitlab.com/gitlab-org/build/omnibus-mirror)
1. Pull mirrors of the upstream source, and push mirrors to the corresponding `dev.gitlab.org` project.
1. Publically available. Should be available to the `trigger-package` pipeline, as well as community developers for building custom `omnibus-gitlab` packages
1. `dev.gitlab.org` projects: `https://dev.gitlab.org/omnibus-mirror`
1. Pipelines on `dev.gitlab.org` use these projects to build `omnibus-gitlab` projects. This includes the package releases, as well as builds used by GitLab Team members to build custom `omnibus-gitlab` packages for all supported platforms.
## Adding a project
1. Log into GitLab.com as the `gitlab-omnibus-mirror-bot`. Login details are in the Build vault in 1Password
1. Click on **New Project** to create a project
1. **Project name**: usually should match the upstream name
1. **Project URL**: Select `gitlab-org/build/omnibus-mirror` from the **Groups** sub-group in the drop-down
1. **Visibility Level**: Set this to **Public**
1. Disable any options that would create files, such as initializing with a `README.md`
1. Leave the remaining options as their default, and click on **Create project**
1. Set up the pull mirror
1. Click **Settings -> Repository** in the left hand menu
1. Click **Expand** next to the **Mirroring repositories** option
1. Set **Git repository URL** to the upstream URL we'll be mirroring
1. Change **Mirror direction** to **Pull**
1. Select the **Overwrite diverged branches** option
1. Click **Mirror repository** to add the pull mirror. Depending on the size of the repository, this can take a few minutes to run
1. Leave this page open as you will need it later
1. Log into `dev.gitlab.org` as the `build_mirror_bot`. Login details are in the Build vault in 1Password
1. Click on **New Project** to create a project
1. **Project name**: usually should match the upstream name
1. **Project URL**: Select `omnibus-mirror` from the **Groups** sub-group in the drop-down
1. **Visibility Level**: Set this to **Private**
1. Leave the remaining options as their default and click on **Create project**
1. Enable the `omnibus-builder` deploy key
1. Navigate to **Settings -> Repository**
1. Click on **Expand** next to the **Deploy Keys** section
1. Click the **Privately accessible deploy keys** tab
1. Find the `omnibus-builder deploy key` and click on the **Enable** button next to it
1. Setup the push mirror
1. Return to the **Mirroring repositories** page for the GitLab.com project
1. In **Git repository URL** enter the path to the `dev.gitlab.org` with `build_mirror_bot@` inserted between `https://` and `dev.gitlab.org`
1. In **Password** enter the `PAT` from the **Build Team Mirror Bot** entry in the **Build** vault in 1 Password
1. Disable any options that would create files, such as initializing with a `README.md`
1. Leave the remaining options as their default, and click on **Mirror repository**
1. Click the **Update Now** button to trigger initial mirroring
doc/development/package_version_format.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Version format for the packages and Docker images
## Packages
Packages built by the `omnibus-gitlab` continuous integration pipelines have
a version string in the format `<build_version>-<build_iteration>`. The
pipelines generally produce three types of packages:
- feature branch builds
- nightly builds
- tagged release builds
NOTE:
The `build_iteration` portion of versions strings conveys a specific meaning
and contributes to how `build_version` is computed. The next sections are
written with that in mind as `build_iteration` must be understood first even
though it comes last in the version string.
### Build Iteration
Version strings use `build_iteration` when packaging related logic changes
that do not contain changes to bundled components.
|Release Type|Example Pipelines|`build_iteration` String|
|-|-|-|
|non-tagged|feature branches, nightly builds|`0`|
|tagged|releases|`ce.<OMNIBUS_RELEASE>` or `ee.<OMNIBUS_RELEASE>` |
The edition component, **ce** or **ee**, instructs package managers such as
`apt` or `yum` to treat Enteprise Edition packages as an upgrade from
Community Edition when the package version is the same because **ee** is
sorted lexographically after **ce**.
The `OMNIBUS_RELEASE` component is deprecated and always set to `0`.
Historically, `OMNIBUS_RELEASE` indicated quick bug fixes targeting
`omnibus-gitlab` with no changes in GitLab Rails or other bundled
components. The updates frequently required Semantic Version updates because
of user impact. For this reason, `OMNIBUS_RELEASE` was not useful in
practice and is no longer relevant.
### Build version
The build version component of the version string changes based on whether
the package is a [a feature branch](#regular-feature-branch-builds),
[a nightly build](#nightly-builds), or a [tagged release](#tagged-release-builds).
#### Regular feature branch builds
For regular feature branch builds, the version format is
`<latest stable git tag>+rfbranch.<pipeline id>.<omnibus-gitlab SHA>-<build iteration>`.
As noted above, `build iteration` for regular feature branch builds is set to
`0`. An example version string of this type is
`13.1.1+rfbranch.159743.eb538eaf-0`.
The `+rfbranch` string denotes the package as built from a regular feature
branch build. It also lexographically places it after a stable branch
causing package managers such as `apt` or `yum` to view it as an upgrade
from a stable release package.
#### Nightly builds
For nightly packages , the version format is
`<latest stable git tag>+rnightly.<pipeline id>.<omnibus-gitlab SHA>-<build iteration>`.
As noted above, `build iteration` for nightly builds is set to `0`. An example
version string of this type is `13.1.1+rnightly.159756.b2b5f05e-0`.
The `+rnightly` denotes the package as the output from a nightly build. When
compared alphabetically by package managers, `+rnightly` is considered
greater than both the latest stable and the `+rfbranch` packages. Package
managers will always treat a nightly package as a package upgrade.
#### Tagged release builds
For tagged release builds, while the Git tags are of the format
`<SemVer version>+<build iteration>`, the version strings follow the format of
`<SemVer version>-<build iteration>`.
For example, if the tags are `13.1.0+rc42.ce.0`, `13.1.0+ce.0`, and
`13.1.0+ee.0`, the version strings will be `13.1.0-rc42.ee.0`, `13.1.0-ce.0`,
and `13.1.0-ee.0` respectively.
As noted above, unlike feature branch and nightly builds, the `build iteration`
component of these releases are of the format `(ce|ee).0`
## Docker images
Docker images created by `omnibus-gitlab` CI pipelines are based on the Ubuntu
package built in the previous stage. Hence, the Docker image tag also reflects
the same information given by the package version string. Because `+` symbol
used in package versions is not a supported character for image tags, it is
replaced with a `-` to get a slug.
As a general rule, Docker images will use a slug of the package version as tags.
Also, all Docker images will be pushed to Docker container registries
corresponding to the host where they are being built.
The entire image reference will be of the form
```plaintext
dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-(ce|ee):<slug of package version>
```
For example,
`dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ce:13.2.1-rfbranch.163015.32ed1c58-0`
will be the image tag for a Docker built from a GitLab CE package with version
`13.2.1+rfbranch.163015.32ed1c58-0`
The special cases that deviate from this general rule is listed below.
### Triggered builds in [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror)
Docker images created as part of triggered pipeline (for Package and QA run) has
their tag decided depending on where the trigger originated. If the trigger
originated from an `omnibus-gitlab` pipeline, the image tag is the slug of the
package version. But, if the trigger originated from a GitLab or GitLab-FOSS
pipeline, the image tag will be set to the SHA of the commit corresponding to
that pipeline.
These jobs are not published to Dockerhub repository.
### Nightly builds
In addition to the general naming scheme, Docker images built by scheduled
nightly pipelines gets tagged with the `nightly` tag and both tags are pushed to
the Dockerhub repository also.
`gitlab/gitlab-ee:nightly` and `gitlab/gitlab-ce:nightly` denote image
references for the two editions available.
## Tagged release builds
In addition to the general naming scheme, Docker images built by tagged release
pipelines also gets tagged as `latest` and both tags are pushed to the Dockerhub
repository also.
doc/development/pipelines.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Repositories, branches, and CI pipelines
`omnibus-gitlab` CI pipeline is a moderately complex one with pipelines split
between all the mirrors of the project:
1. [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) for normal feature development.
1. [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) for building release artifacts.
1. [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) for security development.
1. [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) for developers to run a package build and QA for development
purposes.
## Branch and tag protection
### Protected branches
NOTE:
Unless specified otherwise, the listed users/groups have permission to both merge, and push to
protected branches.
1. Development repository
1. `master`: Maintainers, [Delivery team](https://gitlab.com/gitlab-org/delivery)
1. `*-stable` : [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
1. `*-stable-ee` : [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
1. `*-auto-deploy-*` : Maintainers, [`delivery` group](https://gitlab.com/gitlab-org/delivery), [`managers` group](https://gitlab.com/gitlab-org/release/managers)
1. Release mirror:
1. `master`: Maintainers
1. `*-stable` : Maintainers
1. `*-stable-ee` : Maintainers
1. `*-auto-deploy-*` : Maintainers
1. Security mirror:
1. `master`: [GitLab Release Tools Bot](https://gitlab.com/gitlab-release-tools-bot), [GitLab Bot](https://gitlab.com/gitlab-bot), [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
1. `*-stable`: [GitLab Release Tools Bot](https://gitlab.com/gitlab-release-tools-bot), [GitLab Bot](https://gitlab.com/gitlab-bot), [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
1. `*-stable-ee`: [GitLab Release Tools Bot](https://gitlab.com/gitlab-release-tools-bot), [GitLab Bot](https://gitlab.com/gitlab-bot), [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
1. `*-auto-deploy-*`: [GitLab Release Tools Bot](https://gitlab.com/gitlab-release-tools-bot), [GitLab Bot](https://gitlab.com/gitlab-bot), [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
1. QA mirror:
1. `master`: Developers (Merge only), Maintainers
NOTE:
Developers get access to `master` branch in QA mirror because
that's required to run a triggered pipeline against the branch. There is an
[open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24585) to change this
situation.
### Protected Tags
NOTE:
Unless specified otherwise, the listed users/groups have permission to both merge, and push to
protected tags.
1. Development repository:
1. `*` : Maintainers, [Delivery team](https://gitlab.com/gitlab-org/delivery), [Release Managers](https://gitlab.com/gitlab-org/release/managers)
1. Release mirror:
1. `*`: Maintainers
1. Security mirror: Nil
1. QA mirror: Nil
## Mirroring between repositories
Most of the development happens in the [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab)
and security related changes go to [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab).
These changes then gets mirrored to [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab)
and [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror). The
following diagram describes what all gets mirrored between each of these
repositories.
```mermaid
graph TD
A[Development repository] -->|Protected branches| B[Security mirror]
A[Development repository] -->|Protected branches| C[Release mirror]
A[Development repository] -->|All branches| D[QA mirror]
B[Security mirror] -->|Protected branches| C[Release mirror]
B[Security mirror] -->|All branches| D[QA mirror]
```
## Types of pipelines
Since the jobs in `omnibus-gitlab` are spread across multiple mirrors, the CI
config becomes relatively complex when using `rules` keyword for specifying
where/when to run jobs. Hence, a different approach is used for
`omnibus-gitlab`, which marks each necessary pipeline type with a specific
variable named `PIPELINE_TYPE`, and attach jobs to various pipeline types as
needed. The different pipeline types are documented in the below table:
| Pipeline type | Mirror(s) where the pipeline runs | Remarks |
| -------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `DEPENDENCY_SCANNING_PIPELINE` | Canonical | Check for security vulnerabilities in dependencies. Requires `DEPENDENCY_SCANNING` variable to be set to `true`. |
| `DEPS_IO_VERSION_BUMP_PIPELINE` | Canonical | On branch pushes done by `deps.io`. Requires branches to have `deps-` prefix. |
| `DEPS_IO_VERSION_CHECK_PIPELINE` | Canonical | Run `deps` to detect updates. Requires `DEPS_PIPELINE` variable to be set to `true`. |
| `LICENSE_PAGE_UPDATE_PIPELINE` | Canonical | Update license pages. Requires `PAGES_UPDATE` variable to be set to `true`. |
| `CACHE_UPDATE_PIPELINE` | Canonical, QA | Update the gem cache and package build cache. Requires `CACHE_UPDATE` variable to be set to `true`. |
| `DURATION_PLOTTER_PIPELINE` | QA | Package build to plot duration of builds. Requires `DURATION_PLOTTER` variable to be set to `true`. |
| `DOCS_PIPELINE` | Canonical, Security | Requires branch names to either have `docs-` as prefix or `-docs` as suffix. |
| `PROTECTED_TEST_PIPELINE` | Canonical, Security | On protected branches and tags. Does not include trigger jobs and other unwanted jobs like danger-review. |
| `GITLAB_BRANCH_TEST_PIPELINE` | Canonical, Security | On non-protected branch pushes. Does not include unwanted jobs like danger-review. Includes trigger jobs. |
| `GITLAB_MR_PIPELINE` | Canonical, Security | On MRs with branches in Canonical/Security (i.e. MRs from GitLab and team members). Has trigger jobs. |
| `AUTO_DEPLOY_BUILD_PIPELINE` | Release | Builds when auto-deploy tags are pushed. |
| `CE_BRANCH_BUILD_PIPELINE` | Release | Builds when regular branches are pushed. |
| `CE_NIGHTLY_BUILD_PIPELINE` | Release | Nightly CE builds. |
| `CE_RC_BUILD_PIPELINE` | Release | Builds when CE RC tags are pushed. |
| `CE_TAG_BUILD_PIPELINE` | Release | Builds when stable CE tags are pushed. |
| `EE_BRANCH_BUILD_PIPELINE` | Release | Builds when regular branches are pushed, but forced to be an EE pipeline. |
| `EE_NIGHTLY_BUILD_PIPELINE` | Release | Nightly EE builds. Requires pipeline to be forced to be an EE one. |
| `EE_RC_BUILD_PIPELINE` | Release | Builds when EE RC tags are pushed. |
| `EE_TAG_BUILD_PIPELINE` | Release | Builds when stable EE tags are pushed. |
| `TRIGGER_CACHE_UPDATE_PIPELINE` | QA | Updates build cache of triggered QA pipelines. Requires `CACHE_UPDATE` variable to be set to `true`. |
| `TRIGGERED_CE_PIPELINE` | QA | Triggered e2e:package-and-test build with CE packages and images. |
| `TRIGGERED_EE_PIPELINE` | QA | Triggered e2e:package-and-test build with EE packages and images. |
| `FORK_BRANCH_TEST_PIPELINE` | Forks | Test suite when run on forks of the project. Does not include trigger jobs and other unwanted jobs like danger-review. |
| `FORK_MR_PIPELINE` | Forks | On MRs from forks of the project. Does not include trigger jobs. |
### Branch pipelines
`omnibus-gitlab` doesn't make use of detached merge request pipelines yet. So,
pipelines caused by branch pushes are the common pipelines encountered in this
project. These pipelines are used in [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab), [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab),
and [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab).
### Tag pipelines
Tag pushes to [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) starts a pipeline with the
[release jobs](#release-jobs). Tag pushes to [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and
[Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) behaves as regular branch pushes (except that they don't have an option
to start a e2e:package-and-test pipeline) and run basic stylistic checks and unit
tests.
### Scheduled pipelines
In [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab), there are two scheduled pipelines.
1. `Dependencies Update` - Pipeline to check for outdated dependencies using the
[`dependency_update`](#dependency_update) job.
1. `Generate license pages` - Pipeline to populate [License collection webpage](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html)
with license information from the S3 bucket.
In [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab), there are two scheduled pipelines
1. `CE nightly` - Pipeline to build nightly packages and Docker images for
GitLab CE
1. `EE nightly` - Pipeline to build nightly packages and Docker images for
GitLab EE
Other mirrors doesn't have any scheduled pipelines
### Auto-deploy pipelines
GitLab uses [Auto deploy](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/auto-deploy.md) branches and tags for its release process. These
branches are named `<MAJOR>-<MINOR>-auto-deploy-<TIMESTAMP>` and tags have the
format `<MAJOR>.<MINOR>.<TIMESTAMP>+<gitlab sha>.<omnibus-gitlab sha>`.
Only certain jobs - that are needed to deploy the changes to GitLab
environments - will be part of this pipeline.
### Triggered pipelines
We use triggered pipelines to run a "e2e:package-and-test" pipeline in the
[QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror). These can be triggered either from a pipeline in the
[Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) or [GitLab project](https://gitlab.com/gitlab-org/gitlab).
This pipeline is intended to give the developers a package and an image to test
their changes in addition to automatically doing a QA run against these
artifacts. It also provides an option to do a QA run against an HA instance spun
up using these artifacts.
## CI Jobs
### Development Jobs
#### `danger-review`
This job runs a check against the merge request to ensure it satisfies some
basic requirements using the Danger tool.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
and tag pipelines.
#### `rubocop`
Check the source code files to satisfy certain stylistic requirements.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
and tag pipelines.
#### `docs-lint`
Check the docs files to satisfy certain stylistic requirements.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
and tag pipelines.
#### `review-docs-deploy`
A manual job that will trigger a docs build in `gitlab-docs`, which will deploy a
review app for the GitLab Docs with changes from the current commit.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) on branch pipelines.
#### `review-docs-cleanup`
A manual job to stop the environment created by `review-docs-deploy`.
Automatically ran on merging a merge request.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) on branch pipelines.
#### `<OS_NAME> knapsack`
We have tests covering our Chef recipes and libraries using `RSpec` and
`Chefspec`. They are parallelized with the help of
[knapsack](https://github.com/KnapsackPro/knapsack).
This job pulls in the knapsack reports of previous pipeline runs from the cache
and gets them in place for the current spec runs.
These jobs are run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
and tag pipelines.
#### `<OS_NAME> specs`
Jobs that actually run `rspec` via `knapsack`. Parallelized to a count of 6
using the [`parallel` keyword](https://docs.gitlab.com/ee/ci/yaml/#parallel)
These jobs are run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
and tag pipelines.
#### `update-knapsack`
This job combines the reports from each individual parallel runs of unit tests
and prepares a final JSON report. This report is used by the MR widget to
display spec status, and is also uploaded to the cache for the next pipeline run
to use.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch and
tag pipelines.
#### `Trigger:ce-package`
This is a manual job which when played triggers a pipeline in the [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) to
run a package build and QA for development purposes. This can be used by
developers to get a package or Docker image for testing, or to run a full QA
suite against their MR's changes.
To debug QA failures, refer to the [Investigate QA failures](https://handbook.gitlab.com/handbook/engineering/infrastructure/test-platform/self-managed-platform-team/distribution/#investigate-qa-failures)
section. This job also generates Allure report, see more information and the demo at [Test Report Generation in Omnibus-GitLab](allure-test-report.md).
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
pipelines.
#### `Trigger:ee-package`
Same as `Trigger:ce-package`, but builds an EE package.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and [Security mirror](https://gitlab.com/gitlab-org/security/omnibus-gitlab) on branch
pipelines.
#### `fetch-assets`
For building packages, we make use of the Rails assets that are already compiled
by the [GitLab](https://gitlab.com/gitlab-org/gitlab) or
[GitLab-FOSS](https://gitlab.com/gitlab-org/gitlab-foss) pipelines. Those
pipelines push it as a Docker image, which we pull in here and copy the assets
themselves to a predefined location.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on branch, tag and
triggered pipelines.
#### `Trigger:package`
This job builds a single package that will be available as an artifact.
This job is run only on [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on triggered pipelines.
#### `Trigger:gitlab-docker`
This job builds a GitLab Docker image using the package that was built by
`Trigger:package` job.
This job is run only on [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on triggered pipelines.
#### `qa-subset-test`
This job runs subset of test suite and is automatically triggered when either
`Trigger:CE-package` or `Trigger:EE-package` manual jobs are triggered manually.
This job triggers a pipeline in the [GitLab QA Mirror](https://gitlab.com/gitlab-org/gitlab-qa-mirror), passing the GitLab Docker image created by `Trigger:gitlab-docker` job and the GitLab QA Docker image built by the GitLab Rails pipeline, so that a subset of test as mentioned in [this issue](https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1303#we-should-keep) will run using these images
This job is run only on [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on triggered pipelines.
### `qa-remaining-test-manual`
This is a manual trigger job which runs the remaining tests that donot run in the `qa-subset-test` job.
To run this QA jobs in MR pipeline, you need to either trigger `Trigger:CE-package` or `Trigger:EE-package` manual jobs.
This job triggers a pipeline in the [GitLab QA Mirror](https://gitlab.com/gitlab-org/gitlab-qa-mirror), passing the GitLab
Docker image created by `Trigger:gitlab-docker` job and the GitLab QA Docker
image built by the GitLab Rails pipeline, so that the entire suite is run using these images.
#### `RAT`
This manual job triggers a pipeline in the
[RAT](https://gitlab.com/gitlab-org/distribution/reference-architecture-tester)
project passing the URL to the package built by `Trigger:package` job, which
will spin up a PostgreSQL HA instance with that package using
[GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit), and run
QA against that instance.
This job is run only on [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on triggered EE pipelines.
#### `<OS_NAME>-branch`
These jobs build packages for the specified OS, and pushes the resultant package
to an S3 bucket in addition to making them available as artifacts.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on branch and nightly pipelines.
`NOTE`: Raspberry Pi jobs are run only on CE branches while SLES jobs are run
only on EE branches.
#### `Docker-branch`
This job builds a GitLab Docker image using the package built during the Ubuntu
22.04-branch job. The image is pushed to the GitLab container registry.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on branch and nightly pipelines.
#### `QA-branch`
This job builds a GitLab QA Docker image from the
[qa directory in the Rails codebase](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/Dockerfile).
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on branch and nightly pipelines.
## Release jobs
### `<OS_NAME>`
These jobs build packages for the specified OS, and pushes the resultant package
to an S3 bucket in addition to making them available as artifacts.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on tag pipelines.
NOTE:
Raspberry Pi jobs are run only on CE tags while SLES jobs are run only on EE tags.
### `<OS_NAME>-staging`
These jobs upload the packages built in previous job to an internal staging
repository in [Packagecloud instance](https://packages.gitlab.com).
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on tag pipelines.
NOTE:
Raspberry Pi jobs are run only on CE tags while SLES jobs are run only on EE tags.
### `<OS_NAME>-release`
These jobs pulls the packages from the internal staging repository to a public
repository in [Packagecloud instance](https://packages.gitlab.com).
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on tag pipelines.
NOTE:
Raspberry Pi jobs are run only on CE tags while SLES jobs are run only on EE tags.
### `Docker`
This job builds a GitLab Docker image using the package built during the Ubuntu
22.04-branch job. The image is pushed to the internal GitLab container registry.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on tag pipelines.
### `Docker-Release`
This job pulls the GitLab Docker image from internal GitLab container registry
and pushes it to Dockerhub.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on tag pipelines.
### `Docker-QA`
This job builds a GitLab QA Docker image from the
[qa directory in the Rails codebase](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/Dockerfile) and
pushes it to the internal GitLab container registry.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on tag pipelines.
### `QA-Tag`
This job pulls the GitLab QA Docker image from the internal GitLab container
registry and pushes it to Dockerhub.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on tag pipelines.
### `AWS`
This job builds the unlicensed AWS AMIs using the Ubuntu 20.04 package.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on tag pipelines.
### `AWS-Ultimate`
This job builds an AWS AMI with embedded Ultimate licenses using the Ubuntu
20.04 package.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on EE tag pipelines.
### `AWS-Premium`
This job builds an AWS AMI with embedded Premium licenses using the Ubuntu
20.04 package.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on EE tag pipelines.
### `AWS-CE-Release`
This job updates the [GitLab Community Edition listing in AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-w6ykryurkesjq?ref_=srh_res_product_title) with the current version.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on CE tag pipelines.
### `AWS-Ultimate-Release`
This job updates the [GitLab Ultimate listing in AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-g6ktjmpuc33zk?ref_=srh_res_product_title) with the current version.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on EE tag pipelines.
### `AWS-Premium-Release`
This job updates the [GitLab Premium listing in AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-amk6tacbois2k?ref_=srh_res_product_title) with the current version.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on EE tag pipelines.
### `create_omnibus_manifest`
This job creates a `version-manifest.json` file to be used by
`dependency_scanning` job.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on tag and nightly pipelines.
### `dependency_scanning`
This job runs a dependency scan on all the components of the package to check if
any of them are vulnerable to any known vulnerabilities.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) and on tag and nightly pipelines.
### `RAT-Nightly`
This job triggers a pipeline in the
[RAT](https://gitlab.com/gitlab-org/distribution/reference-architecture-tester)
project passing the URL to the nightly Ubuntu 22.04 package built in this pipeline,
which will spin up a PostgreSQL HA instance with that package using
[GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit), and run
QA against that instance.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on nightly pipelines.
### `RAT-Tag`
This job triggers a pipeline in the
[RAT](https://gitlab.com/gitlab-org/distribution/reference-architecture-tester)
project passing the URL to the Ubuntu 22.04 package built in this pipeline,
which will spin up a PostgreSQL HA instance with that package using
[GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit), and run
QA against that instance.
This job is run only on [Release mirror](https://dev.gitlab.org/gitlab/omnibus-gitlab) on EE tag pipelines.
### `license-upload`
This job compiles the license information of all the dependencies from the
package and uploads it to the S3 bucket. This is used by
[`pages`](#pages) scheduled job in [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) to populate the
[License collection webpage](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html).
## Housekeeping Jobs
### `update-gems-cache` and `update-trigger-package-cache`
Jobs that extends from the `.gems-cache` and `.trigger-package-cache` shared `cache` definitions only `pull` their cache.
These caches are updated in scheduled pipelines by `update-gems-cache` and `update-trigger-package-cache` jobs respectively, when the `CACHE_UPDATE` exists.
### `dependency_update`
This job makes use of Dependencies.io to automatically check for version updates
of various components that we include in our package and open merge request
against [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) if an update is found.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab), on scheduled pipelines when
`DEPS_PIPELINE` variable exists.
### `dependencies_io_check`
This job automatically triggers a e2e:package-and-test pipeline in [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror),
(similar to `Trigger:ce-package` job), when a merge request is made by
`dependency_update` job.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) when the branch name starts with
`deps` (this is the format that `dependency_update` job uses for merge requests)
### `valdiate_packer_changes`
This job checks if the packer configuration files: `https://dev.gitlab.org/gitlab/omnibus-gitlab/-/tree/master/support/packer` are valid or not.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and Security mirror: `https://gitlab.com/gitlab-org/security/omnibus-gitlab`, when any of
the packer configuration files: `https://dev.gitlab.org/gitlab/omnibus-gitlab/-/tree/master/support/packer` are modified.
### `pages`
This job is associated with GitLab Pages, and generates a static website
containing license information of various components included in the package for
each release of GitLab.
This job is run only on [Development repository](https://gitlab.com/gitlab-org/omnibus-gitlab) and on scheduled pipelines which
does not have the `DEPS_PIPELINE` variable (to differentiate it from
`dependency_update` runs).
doc/development/public-attributes.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Working with `public_attributes.json`
Chef stores a copy of a nodes attributes at the end of a reconfigure in `/opt/gitlab/embedded/nodes/$NODE_NAME.json`. Due to the sensitive nature of some of the attributes, it is only readable by the root user. To work around this, we've created a file (defaults to `/var/opt/gitlab/public_attributes.json`) which contains a set of attributes we've whitelisted for use of non-root services. This file is recreated on every run of `gitlab-ctl reconfigure`.
## Adding an entry to `public_attributes.json`
The `public_attributes.json` file is populated by merging the results of a call to the `public_attributes` method of a helper class. For example:
```ruby
class TestHelper < BaseHelper
attr_accessor :node
def public_attributes
{
'gitlab' => {
'test' => node['gitlab']['test']
}
}
end
end
```
The file is generated as part of the [GitLabHandler](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/package/libraries/handlers/gitlab.rb#L36).
## Reading an entry from `public_attributes.json` from a `gitlab-ctl` command
In order to access the public nodes, you should use the provided [`GitlabCtl::Util.get_public_node_attributes` method](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-ctl-commands/lib/gitlab_ctl/util.rb#L60)
```ruby
attributes = GitlabCtl::Util.get_public_node_attributes
puts attributes['gitlab']['test']
```
doc/development/reconfigure_in_detail.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# What happens when `gitlab-ctl reconfigure` is run?
Omnibus GitLab uses [Cinc](https://cinc.sh/) under the hood,
which is a free-as-in-beer distribution of the open source software of [Chef Software Inc](https://docs.chef.io/).
In very basic terms, a [Cinc client](https://cinc.sh/start/client/) run
happens when `gitlab-ctl reconfigure` is run. This document elaborates
the process and details the flow of control during a `gitlab-ctl reconfigure`
run.
`gitlab-ctl reconfigure` is defined in the
[`omnibus-ctl` project](https://gitlab.com/gitlab-org/build/omnibus-mirror/omnibus-ctl/-/blob/0.6.0.1/lib/omnibus-ctl.rb#L517)
and as mentioned above, it performs a
[`cinc-client` run](https://gitlab.com/gitlab-org/build/omnibus-mirror/omnibus-ctl/-/blob/0.6.0.1/lib/omnibus-ctl.rb#L501)
under the hood in [the local mode](https://docs.chef.io/ctl_chef_client/#run-in-local-mode) (using the `-z` flag). This invocation takes
two files as inputs:
- A configuration file named [`solo.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/solo.rb).
- An attribute file named `dna.json`, which is created during build time and loads:
- For CE, the [`gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/files/gitlab-cookbooks/gitlab) cookbook.
- For EE, the [`gitlab-ee`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/files/gitlab-cookbooks/gitlab-ee) cookbook.
Cinc then follows its [two-pass model of execution](https://coderanger.net/two-pass/) for the selected cookbook.
In the load phase, the main cookbook and its dependency cookbooks (mentioned in
the `metadata.rb` file) are loaded. The attributes mentioned in the default
attribute files of these cookbooks are loaded (thus populating the `node` object
with the default values specified in those attribute files) and the custom
resources are all made available for use. Control then moves to the
execution phase. In the `dna.json` file, we specify the cookbook name as the
`run_list`, which makes Cinc use the default recipe as the only entry in the run
list.
The `gitlab-ee` cookbook extends the `gitlab` cookbook with EE-only
features. For explanation purposes, let's first look at the
[`gitlab` cookbook's default recipe](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab/recipes/default.rb).
## The default recipe
The functionality in the default recipe can be summarized as below:
1. Call the `config` recipe to load the attributes from `gitlab.rb` and fully
populate the `node` object.
1. Check for any deprecations and exit early.
1. Check for any problematic settings in the run environment. For example,
if `LD_LIBRARY_PATH` is defined it can interfere with the included
libraries, how software links against them, and raise warnings.
1. Check for non-UTF-8 locales and raise warnings.
1. Create and configure necessary base directories like `/etc/gitlab` (unless
explicitly disabled), `/var/opt/gitlab`, and `/var/log/gitlab`.
1. Call other necessary helper recipes and enable or disable recipes for different
services, etc.
Note that this summary is not complete. Check out the default recipe code to
learn more.
### The config recipe
The config recipe populates the `node` object with the
final values for various settings after merging static default values,
computed default values, and user values specified
in `/etc/gitlab/gitlab.rb` file.
In the above statement, we mention two types of default values:
- **Static:** Static default values are specified in various
attribute files in different cookbooks and are independently set.
- **Computed:** Computed default values are used in scenarios where the
default value for a setting depends on either the static default
value or user-specified value of another setting.
For example, `gitlab_rails['gitlab_port']` defaults to the static
value `80`. It translates to `production.gitlab.port` in the rendered
`gitlab.yml` file that configures the GitLab Rails listener port. The
`gitlab_rails['pages_host']` and `gitlab_rails['pages-port']` values,
which inform GitLab Rails about GitLab Pages, depend on the user
specified value in from `pages_external_url`. The computation of these
default values may only happen **_after_** `gitlab.rb` gets parsed.
#### What goes in to the `gitlab.rb` file?
Omnibus GitLab uses a module named `Gitlab` to store the settings specified in
`gitlab.rb`. This module, which extends `Mixlib::Config` module, can work as a
configuration hash. In the
[definition of this module](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/package/libraries/config/gitlab.rb),
we register the various roles, attribute blocks, and top-level attributes that
can be specified in `gitlab.rb`. The code for this registration is specified in
the `SettingsDSL` module, and is extended by `GitLab` module.
When `Gitlab.from_file` is called in the config recipe, the settings from
`gitlab.rb` are parsed and loaded to the `Gitlab` object, and are accessible via
`Gitlab['<setting_name>']`.
Once `gitlab.rb` has been parsed and its values are available in `Gitlab`
object, we can compute the default values of settings dependent on other settings.
#### Computation of default values
Each component with attributes that require calculation specify a library file
at registration. One method in this file, `parse_variables`, validates user-provided
input and, in typical usage, sets default values if the user did not already specify
a value. It also detects bad configuration and raises errors.
As mentioned above, `parse_variables` sets default values based on static
defaults or user-provided values in related settings. The default values
from these related settings are available in the `node` object after the load
phase of the Cinc run. This `node` object, while available in the recipe, is
not available in the libraries. To make the static default values available
in the libraries, we attach the `node` object to the `GitLab` object in
the config recipe with the code below:
```ruby
Gitlab[:node] = node
```
With this, the static default values of attributes can be accessed in the libraries
using `Gitlab[:node]['<top-level-key>'][<setting>]`.
It is important to note that:
- The `Gitlab` object stores keys as they are mentioned in `gitlab.rb`.
- `node` stores them based on the defined nesting attribute-block-attribute hierarchy.
So, `gitlab_rails` settings from
`gitlab.rb` are available as `Gitlab['gitlab_rails'][*]` while default values of
those settings from attribute files are available at
`Gitlab[:node]['gitlab']['gitlab_rails']`. The `gitlab_rails` key is specified
under the `gitlab` attribute block, so an extra layer of nesting is present
while accessing it via `node`.
While the `Gitlab` object is technically only supposed to hold the settings
specified in `gitlab.rb`, when computing default values of settings based on
other settings, we usually put them under `Gitlab` key itself.
[Issue #3932](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3923) is open to change
this behavior.
#### Handling of secrets
Many attributes required for GitLab functionality are secrets that need to
persist across reconfigure runs and, in multi-node setups, across different
nodes. Moreover, if the user did not specify values for these secrets then
Omnibus GitLab must create them to function. For this purpose, each library file
specifies a `parse_secrets` method similar to the `parse_variables` method. This
method generates secrets, unless explicitly disabled, if none have been specified
in the `gitlab.rb` file.
These secrets are written, unless explicitly disabled, to a file named
`/etc/gitlab/gitlab-secrets.json`. This file is read by subsequent reconfigure
runs and the secret persists across every reconfigure run.
#### Update the node object with final attribute list
After all libraries parse their respective attributes and secrets, the
final configuration is ready to merge with default attributes already
present in `node`. The `node.consume_attributes` method merges the
final configuration with the default configuration populated in the load
phase. Any configuration read from `gitlab.rb` or computed in the libraries
overwrite the values for keys matched in the `node` object. At this point,
the `node` object contains the final attribute list.
#### `gitlab-cluster.json` file
A user configures the system with `gitlab.rb` to match the requirement. In
certain scenarios, however, we need to perform alterations without changes
to the `gitlab.rb` file. Omnibus GitLab writes to a different file,
`/etc/gitlab/gitlab-cluster.json`, that overrides user-specified values in
`gitlab.rb`. The `gitlab-ctl` command or a reconfigure dynamically populates
this file and it gets read and merged over the `node` attributes at the end
of the config recipe.
The `gitlab-ctl geo promote` command, when used on a multi-node PostgreSQL
instance with Patroni, must disable the Patroni standby server. In this
example, the standby server would normally be disabled via
`patroni['standby_cluster']['enable']` in `gitlab.rb`. The `gitlab.rb` file
should remain read-only for the duration of the Cinc run, so this setting
is changed in the `gitlab-cluster.json` file. Future reconfigure runs parse
the `gitlab-cluster.json` file at the end and `node['patroni']['standby_cluster']['enable']`
will evaluate `false`.
The Cinc run executes helper and service-specific recipes after
the config recipe. Once these are complete, the node object is
fully populated and may be used in recipes and resources.
The default recipe in EE cookbook essentially calls `gitlab::default` recipe,
and then handles the EE-specific components separately.
doc/development/s390x.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Developing for s390x architecture
## Endianness
[Linux on IBM Z](https://en.wikipedia.org/wiki/Linux_on_IBM_Z) runs on the s390x architecture. Unlike x86,
the s390x architecture is [big-endian](https://en.wikipedia.org/wiki/Endianness). Many open source projects only
test against little-endian systems, so there may be endian-related bugs
lurking. For example, in the Ruby implementation of Protocol Buffers,
we [fixed a handful of big-endian bugs](https://github.com/protocolbuffers/protobuf/issues/9141#issuecomment-962698503).
## VM provisioning
There are several ways to develop for s390x. You can use:
- [A LinuxONE virtual server](https://community.ibm.com/zsystems/l1cc/).
- [QEMU to emulate s390x](https://qemu-project.gitlab.io/qemu/system/target-s390x.html).
Use the following commands to spin up an emulated s390x virtual machine with Docker:
```shell
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker run --rm -it s390x/ubuntu bash
```
Note that the QEMU virtual machine is usually slower than a LinuxONE
machine.
## Bazel
[Bazel](https://bazel.build/) is needed by a number of Google projects,
such as Protocol Buffers. When compiling from source, you might run [into these fatal compiler warnings](https://github.com/protocolbuffers/protobuf/issues/9141#issuecomment-955997742).
There is a [workaround to disable those warnings](https://github.com/bazelbuild/bazel/issues/13597#issuecomment-930606195).
In addition, IBM [posted these instructions](https://github.com/linux-on-ibm-z/docs/wiki/Building-Bazel).
## Omnibus build image
To build Omnibus GitLab for s390x, you can use [this Dockerfile template](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/blob/master/docker/Dockerfile_debian_10_s390x.erb). Generate the Dockerfile with:
```shell
./docker/scripts/generate-dockerfile docker/Dockerfile_debian_10_s390x.erb
```
doc/development/setup.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Set up your development environment
Development of the Linux package can be done using an existing package available
from the [Downloads page](https://about.gitlab.com/install/). To know how to set up
a build environment to build these packages and use them, please read
[Setting up a Build Environment](../build/build_package.md#prepare-a-build-environment)
Choose one of the GitLab installation methods below. To provide isolation and
to prevent rebuilding of the package for each and every change, it is preferred
to use a container for development.
## Set up a container
1. Install Docker for your OS as per the [official Docker installation docs](https://docs.docker.com/get-docker/).
1. Pull the GitLab CE nightly image:
```shell
docker pull gitlab/gitlab-ce:nightly
```
1. Run the Docker image with a shell prompt:
```shell
docker run -it --publish 443:443 --publish 80:80 --publish 22:22 gitlab/gitlab-ce:nightly bash
```
This command runs Docker with the GitLab nightly image. You start with a
bash prompt, where you run the following commands.
1. Initialize GitLab by first starting runsv, followed by `reconfigure`:
```shell
/opt/gitlab/embedded/bin/runsvdir-start &
gitlab-ctl reconfigure
```
If you have sysctl errors after running `reconfigure`, there is a workaround in the
[common installation problems doc](../troubleshooting.md#failed-to-modify-kernel-parameters-with-sysctl).
1. (Optional) Take a snapshot of your container, so you can revert to this image if required. Run these commands on the Docker host:
```shell
docker ps # Find the container ID of our container.
docker commit <container_id> gitlab_nightly_post_install
```
## Use an official nightly package
1. Get the GitLab CE nightly package from the [Nightly Build repository](https://packages.gitlab.com/gitlab/nightly-builds)
and install it using the instructions given on that page.
NOTE:
On Ubuntu Xenial, you may have to install `tzdata`. This
[issue is reported in #4769](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4679).
1. Configure and start GitLab.
1. Check if you can access the GitLab instance from your host browser on `<ip address of host>`.
1. Install the basic tools used for developing Omnibus GitLab:
```shell
sudo apt-get install git
```
## Get the source of Omnibus GitLab
1. Get the source code of Omnibus GitLab from the [repository on GitLab.com](https://gitlab.com/gitlab-org/omnibus-gitlab):
```shell
git clone https://gitlab.com/gitlab-org/omnibus-gitlab.git ~/omnibus-gitlab
```
We will be doing the development inside the `~/omnibus-gitlab` directory.
1. Instructing GitLab to apply the changes we make to the cookbooks.
During development, we need the changes we make to the cookbooks to be
applied immediately to the running GitLab instance. So, we have to configure
GitLab to use those cookbooks instead of the ones shipped during
installation. This involves backing up of the existing cookbooks directories
and symlinking the directories where we make modifications to its location:
```shell
cd ~/omnibus-gitlab/files/gitlab-cookbooks
for i in $(ls); do
mv "/opt/gitlab/embedded/cookbooks/${i}" "/opt/gitlab/embedded/cookbooks/${i}.$(date +%s)"
ln -s "$(pwd)/${i}" "/opt/gitlab/embedded/cookbooks/${i}"
done
```
Now, you can make any necessary changes in the cookbooks inside `~/omnibus-gitlab/files/gitlab-cookbooks/`
and run `sudo gitlab-ctl reconfigure` for those changes to take effect.
## Run GitLab QA Against Your Development Environment
You can run [GitLab QA](https://gitlab.com/gitlab-org/gitlab-qa) tests against your development instance.
This ensures that your new work is behaving as expected, and not breaking anything else. You can even add your own tests to QA to validate what you are working on.
1. Create a user account on your development instance for GitLab QA to use
Then, from any machine that can reach your development instance:
1. Clone the [GitLab EE](https://gitlab.com/gitlab-org/gitlab) repository
```shell
git clone git@gitlab.com:gitlab-org/gitlab.git
```
1. Change to the `qa` directory
```shell
cd gitlab/qa
```
1. Install the required gems
```shell
bundle install
```
1. Run the tests
```shell
GITLAB_USERNAME=$USERNAME GITLAB_PASSWORD=$PASSWORD bundle exec bin/qa Test::Instance $DEV_INSTANCE_URL
```
### Trigger QA pipeline against deployed instance
If there is sustained network access to the deployed instance, you can trigger GitLab QA tests against the deployed instance using the [GitLab QA Executor](https://gitlab.com/gitlab-org/quality/gitlab-qa-executor) project. It contains CI configuration to run GitLab QA against self-managed GitLab environments with parallelization.
## Run specific chefspec tests
You can also test your changes against the current tests (or to test your newly added tests).
1. Install `bundler` and `ruby-dev`, which are required to build the necessary gems:
```shell
sudo apt install bundler ruby-dev
```
1. Change to the `omnibus-gitlab` directory:
```shell
cd ~/omnibus-gitlab
```
1. Install the required gems inside the omnibus directory:
```shell
/usr/bin/bundle install --path vendor/bundle
```
If you use the GitLab Nightly Docker images, `/opt/gitlab/embedded/bin` is prepended to the `$PATH`, so using `bundle` alone uses the binary
that is included as part of GitLab. That's why we use the absolute path to the system `bundle`.
1. Run your desired tests. The tests may need to run as root, as they need access to read the secrets file:
```shell
sudo bundle exec rspec spec/<path_to_spec_file>
```
## Use chef-shell with `omnibus-gitlab` cookbooks
You can use [chef-shell](https://docs.chef.io/workstation/chef_shell/) to debug changes to the cookbooks in your instance.
As root in your development server run:
```shell
/opt/gitlab/embedded/bin/cinc-shell -z -c /opt/gitlab/embedded/cookbooks/solo.rb -s -j /opt/gitlab/embedded/cookbooks/dna.json
```
## Use Customers Portal Staging in GitLab
To connect your GitLab instance to Customers Portal Staging, you can set the following
[custom environment variables](../settings/environment-variables.md#setting-custom-environment-variables)
in `/etc/gitlab/gitlab.rb` by supplying them in a `gitlab_rails['env']` hash. Set:
- `GITLAB_LICENSE_MODE` to `test`
- `CUSTOMER_PORTAL_URL` to `https://customers.staging.gitlab.com`
For example:
```ruby
gitlab_rails['env'] = {
"GITLAB_LICENSE_MODE" => "test",
"CUSTOMER_PORTAL_URL" => "https://customers.staging.gitlab.com"
}
```
NOTE:
For [GitLab Geo](https://docs.gitlab.com/ee/administration/geo/), deployments using the above configuration must be added to each secondary site.
## OpenShift GitLab Development Setup
See Omnibus GitLab [development setup](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/doc/developer/guide.md) documentation.
doc/development/upgrading-chef.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Upgrading Chef
Chef is a major part of `omnibus-gitlab`, and periodically needs to be updated. Follow these steps
to upgrade to the latest version and verify the functionality.
## Upgrade steps
1. Create a new branch in `omnibus-gitlab`
```shell
git switch -c upgrade-chef-to-X
```
1. Update the appropriate software entries in `config/templates/omnibus-gitlab-gems/Gemfile`.
1. In the `config/templates/omnibus-gitlab-gems` directory, run `bundle update` for the updated gems:
```shell
bundle update chef ohai
```
1. Update entries in `Gemfile` to the new version. At a minimum, the `chef` and `ohai` entries will need to be updated.
1. Update the bundle
1. If this is a major version upgrade
```shell
bundle update chef ohai
```
1. If this is a minor version upgrade
```shell
bundle update chef ohai --conservative
```
It may be necessary to chase down errors related to dependencies being upgraded
1. Commit the changes
```shell
git add config/templates/omnibus-gitlab-gems/Gemfile{,.lock}
git add Gemfile{,.lock}
git commit
git push
```
1. Ensure the pipelines pass on GitLab.com
1. Trigger an EE package pipeline to ensure we get a `gitlab-qa` run
1. When available, trigger an HA validation job
1. Check QA jobs for the package pipeline, and the HA validation job, ensure pipelines are green, or any failures are unrelated.
1. Push to `omnibus-gitlab` on `dev.gitlab.org` and ensure the package builds on all platforms
1. Download a package to a dev environment, and verify you can upgrade from an older version of `omnibus-gitlab`, to the newer package
1. Verify a Geo installation is successful using the newer package
## Bonus points
1. Read through the Changelog and Release notes, identify any improvements, new features, or bug fixes which may apply to omnibus and open follow up issues
doc/development/upgrading-software-components.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Upgrading software components
The Linux package is created from a series of software components, some of which
are developed by GitLab and others which are sourced from free and open source
projects. Software components can be updated individually as new features, bug
fixes, and security vulnerabilities become available.
Software component upgrades can be risky, especially when non-backwards
compatible changes are made. Considering [Semantic versioning](https://semver.org/)], examining changelogs, and examining release
notes can give a sense of the amount of risk involved in an upgrade. In all
cases, upgrades should be thoroughly tested before merging.
The [CNG](https://gitlab.com/gitlab-org/build/CNG) project uses some of these
same software components. Components common to both projects should be updated
in both.
## Types of software components
There are two types of software components used in the Linux package:
- External software components
- Internal software components
### External software components
External software components source is downloaded directly from an external site
or copied from the
[`omnibus-mirror`](https://gitlab.com/gitlab-org/build/omnibus-mirror) repository.
A component can be provided using a `git clone`, extracting from a source
tarball, performing a `gem install` (for Ruby modules), or performing a `pip
install` (for Python modules).
### Internal software components
Internal software component are developed by GitLab and external contributors.
Source for internal software components is downloaded from a project's GitLab
repository. Versions used in a build are set by the Git reference contained in
the project's corresponding `*VERSION` files. These versions can be overridden
by environment variables. For more information, see
[Use specific branches or versions of a GitLab component](../build/team_member_docs.md#use-specific-branches-or-versions-of-a-gitlab-component).
Updates to internal software components are done by merge requests in the
corresponding repository.
## Internal software component update workflow
A typical workflow for updating an internal software component.
### Create a fork/branch
External contributors should create a fork of the
[`gitlab-org/omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) repository.
Create a new branch from the target branch (usually `master` of the
[`gitlab-org/omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) repository.
### Modify `config/software/<component.rb>`
1. Find the corresponding configure file for the component that you want to
update in the `config/software` directory. For example
`config/software/prometheus.rb` is used for the Prometheus component.
1. Change the version to the version you want to update to. If applicable, also
change the corresponding `sha256` to the value of the corresponding version
source tarball.
### Add or modify any required patches
The new component version may require:
- Adding new patches.
- Removing existing patches.
- Changing existing patches.
All patch files go in `config/patches/<component name>`. They are then
referenced in the corresponding `config/software/<component name>.rb` file.
Examples can be found at:
- [unzip component](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/config/software/unzip.rb)
- [unzip patches](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/config/patches/unzip).
### Push branch
Push the branch to the to the upstream repository.
### Create merge request (MR)
Create a merge request using your development branch and the target branch.
### Build
Build the Linux package either:
- Using the CI/CD system.
- Building locally using [Build local Linux package](../build/build_package.md).
You must build on all architectures using the CI/CD system before a merge
request for an updated software component can be accepted.
### Test
Install the resulting Linux package and test the component changes.
## Testing software component updates
### Minimum test requirements
At a minimum, the following tests should be performed when updating a software
component.
- Perform a successful GitLab Enterprise Edition (EE) build on all supported platforms.
- Run `qa-subset-test` as well as manual `qa-remaining-test-manual` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
- Install and verify that component version has been upgraded.
- Verify basic functionality of the software component.
Additional testing is almost always required and varies depending on the
software component.
### Test plans
Here are test plans for individual components. They are meant to be copied into
the merge request where their execution can be recorded.
Not every component is listed here. Please consider creating a merge request to
add one for a component upgrade that you are working on. Use
`test-plans/upgrade-component-testplan-template.md` as a starting point.
These test plans are not exhaustive. The might need to be supplemented depending
on the degree of change made to the component. Record these additions in the
merge request and consider adding them here. Use the following filename pattern
when creating the test plan file:
```plaintext
upgrade-<component-name>-testplan.md
```
And add a link in `test-plans/index.md`.
doc/development/vulnerabilities.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Handle vulnerabilities detected by dependency scanning
A scheduled pipeline runs `dependency_scanning` job every night. This job adds new
vulnerabilities to the
[Vulnerability Report](https://gitlab.com/gitlab-org/omnibus-gitlab/-/security/vulnerability_report/).
Slack notifications tell `#g_distribution` on Slack when new
vulnerabilities are detected. Complete the following steps when you receive this notification.
1. Visit the [Omnibus Vulnerability Report](https://gitlab.com/gitlab-org/omnibus-gitlab/-/security/vulnerability_report)
and locate the appropriate vulnerability. If the vulnerability is legitimate:
- Select `Create Issue` to open a confidential issue in the
[`omnibus-gitlab` issue tracker](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues).
- Change the vulnerability status to `Confirmed`. If the vulnerability turns
out to be a false positive, duplicate, or otherwise not actionable, change the
status to `Dismiss`.
1. Label the issue with the `security` and `For Scheduling` labels. The GitLab
Security team is then made aware of this issue due to the automation by
escalator: `https://gitlab.com/gitlab-com/gl-security/automation/escalator`.
1. The Security team triages and schedules the issue with the help of Distribution.
1. If the issue is actionable for us, the Security team:
- Schedules the issue based on its severity and priority.
- Creates the needed merge requests (MRs) to target all relevant branches.
1. After the MR that fixes the vulnerability has been merged, and the corresponding
issue is closed:
- Visit the [Omnibus Vulnerability Report](https://gitlab.com/gitlab-org/omnibus-gitlab/-/security/vulnerability_report).
- Locate the appropriate vulnerability and set the status to `Resolved` if not
already done automatically.
1. If the issue is a no-op for our use case, set its status to `Dismissed` in the
[Vulnerability Report](https://gitlab.com/gitlab-org/omnibus-gitlab/-/security/vulnerability_report)
page and close the corresponding issue.
doc/index.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Linux package documentation
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
The Linux package has different services and tools required to run GitLab. Most users can install it without laborious
configuration.
## Package information
- [Checking the versions of bundled software](https://docs.gitlab.com/ee/administration/package_information/index.html#checking-the-versions-of-bundled-software)
- [Package defaults](https://docs.gitlab.com/ee/administration/package_information/defaults.html)
- [Components included](https://docs.gitlab.com/ee/development/architecture.html#component-list)
- [Deprecated Operating Systems](https://docs.gitlab.com/ee/administration/package_information/supported_os.html#os-versions-that-are-no-longer-supported)
- [Signed Packages](https://docs.gitlab.com/ee/administration/package_information/signed_packages.html)
- [Deprecation Policy](https://docs.gitlab.com/ee/administration/package_information/deprecation_policy.html)
- [Licenses of bundled dependencies](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html)
## Installation
For installation details, see [Install GitLab with the Linux package](installation/index.md).
## Running on a low-resource device (like a Raspberry Pi)
You can run GitLab on supported low-resource computers like the Raspberry Pi 3, but you must tune the settings
to work best with the available resources. Check out the [documentation](settings/rpi.md) for suggestions on what to adjust.
## Maintenance
- [Get service status](maintenance/index.md#get-service-status)
- [Starting and stopping](maintenance/index.md#starting-and-stopping)
- [Invoking Rake tasks](maintenance/index.md#invoking-rake-tasks)
- [Starting a Rails console session](maintenance/index.md#starting-a-rails-console-session)
- [Starting a PostgreSQL superuser `psql` session](maintenance/index.md#starting-a-postgresql-superuser-psql-session)
- [Container registry garbage collection](maintenance/index.md#container-registry-garbage-collection)
## Configuring
- [Configuring the external URL](settings/configuration.md#configure-the-external-url-for-gitlab)
- [Configuring a relative URL for GitLab (experimental)](settings/configuration.md#configure-a-relative-url-for-gitlab)
- [Storing Git data in an alternative directory](settings/configuration.md#store-git-data-in-an-alternative-directory)
- [Changing the name of the Git user group](settings/configuration.md#change-the-name-of-the-git-user-or-group)
- [Specify numeric user and group identifiers](settings/configuration.md#specify-numeric-user-and-group-identifiers)
- [Start Linux package installation services only after a given file system is mounted](settings/configuration.md#start-linux-package-installation-services-only-after-a-given-file-system-is-mounted)
- [Disable user and group account management](settings/configuration.md#disable-user-and-group-account-management)
- [Disable storage directory management](settings/configuration.md#disable-storage-directories-management)
- [Failed authentication ban](settings/configuration.md#configure-a-failed-authentication-ban)
- [SMTP](settings/smtp.md)
- [NGINX](settings/nginx.md)
- [LDAP](https://docs.gitlab.com/ee/administration/auth/ldap/index.html)
- [Puma](https://docs.gitlab.com/ee/administration/operations/puma.html)
- [ActionCable](settings/actioncable.md)
- [Redis](settings/redis.md)
- [Logs](settings/logs.md)
- [Database](settings/database.md)
- [Reply by email](https://docs.gitlab.com/ee/administration/reply_by_email.html)
- [Environment variables](settings/environment-variables.md)
- [`gitlab.yml`](settings/gitlab.yml.md)
- [Backups](settings/backups.md)
- [Pages](https://docs.gitlab.com/ee/administration/pages/index.html)
- [SSL](settings/ssl/index.md)
- [GitLab and Registry](https://docs.gitlab.com/ee/administration/packages/container_registry.html)
- [Configuring an asset proxy server](https://docs.gitlab.com/ee/security/asset_proxy.html)
- [Image scaling](settings/image_scaling.md)
- [GitLab Agent](https://docs.gitlab.com/ee/administration/clusters/kas.html)
## Upgrading
- [Upgrade guidance](https://docs.gitlab.com/ee/update/package/), including [supported upgrade paths](https://docs.gitlab.com/ee/update/index.html#upgrade-paths).
- [Upgrade from Community Edition to Enterprise Edition](https://docs.gitlab.com/ee/update/package/convert_to_ee.html)
- [Upgrade to the latest version](https://docs.gitlab.com/ee/update/package/#upgrade-using-the-official-repositories)
- [Downgrade to an earlier version](https://docs.gitlab.com/ee/update/package/downgrade.html)
- [Upgrade from a non-Linux package PostgreSQL to a Linux package installation using a backup](update/convert_to_omnibus.md#upgrading-from-a-non-linux-package-postgresql-to-a-linux-package-installation-using-a-backup)
- [Upgrade from a non-Linux package PostgreSQL to a Linux package installation in-place](update/convert_to_omnibus.md#upgrading-from-a-non-linux-package-postgresql-to-a-linux-package-installation-in-place)
## Troubleshooting
For troubleshooting details, see [Troubleshooting Omnibus GitLab installation issues](troubleshooting.md).
## Omnibus GitLab developer documentation
See the [development documentation](development/index.md)
doc/installation/index.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Install GitLab with the Linux package
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
## Prerequisites
- [Installation Requirements](https://docs.gitlab.com/ee/install/requirements.html).
- If you want to access your GitLab instance by using a domain name, like `mygitlabinstance.com`,
make sure the domain correctly points to the IP of the server where GitLab is being
installed. You can check this using the command `host mygitlabinstance.com`.
- If you want to use HTTPS on your GitLab instance, make sure you have the SSL
certificates for the domain ready. (Note that certain components like
Container Registry which can have their own subdomains requires certificates for
those subdomains also.)
- If you want to send notification emails, install and configure a mail server (MTA)
like Sendmail or Postfix. Alternatively, you can use other [third party SMTP servers](../settings/smtp.md).
## Installation and Configuration
These configuration settings are commonly used when configuring a Linux package installation.
For a complete list of settings, see the [README](../index.md#configuring) file.
- [Installing GitLab](https://about.gitlab.com/install/).
- [Manually downloading and installing a GitLab package](https://docs.gitlab.com/ee/update/package/#upgrade-using-a-manually-downloaded-package).
- [Setting up a domain name/URL](../settings/configuration.md#configure-the-external-url-for-gitlab)
for the GitLab Instance so that it can be accessed easily.
- [Enabling HTTPS](../settings/nginx.md#enable-https).
- [Enabling notification emails](../settings/smtp.md#smtp-settings).
- [Enabling replying via email](https://docs.gitlab.com/ee/administration/reply_by_email.html#set-it-up).
- [Installing and configuring Postfix](https://docs.gitlab.com/ee/administration/reply_by_email_postfix_setup.html).
- [Enabling container registry on GitLab](https://docs.gitlab.com/ee/administration/packages/container_registry.html#container-registry-domain-configuration).
- You require SSL certificates for the domain used for container registry.
- [Enabling GitLab Pages](https://docs.gitlab.com/ee/administration/pages/).
- If you want HTTPS enabled, you must get wildcard certificates.
- [Enabling Elasticsearch](https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch.html).
- [GitLab Mattermost](https://docs.gitlab.com/ee/integration/mattermost/). Set up the Mattermost messaging app that ships with the Linux package.
- [GitLab Prometheus](https://docs.gitlab.com/ee/administration/monitoring/prometheus/index.html)
Set up the Prometheus monitoring included in the Linux package.
- [GitLab High Availability Roles](../roles/index.md).
### Set up the initial account
By default, a Linux package installation automatically generates a password for the
initial administrator user account (`root`) and stores it to
`/etc/gitlab/initial_root_password` for at least 24 hours. For security reasons,
after 24 hours, this file is automatically removed by the first `gitlab-ctl reconfigure`.
The default account is tied to a randomly-generated email address. To override
this, pass the `GITLAB_ROOT_EMAIL` environment variable to the installation command.
NOTE:
If GitLab can't detect a valid hostname for the server during the
installation, a reconfigure does not run.
To provide a custom initial root password, you have two options:
- Pass the `GITLAB_ROOT_PASSWORD` environment variable to the
[installation command](https://about.gitlab.com/install/) provided
the hostname for the server is set up correctly:
```shell
sudo GITLAB_ROOT_EMAIL="<gitlab_admin@example.com>" GITLAB_ROOT_PASSWORD="<strongpassword>" EXTERNAL_URL="http://gitlab.example.com" apt install gitlab-ee
```
If during the installation GitLab doesn't automatically perform a
reconfigure, you have to pass the `GITLAB_ROOT_PASSWORD` or `GITLAB_ROOT_EMAIL`
variable to the first `gitlab-ctl reconfigure` run.
- Before the first reconfigure, edit `/etc/gitlab/gitlab.rb` (create it if it
doesn't exist) and set:
```ruby
gitlab_rails['initial_root_password'] = '<my_strong_password>'
```
Both of these methods apply only during the initial database seeding, which happens
during the first reconfigure. For subsequent reconfigure runs, neither of
the aforementioned methods have any effect. In that case, use the random
password in `/etc/gitlab/initial_root_password` to log in, or
[reset the root password](https://docs.gitlab.com/ee/security/reset_user_password.html).
## Using Docker image
You can also use the Docker images provided by GitLab to install and configure a GitLab instance.
Check the [documentation](https://docs.gitlab.com/ee/install/docker.html) to know more.
## Uninstall the Linux package (Omnibus)
To uninstall the Linux package, you can opt to either keep your data (repositories,
database, configuration) or remove all of them:
1. Optional. To remove
[all users and groups created by the Linux package](../settings/configuration.md#disable-user-and-group-account-management)
before removing the package (with `apt` or `yum`):
```shell
sudo gitlab-ctl stop && sudo gitlab-ctl remove-accounts
```
NOTE:
If you have problems removing accounts or groups, run `userdel` or `groupdel` manually
to delete them. You might also want to manually remove the leftover user home directories
from `/home/`.
1. Choose whether to keep your data or remove all of them:
- To preserve your data (repositories, database, configuration), stop GitLab and
remove its supervision process:
```shell
sudo systemctl stop gitlab-runsvdir
sudo systemctl disable gitlab-runsvdir
sudo rm /usr/lib/systemd/system/gitlab-runsvdir.service
sudo systemctl daemon-reload
sudo systemctl reset-failed
sudo gitlab-ctl uninstall
```
- To remove all data:
```shell
sudo gitlab-ctl cleanse && sudo rm -r /opt/gitlab
```
1. Uninstall the package (replace with `gitlab-ce` if you have GitLab FOSS installed):
```shell
# Debian/Ubuntu
sudo apt remove gitlab-ee
# RedHat/CentOS
sudo yum remove gitlab-ee
```
doc/jihu_edition.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# JiHu Edition
NOTE:
This section is only relevant if you are a customer in the Chinese market.
GitLab licensed its technology to a new independent Chinese company, called JiHu.
This independent company will help drive adoption of the GitLab complete DevOps
platform in China and foster the GitLab community and open source contributions.
For more information, see the
[blog post announcement](https://about.gitlab.com/blog/2021/03/18/gitlab-licensed-technology-to-new-independent-chinese-company/)
and the [FAQ](https://about.gitlab.com/pricing/faq-jihu/).
## Prerequisites
Before installing GitLab JiHu Edition, it is of critical importance to review the system [requirements](https://docs.gitlab.com/ee/install/requirements.html). The system requirements include details on the minimum hardware, software, database, and additional requirements to support GitLab.
Once you have contracted with JiHu, a JiHu representative will contact you to supply you with a license
that you can use as part of the installation process.
## Install or update a JiHu Edition package
NOTE:
If you are installing for the first time, you have to pass the
`EXTERNAL_URL="<GitLab URL>"` variable to set your preferred domain name. Installation
automatically configures and starts GitLab at that URL. Enabling HTTPS requires
[additional configuration](settings/nginx.md#enable-https) to specify the certificates.
Please refer to the [GitLab Jihu Edition Install](https://gitlab.cn/install/) page
for more details on installing or updating a JiHu Edition package.
### Set initial password and apply license
The first time GitLab JiHu Edition is installed, you are redirected to a password reset screen. Provide
the password for the initial administrator account and you are redirected
back to the login screen. Use the default account's username `root` to log in.
For detailed instructions, see [installation and configuration](installation/index.md#installation-and-configuration).
Additionally, you can navigate to the GitLab administration panel of your server and
[upload your JiHu Edition license file](https://docs.gitlab.com/ee/administration/license.html#uploading-your-license).
## Update GitLab Enterprise Edition to JiHu Edition
To update an existing GitLab Enterprise Edition (EE) server installed using the Omnibus GitLab
packages to GitLab JiHu Edition (JH), you install the JiHu Edition (JH)
package on top of EE.
The available options are:
- (Recommended) Updating from the same version of EE to JH.
- Updating from a lower version of EE to a higher version of JH, provided that this is a supported [upgrade path](https://docs.gitlab.com/ee/update/index.html#upgrade-paths) (for example, EE 13.5.4 to JH 13.10.0).
In the following steps we assume that
you are updating the same version (for example, EE 13.10.0 to JH 13.10.0).
To update EE to JH:
- If you installed GitLab using a deb/rpm package:
1. Take a [backup](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html).
1. Find the currently installed GitLab version:
**For Debian/Ubuntu**
```shell
sudo apt-cache policy gitlab-ee | grep Installed
```
The output should be similar to `Installed: 13.10.0-ee.0`, so the installed
version is `13.10.0-ee.0`.
**For CentOS/RHEL**
```shell
sudo rpm -q gitlab-ee
```
The output should be similar to `gitlab-ee-13.10.0-ee.0.el8.x86_64`, so
the installed version is `13.10.0-ee.0`.
1. Follow the same steps as when
[installing the JiHu Edition package](#install-or-update-a-jihu-edition-package) for
your operating system, and make sure to pick the same version as the one
noted in the previous step. Replace `<url>` with the URL of your package.
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
1. Go to the GitLab administration panel of your server (`/admin/license/new`) and
upload your JiHu Edition license file. If you have an EE license already installed
prior to updating to JiHu, the EE license is automatically deactivated when JH
is installed.
1. Confirm that GitLab is working as expected, then remove the old
Enterprise Edition repository:
**For Debian/Ubuntu**
```shell
sudo rm /etc/apt/sources.list.d/gitlab_gitlab-ee.list
```
**For CentOS/RHEL**
```shell
sudo rm /etc/yum.repos.d/gitlab_gitlab-ee.repo
sudo dnf config-manager --disable gitlab_gitlab-ee
```
- If you installed GitLab using Docker:
1. Follow the [Docker update guide](https://docs.gitlab.com/ee/install/docker.html)
and replace `gitlab/gitlab-ee:latest` with the following:
```shell
registry.gitlab.com/gitlab-jh/omnibus-gitlab/gitlab-jh:<version>
```
Where `<version>` is the currently installed GitLab version, which
you can find with:
```shell
sudo docker ps | grep gitlab/gitlab-ee | awk '{print $2}'
```
The output should be similar to: `gitlab/gitlab-ee:13.10.0-ee.0`, so
in this case, `<version>` equals to `13.10.0`.
1. Go to the GitLab administration panel of your server (`/admin/license/new`) and
upload your JiHu Edition license file. If you have an EE license already installed
prior to updating to JiHu, the EE license is automatically deactivated when JH
is installed.
That's it! You can now use GitLab JiHu Edition! To update to a newer
version, see [Install or update a JiHu Package](#install-or-update-a-jihu-edition-package).
## Go back to GitLab Enterprise Edition
To downgrade the JiHu Edition installation to GitLab Enterprise Edition (EE), install the same version of the Enterprise Edition package on top of the currently installed one.
Depending on the preferred installation method for GitLab EE, either:
- Use the official GitLab package repository and [install GitLab EE](https://about.gitlab.com/install/?version=ee), or
- Download the GitLab EE package and [manually install it](https://docs.gitlab.com/ee/update/package/#upgrade-using-a-manually-downloaded-package).
doc/maintenance/index.md 0 → 100644
View file @ edb06ab2
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Maintenance commands
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
The following commands can be run after installation.
## Get service status
Run `sudo gitlab-ctl status` to see the current state and uptime of each GitLab component.
The output will look similar to this:
```plaintext
run: nginx: (pid 972) 7s; run: log: (pid 971) 7s
run: postgresql: (pid 962) 7s; run: log: (pid 959) 7s
run: redis: (pid 964) 7s; run: log: (pid 963) 7s
run: sidekiq: (pid 967) 7s; run: log: (pid 966) 7s
run: puma: (pid 961) 7s; run: log: (pid 960) 7s
```
As a demonstration, the first line of the previous example can be interpreted as follows:
- `Nginx` is the process name.
- `972` is the process identifier.
- NGINX has been running for 7 seconds (`7s`).
- `log` indicates a [svlogd logging process](https://manpages.ubuntu.com/manpages/lunar/en/man8/svlogd.8.html) attached to the preceding process.
- `971` is the process identifier for the logging process.
- The logging process has been running for 7 seconds (`7s`).
## Tail process logs
See [settings/logs.md.](../settings/logs.md)
## Starting and stopping
After Omnibus GitLab is installed and configured, your server has a runit
service directory (`runsvdir`) process running that gets started at boot via
`/etc/inittab` or the `/etc/init/gitlab-runsvdir.conf` Upstart resource. You
should not have to deal with the `runsvdir` process directly; you can use the
`gitlab-ctl` front-end instead.
You can start, stop or restart GitLab and all of its components with the
following commands.
```shell
# Start all GitLab components
sudo gitlab-ctl start
# Stop all GitLab components
sudo gitlab-ctl stop
# Restart all GitLab components
sudo gitlab-ctl restart
```
Note that on a single-core server it may take up to a minute to restart Puma and
Sidekiq. Your GitLab instance will give a 502 error until Puma is up again.
It is also possible to start, stop or restart individual components.
```shell
sudo gitlab-ctl restart sidekiq
```
Puma does support almost zero-downtime reloads. These can be triggered as
follows:
```shell
sudo gitlab-ctl hup puma
```
Note that you must wait for the `hup` command to finish. This could take some time. Leave the node out of the pool and do not restart services on the node where this is invoked until this completes. You also cannot use a Puma reload to update the Ruby runtime.
Puma has the following signals to control application behavior:
| Signal | Puma |
| -------- | ------ |
| `HUP` | reopen log files defined, or stop the process to force restart |
| `INT` | gracefully stops requests processing |
| `USR1` | restart workers in phases, a rolling restart, without config reload |
| `USR2` | restart workers and reload config |
| `QUIT` | exit the main process |
For Puma, `gitlab-ctl hup puma` will send a sequence of `SIGINT` and `SIGTERM`
(if process does not restart) signals. Puma stops accepting new connections as
soon as `SIGINT` is received. It finishes all running requests. Then `runit`
restarts the service.
## Invoking Rake tasks
To invoke a GitLab Rake task, use `gitlab-rake`. For example:
```shell
sudo gitlab-rake gitlab:check
```
Leave out `sudo` if you are the `git` user.
Contrary to with a traditional GitLab installation, there is no need to change
the user or the `RAILS_ENV` environment variable; this is taken care of by the
`gitlab-rake` wrapper script.
## Starting a Rails console session
For more information, see
[Rails console](https://docs.gitlab.com/ee/administration/operations/rails_console.html#starting-a-rails-console-session).
## Starting a PostgreSQL superuser `psql` session
If you need superuser access to the bundled PostgreSQL service you can
use the `gitlab-psql` command. It takes the same arguments as the
regular `psql` command.
```shell
# Superuser psql access to GitLab's database
sudo gitlab-psql -d gitlabhq_production
```
This will only work after you have run `gitlab-ctl reconfigure` at
least once. The `gitlab-psql` command cannot be used to connect to a
remote PostgreSQL server, nor to connect to a local non-Omnibus PostgreSQL
server.
### Starting a PostgreSQL superuser `psql` session in Geo tracking database
Similar to the previous command, if you need superuser access to the bundled
Geo tracking database (`geo-postgresql`), you can use the `gitlab-geo-psql`.
It takes the same arguments as the regular `psql` command. For HA, see more
about the necessary arguments in:
[Checking Configuration](https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html#checking-configuration)
```shell
# Superuser psql access to GitLab's Geo tracking database
sudo gitlab-geo-psql -d gitlabhq_geo_production
```
## Container Registry garbage collection
Container Registry can use considerable amounts of disk space. To clear up
unused layers, the registry includes a
[garbage collect command](https://docs.gitlab.com/ee/administration/packages/container_registry.html#container-registry-garbage-collection).
## Restrict users from logging into GitLab
If you need to temporarily restrict users from logging into GitLab, you can use
`sudo gitlab-ctl deploy-page up`. When a user goes to your GitLab URL, they
will be shown an arbitrary `Deploy in progress` page.
To remove the page, you simply run `sudo gitlab-ctl deploy-page down`. You can also check the status of the deploy page with `sudo gitlab-ctl deploy-page status`.
As a side note, if you would like to restrict logging into GitLab and restrict
changes to projects, you can [set projects as read-only](https://docs.gitlab.com/ee/administration/troubleshooting/gitlab_rails_cheat_sheet.html#make-a-project-read-only-can-only-be-done-in-the-console)
, then put up the `Deploy in progress` page.
## Rotate the secrets file
If required for security purposes, you can rotate the `/etc/gitlab/gitlab-secrets.json` secrets file. In this file:
- Do not rotate the `gitlab_rails` secrets because it contains the database encryption keys. If this secret is rotated, you see the same
behavior as [when the secrets file is lost](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#when-the-secrets-file-is-lost).
- You can rotate all other secrets.
If you have multiple nodes in your GitLab environment, choose one of your Rails node to perform the initial steps.
To rotate the secrets:
1. [Verify that the database values can be decrypted](https://docs.gitlab.com/ee/administration/raketasks/check.html#verify-database-values-can-be-decrypted-using-the-current-secrets) and either make note of any decryption errors shown, or resolve
them before proceeding.
1. Recommended. Extract your current secrets for `gitlab_rails`. Save the output because you need this later:
```shell
sudo grep "secret_key_base\|db_key_base\|otp_key_base\|encrypted_settings_key_base\|openid_connect_signing_key\|ci_jwt_signing_key" /etc/gitlab/gitlab-secrets.json
```
1. Move your current secrets file to a different location:
```shell
sudo mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.old
```
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#reconfigure-a-linux-package-installation). GitLab will
then generate a new `/etc/gitlab/gitlab-secrets.json` file with new secret values.
1. If you extracted the previous secrets for `gitlab_rails`, edit the new `/etc/gitlab/gitlab-secrets.json` file and replace the key/value pairs
under `gitlab_rails` with the previous secrets output obtained earlier.
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#reconfigure-a-linux-package-installation) again
so the changes made to the secrets file are applied.
1. [Restart GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#restart-a-linux-package-installation) to ensure all services
are using the new secrets.
1. If you have multiple nodes in your GitLab environment, you must copy the secrets to all of your other nodes:
1. On all other nodes, move your current secrets file to a different location:
```shell
sudo mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.old
```
1. Copy the new `/etc/gitlab/gitlab-secrets.json` file from your Rails node onto all of your other GitLab nodes.
1. On all other nodes, [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#reconfigure-a-linux-package-installation)
on each node.
1. On all other nodes, [restart GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#restart-a-linux-package-installation) on each
node to ensure all services are using the new secrets.
1. On all nodes, run a checksum match on the `/etc/gitlab/gitlab-secrets.json` file to confirm that the secrets match:
```shell
sudo md5sum /etc/gitlab/gitlab-secrets.json
```
1. [Verify that the database values can be decrypted](https://docs.gitlab.com/ee/administration/raketasks/check.html#verify-database-values-can-be-decrypted-using-the-current-secrets). The output should match with the previous execution.
1. Confirm that GitLab is working as expected. If it is, it should be safe to delete the old secrets.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment