<!-- This issue template can be used as a great starting point for feature requests. Learn more about the process: https://handbook.gitlab.com/handbook/product/how-to-engage/#customer-feature-requests. The section "Release notes" can be used as a summary of the feature and is also required if you want to have your release post blog MR auto generated using the release post item generator: https://about.gitlab.com/handbook/marketing/blog/release-posts/#release-post-item-generator. The remaining sections are the backbone for every feature in GitLab.
The goal of this template is brevity for quick/smaller iterations. For a more thorough list of considerations for larger features or feature sets, you can leverage the detailed [feature proposal](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20proposal%20-%20detailed.md). -->
### Release notes
<!-- What is the problem and solution you're proposing? This content sets the overall vision for the feature and serves as the release notes that will populate in various places, including the [release post blog](https://about.gitlab.com/releases/categories/releases/) and [Gitlab project releases](https://gitlab.com/gitlab-org/gitlab/-/releases). " -->
### Problem to solve
<!-- What is the user problem you are trying to solve with this issue? -->
### Proposal
<!-- Use this section to explain the feature and how it will work. It can be helpful to add technical details, design proposals, and links to related epics or issues. -->
### Intended users
<!-- Who will use this feature? If known, include any of the following: types of users (e.g. Developer), personas, or specific company roles (e.g. Release Manager). It's okay to write "Unknown" and fill this field in later.
Personas are described at https://handbook.gitlab.com/handbook/product/personas/
<!-- How are you going to track usage of this feature? Think about user behavior and their interaction with the product. What indicates someone is getting value from it?
Explore (../../doc/development/internal_analytics/internal_event_instrumentation/quick_start.md) for a guide.
Make sure to add the appropriate labels for the product stage and/or group (e.g ~"devops::plan") if known and add a comment tagging the appropriate Product Manager.
Use the following resources to find the appropriate labels:
- Use only one tier label choosing the lowest tier this is intended for
<!-- This issue template can be used as a great starting point for feature requests. Learn more about the process: https://handbook.gitlab.com/handbook/product/how-to-engage/#customer-feature-requests. The first section "Release notes" is required if you want to have your release post blog MR auto generated. Currently in BETA, details on the **release post item generator** can be found in the handbook: https://about.gitlab.com/handbook/marketing/blog/release-posts/#release-post-item-generator and this video: https://www.youtube.com/watch?v=rfn9ebgTwKg. The next four sections: "Problem to solve", "Intended users", "User experience goal", and "Proposal", are strongly recommended in your first draft, while the rest of the sections can be filled out during the problem validation or breakdown phase. However, keep in mind that providing complete and relevant information early helps our product team validate the problem and start working on a solution. -->
### Release notes
<!-- What is the problem and solution you're proposing? This content sets the overall vision for the feature and serves as the release notes that will populate in various places, including the [release post blog](https://about.gitlab.com/releases/categories/releases/) and [Gitlab project releases](https://gitlab.com/gitlab-org/gitlab/-/releases). " -->
### Problem to solve
<!-- What problem do we solve? Try to define the who/what/why of the opportunity as a user story. For example, "As a (who), I want (what), so I can (why/value)." -->
### Intended users
<!-- Who will use this feature? If known, include any of the following: types of users (e.g. Developer), personas, or specific company roles (e.g. Release Manager). It's okay to write "Unknown" and fill this field in later.
Personas are described at https://handbook.gitlab.com/handbook/product/personas/
<!-- How are we going to solve the problem? Try to include the user journey! https://about.gitlab.com/handbook/journeys/#user-journey -->
### Further details
<!-- Include use cases, benefits, goals, or any other details that will help us understand the problem better. -->
### Permissions and Security
<!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)?
Consider adding checkboxes and expectations of users with certain levels of membership https://docs.gitlab.com/ee/user/permissions.html
* [ ] Add expected impact to members with no access (0)
* [ ] Add expected impact to Guest (10) members
* [ ] Add expected impact to Reporter (20) members
* [ ] Add expected impact to Developer (30) members
* [ ] Add expected impact to Maintainer (40) members
* [ ] Add expected impact to Owner (50) members
Please consider performing a threat model for the code changes that are introduced as part of this feature. To get started, refer to our Threat Modeling handbook page https://about.gitlab.com/handbook/security/threat_modeling/#threat-modeling.
Don't hesitate to reach out to the Application Security Team (`@gitlab-com/gl-security/appsec`) to discuss any security concerns.
-->
### Documentation
<!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/workflow.html#for-a-product-change
* Add all known Documentation Requirements in this section. See https://docs.gitlab.com/ee/development/documentation/workflow.html
* If this feature requires changing permissions, update the permissions document. See https://docs.gitlab.com/ee/user/permissions.html -->
### Availability & Testing
<!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier.
What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing?
Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance.
* Unit test changes
* Integration test changes
* End-to-end test change
See the Quality Engineering quad planning and test planning processes and reach out to your counterpart Software Engineer in Test for assistance.
Test Planning: https://handbook.gitlab.com/handbook/engineering/infrastructure/test-platform/test-engineering/#test-planning -->
### Available Tier
<!-- This section should be used for setting the appropriate tier that this feature will belong to. Pricing can be found here: https://about.gitlab.com/pricing/
* Free
* Premium/Silver
* Ultimate/Gold
-->
### Feature Usage Metrics
<!-- How are you going to track usage of this feature? Think about user behavior and their interaction with the product. What indicates someone is getting value from it?
-->
### What does success look like, and how can we measure that?
<!--
Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this.
Explore (../../doc/development/internal_analytics/internal_event_instrumentation/quick_start.md) for a guide.
-->
### What is the type of buyer?
<!-- What is the buyer persona for this feature? See https://handbook.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/roles-personas/buyer-persona/
In which enterprise tier should this feature go? See https://handbook.gitlab.com/handbook/company/pricing/#three-tiers -->
### Is this a cross-stage feature?
<!-- Communicate if this change will affect multiple Stage Groups or product areas. We recommend always start with the assumption that a feature request will have an impact into another Group. Loop in the most relevant PM and Product Designer from that Group to provide strategic support to help align the Group's broader plan and vision, as well as to avoid UX and technical debt. https://about.gitlab.com/handbook/product/#cross-stage-features -->
### What is the competitive advantage or differentiation for this feature?
### Links / references
<!-- Label reminders
Make sure to add the appropriate labels for the product stage and/or group (e.g ~"devops::plan") if known and add a comment tagging the appropriate Product Manager.
Use the following resources to find the appropriate labels:
- Use only one tier label choosing the lowest tier this is intended for
<!-- This issue template can be used as a starting point for a UX Issue. This is not a feature request, rather an issue that is being created for a product designer to solve a problem.
The goal of this template is to ensure we have captured all the information available to the product designer so they can approach the problem creatively and efficiently. Please add links to SSOT if this informatin exists elsewhere. -->
### Who will use this solution?
<!-- If known, include any of the following: types of users (e.g. Developer), personas, or specific company roles (e.g. Release Manager). It's okay to write "Unknown" and fill this field in later.
Personas are described at https://handbook.gitlab.com/handbook/product/personas/
This template is based on a model named `CoolWidget`.
To adapt this template, find and replace the following:
Template placeholders
- name: Cool Widgets
description: the human-readable name of the model (plural)
- name: Cool Widget
description: the human-readable name of the model (singular)
- name: cool_widgets
description: the snake-cased name of the model (plural)
- name: cool_widget
description: the snake-cased name of the model (singular)
- name: CoolWidget
description: the ActiveRecord class name of the model
- name: coolWidget
description: the camel-cased name of the model
-->
## Replicate Cool Widgets - Repository
This issue is for implementing Geo replication and verification of Cool Widgets.
For more background, see [Geo self-service framework](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo/framework.md).
In order to implement and test this feature, you need to first [set up Geo locally](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/geo.md).
There are three main sections below. It is a good idea to structure your merge requests this way as well:
1. Modify database schemas to prepare to add Geo support for Cool Widgets
1. Implement Geo support of Cool Widgets behind a feature flag
1. Release Geo support of Cool Widgets
It is also a good idea to first open a proof-of-concept merge request. It can be helpful for working out kinks and getting initial support and feedback from the Geo team. As an example, see the [Proof of Concept to replicate Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56423).
You can look into the following example for implementing replication/verification for a new Git repository type:
### Modify database schemas to prepare to add Geo support for Cool Widgets
#### Add the registry table to track replication and verification state
Geo secondary sites have a [Geo tracking database](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo.md#tracking-database) independent of the main database. It is used to track the replication and verification state of all replicables. Every Model has a corresponding "registry" table in the Geo tracking database.
- [ ] Create the migration file in `ee/db/geo/migrate`:
- [ ] Replace the contents of the migration file with the following. Note that we cannot add a foreign key constraint on `cool_widget_id` because the `cool_widgets` table is in a different database. The application code must handle logic such as propagating deletions.
# To optimize performance of CoolWidgetRegistry.verification_failed_batch
t.index:verification_retry_at,
name: :cool_widget_registry_failed_verification,
order: "NULLS FIRST",
where: "((state = 2) AND (verification_state = 3))"
# To optimize performance of CoolWidgetRegistry.needs_verification_count
t.index:verification_state,
name: :cool_widget_registry_needs_verification,
where: "((state = 2) AND (verification_state = ANY (ARRAY[0, 3])))"
# To optimize performance of CoolWidgetRegistry.verification_pending_batch
t.index:verified_at,
name: :cool_widget_registry_pending_verification,
order: "NULLS FIRST",
where: "((state = 2) AND (verification_state = 0))"
end
end
end
```
-[ ] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
-[ ] Add the new table to the [database dictionary](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/database/database_dictionary.md) defined in [`ee/db/geo/docs/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/db/geo/docs):
```yaml
table_name:cool_widget_registry
description:Description example
introduced_by_url:Merge request link
milestone:Milestone example
feature_categories:
-Feature category example
classes:
-Class example
gitlab_schema:gitlab_geo
```
- [ ] Run Geo tracking database migrations:
```shell
bin/rake db:migrate:geo
```
- [ ] Be sure to commit the relevant changes in `ee/db/geo/structure.sql` and the file under `ee/db/geo/schema_migrations`
### Add verification state to the Model
The Geo primary site needs to checksum every replicable so secondaries can verify their own checksums. To do this, Geo requires the Model to have an associated table to track verification state.
where: "(verification_state = 0 OR verification_state = 3)",
name: NEEDS_VERIFICATION_INDEX_NAME
end
end
defdown
drop_table:cool_widget_states
end
end
```
-[ ] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
-[ ] If `cool_widgets` is a high-traffic table, follow [the database documentation to use `with_lock_retries`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/migration_style_guide.md#when-to-use-the-helper-method)
-[ ] Add the new table to the [database dictionary](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/database/database_dictionary.md) defined in [`db/docs/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/db/docs):
```yaml
---
table_name:cool_widget_states
description:Separate table for Cool Widget verification states
- [ ] Be sure to commit the relevant changes in `db/structure.sql` and the file under `db/schema_migrations`
That's all of the required database changes.
### Implement Geo support of Cool Widgets behind a feature flag
#### Step 1. Implement replication and verification
- [ ] Add the following lines to the `cool_widget` model to accomplish some important tasks:
- Include `::Geo::ReplicableModel` in the `CoolWidget` class, and specify the Replicator class `with_replicator Geo::CoolWidgetReplicator`.
- Include the `::Geo::VerifiableModel` concern.
- Delegate verification related methods to the `cool_widget_state` model.
- For verification, override some scopes to use the `cool_widget_states` table instead of the model table.
- Implement the `verification_state_object` method to return the object that holds
the verification details
- Override some methods to use the `cool_widget_states` table in verification-related queries.
Pay some attention to method `pool_repository`. Not every repository type uses repository pooling. As Geo prefers to use repository snapshotting, it can lead to data loss. Make sure to overwrite `pool_repository` so it returns nil for repositories that do not have pools.
At this point the `CoolWidget` class should look like this:
- [ ] Ensure `CoolWidget.replicables_for_current_secondary` is well-tested. Search the codebase for `replicables_for_current_secondary` to find examples of parameterized table specs. You may need to add more `FactoryBot` traits.
- [ ] Add the following shared examples to `ee/spec/models/ee/cool_widget_spec.rb`:
```ruby
include_examples'a replicable model with a separate table for verification state'do
let(:verifiable_model_record){build(:cool_widget)}# add extra params if needed to make sure the record is in `Geo::ReplicableModel.verifiables` scope
let(:unverifiable_model_record){build(:cool_widget)}# add extra params if needed to make sure the record is NOT included in `Geo::ReplicableModel.verifiables` scope
end
```
- [ ] Create `ee/app/replicators/geo/cool_widget_replicator.rb`. Implement the `#repository` method which should return a `<Repository>` instance, and implement the class method `.model` to return the `CoolWidget` class:
```ruby
# frozen_string_literal: true
moduleGeo
classCoolWidgetReplicator<Gitlab::Geo::Replicator
include::Geo::RepositoryReplicatorStrategy
extend::Gitlab::Utils::Override
defself.model
::CoolWidget
end
defself.git_access_class
::Gitlab::GitAccessCoolWidget
end
defself.no_repo_message
git_access_class.error_message(:no_repo)
end
override:verification_feature_flag_enabled?
defself.verification_feature_flag_enabled?
# We are adding verification at the same time as replication, so we
# don't need to toggle verification separately from replication. When
# the replication feature flag is off, then verification is also off
# (see `VerifiableReplicator.verification_enabled?`)
true
end
override:housekeeping_enabled?
defself.housekeeping_enabled?
# Remove this method if the new Git repository type supports git
# repository housekeeping and the ::CoolWidget#git_garbage_collect_worker_klass
# is implemented. If the data type requires any action to be performed
# before running the housekeeping override the `before_housekeeping` method
# (see `RepositoryReplicatorStrategy#before_housekeeping`)
false
end
defrepository
model_record.repository
end
end
end
```
- [ ] Make sure Geo push events are created. Usually it needs some change in the `app/workers/post_receive.rb` file. Example:
```ruby
defreplicate_cool_widget_changes(cool_widget)
if::Gitlab::Geo.primary?
cool_widget.geo_handle_after_updateifcool_widget
end
end
```
See `app/workers/post_receive.rb` for more examples.
- [ ] Make sure the repository removal is also handled. You may need to add something like the following in the destroy service of the repository:
-[ ] Make sure a Geo secondary site can request and download Cool Widgets on the Geo primary site. You may need to make some changes to `Gitlab::GitAccessCoolWidget`. For example, see [this change for Group-level Wikis](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54914/diffs?commit_id=0f2b36f66697b4addbc69bd377ee2818f648dd33).
-[ ] Make sure a Geo secondary site marks Cool Widgets as missing on primary when a repository does not exist on the Geo primary site. The only way to know about this is to parse the error text. You may need to make some changes to `Gitlab::CoolWidgetReplicator.no_repo_message` to return the proper error message. For example, see [this change for Group-level Wikis](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74133). Alternatively, if a repository should *always* exist on the primary, then a Geo-specific workaround is to create an empty repository when verifying on primary. See examples for [project wikis](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123869) and [design repositories](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123917).
- [ ] Generate the feature flag definition files by running the feature flag commands and following the command prompts:
```shell
bin/feature-flag --ee geo_cool_widget_replication --type development --group'group::geo'
```
- [ ] Add this replicator class to the method `replicator_classes` in
`ee/lib/gitlab/geo.rb`:
```ruby
REPLICATOR_CLASSES=[
::Geo::PackageFileReplicator,
::Geo::CoolWidgetReplicator
]
```
- [ ] Create `ee/spec/replicators/geo/cool_widget_replicator_spec.rb` and perform the necessary setup to define the `model_record` variable for the shared examples:
If there is not an existing factory for the object in `spec/factories/cool_widgets.rb`, wrap the traits in `FactoryBot.create` instead of `FactoryBot.modify`.
- [ ] Make sure the factory also allows setting a `project` attribute. If the model does not have a direct relation to a project, you can use a `transient` attribute. Check out `spec/factories/merge_request_diffs.rb` for an example.
-[ ] Following [the example of Merge Request Diffs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63309) add a `Geo::CoolWidgetState` model in `ee/app/models/geo/cool_widget_state.rb`:
- [ ] Add integration tests for providing CoolWidget registry data to the frontend via the GraphQL API, by duplicating and modifying the following shared examples in `ee/spec/requests/api/graphql/geo/registries_spec.rb`:
```ruby
it_behaves_like'gets registries for',{
field_name: 'coolWidgetRegistries',
registry_class_name: 'CoolWidgetRegistry',
registry_factory: :geo_cool_widget_registry,
registry_foreign_key_field_name: 'coolWidgetId'
}
```
To allow the new replicable to resync and reverify via GraphQL:
- [ ] Add the `CoolWidgetRegistryType` to the `GEO_REGISTRY_TYPE` constant in `ee/app/graphql/types/geo/registrable_type.rb`:
Individual Cool Widget replication and verification data should now be available via the GraphQL API.
#### Step 4. Handle batch destroy
If batch destroy logic is implemented for a replicable, then that logic must be "replicated" by Geo secondaries. The easiest way to do this is use `Geo::BatchEventCreateWorker` to bulk insert a delete event for each replicable.
For example, if `FastDestroyAll` is used, then you may be able to [use `begin_fast_destroy` and `finalize_fast_destroy` hooks, like we did for uploads](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69763).
Or if a special service is used to batch delete records and their associated data, then you probably need to [hook into that service, like we did for job artifacts](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79530).
As illustrated by the above two examples, batch destroy logic cannot be handled automatically by Geo secondaries without restricting the way other teams perform batch destroys. It is up to you to produce `Geo::BatchEventCreateWorker` attributes before the records are deleted, and then enqueue `Geo::BatchEventCreateWorker` after the records are deleted.
- [ ] Ensure that any batch destroy of this replicable is replicated to secondary sites
- [ ] Regardless of implementation details, please verify in specs that when the parent object is removed, the new `Geo::Event` records are created:
```ruby
describe'#destroy'do
subject{create(:cool_widget)}
context'when running in a Geo primary node'do
let_it_be(:primary){create(:geo_node,:primary)}
let_it_be(:secondary){create(:geo_node)}
it'logs an event to the Geo event log when bulk removal is used',:sidekiq_inlinedo
- [ ] Include a comment mentioning that the change is based on a documented template.
-[ ] `replicables_for_current_secondary` and `available_replicables` may differ per Model. If their queries are new, then add [query plans](https://docs.gitlab.com/ee/development/database_review.html#query-plans) to the MR description. An easy place to gather SQL queries is your GDK's `log/test.log` when running tests of these methods.
### Release Geo support of Cool Widgets
- [ ] In the rollout issue you created when creating the feature flag, modify the Roll Out Steps:
- [ ] Cross out any steps related to testing on production GitLab.com, because Geo is not running on production GitLab.com at the moment.
- [ ] Add a step to `Test replication and verification of Cool Widgets on a non-GDK-deployment. For example, using GitLab Environment Toolkit`.
- [ ] Add a step to `Ping the Geo PM and EM to coordinate testing`. For example, you might add steps to generate Cool Widgets, and then a Geo engineer may take it from there.
- [ ] In `ee/config/feature_flags/development/geo_cool_widget_replication.yml`, set `default_enabled: true`
- [ ] In `ee/app/graphql/types/geo/geo_node_type.rb`, remove the `alpha` option for the released type:
description: 'Find Cool Widget registries on this Geo node. '\
'Ignored if `geo_cool_widget_replication` feature flag is disabled.',
alpha: {milestone: '15.5'}# Update the milestone
```
- [ ] Run `bundle exec rake gitlab:graphql:compile_docs` after the step above to regenerate the GraphQL docs.
-[ ] Add a row for Cool Widgets to the `Data types` table in [Geo data types support](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/datatypes.md#data-types)
-[ ] Add a row for Cool Widgets to the `Limitations on replication/verification` table in [Geo data types support](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/datatypes.md#limitations-on-replicationverification). If the row already exists, then update it to show that Replication and Verification is released in the current version.
This template is based on a model named `CoolWidget`.
To adapt this template, find and replace the following:
Template placeholders
- name: Cool Widgets
description: the human-readable name of the model (plural)
- name: Cool Widget
description: the human-readable name of the model (singular)
- name: cool_widgets
description: the snake-cased name of the model (plural)
- name: cool_widget
description: the snake-cased name of the model (singular)
- name: CoolWidget
description: the ActiveRecord class name of the model
- name: coolWidget
description: the camel-cased name of the model
-->
## Replicate Cool Widgets - Blob
This issue is for implementing Geo replication and verification of Cool Widgets.
For more background, see [Geo self-service framework](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo/framework.md).
In order to implement and test this feature, you need to first [set up Geo locally](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/geo.md).
There are three main sections below. It is a good idea to structure your merge requests this way as well:
1. Modify database schemas to prepare to add Geo support for Cool Widgets
1. Implement Geo support of Cool Widgets behind a feature flag
1. Release Geo support of Cool Widgets
It is also a good idea to first open a proof-of-concept merge request. It can be helpful for working out kinks and getting initial support and feedback from the Geo team. As an example, see the [Proof of Concept to replicate Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56423).
You can look into the following examples of MRs for implementing replication/verification for a new blob type:
-[Add db changes](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60935) and [add verification for MR diffs using SSF](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63309)
-[Verify Terraform state versions](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/58800)
### Modify database schemas to prepare to add Geo support for Cool Widgets
#### Add the registry table to track replication and verification state
Geo secondary sites have a [Geo tracking database](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo.md#tracking-database) independent of the main database. It is used to track the replication and verification state of all replicables. Every Model has a corresponding "registry" table in the Geo tracking database.
- [ ] Create the migration file in `ee/db/geo/migrate`:
- [ ] Replace the contents of the migration file with the following. Note that we cannot add a foreign key constraint on `cool_widget_id` because the `cool_widgets` table is in a different database. The application code must handle logic such as propagating deletions.
# To optimize performance of CoolWidgetRegistry.verification_failed_batch
t.index:verification_retry_at,
name: :cool_widget_registry_failed_verification,
order: "NULLS FIRST",
where: "((state = 2) AND (verification_state = 3))"
# To optimize performance of CoolWidgetRegistry.needs_verification_count
t.index:verification_state,
name: :cool_widget_registry_needs_verification,
where: "((state = 2) AND (verification_state = ANY (ARRAY[0, 3])))"
# To optimize performance of CoolWidgetRegistry.verification_pending_batch
t.index:verified_at,
name: :cool_widget_registry_pending_verification,
order: "NULLS FIRST",
where: "((state = 2) AND (verification_state = 0))"
end
end
end
```
-[ ] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
-[ ] Add the new table to the [database dictionary](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/database/database_dictionary.md) defined in [`ee/db/geo/docs/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/db/geo/docs):
```yaml
table_name:cool_widget_registry
description:Description example
introduced_by_url:Merge request link
milestone:Milestone example
feature_categories:
-Feature category example
classes:
-Class example
gitlab_schema:gitlab_geo
```
- [ ] Run Geo tracking database migrations:
```shell
bin/rake db:migrate:geo
```
- [ ] Be sure to commit the relevant changes in `ee/db/geo/structure.sql` and the file created under `ee/db/geo/schema_migrations`
### Add verification state fields on the Geo primary site
The Geo primary site needs to checksum every replicable so secondaries can verify their own checksums. To do this, Geo requires fields on the Model. Add verification state fields to a separate table. Consult a database expert if needed.
where: "(verification_state = 0 OR verification_state = 3)",
name: NEEDS_VERIFICATION_INDEX_NAME
end
end
defdown
drop_table:cool_widget_states
end
end
```
-[ ] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
-[ ] If `cool_widgets` is a high-traffic table, follow [the database documentation to use `with_lock_retries`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/migration_style_guide.md#when-to-use-the-helper-method)
-[ ] Add the new table to the [database dictionary](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/database/database_dictionary.md) defined in [`db/docs/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/db/docs):
```yaml
---
table_name:cool_widget_states
description:Separate table for Cool Widget verification states
- [ ] Ensure `CoolWidget.replicables_for_current_secondary` is well-tested. Search the codebase for `replicables_for_current_secondary` to find examples of parameterized table specs. You may need to add more `FactoryBot` traits.
- [ ] Add the following shared examples to `ee/spec/models/ee/cool_widget_spec.rb`:
```ruby
include_examples'a replicable model with a separate table for verification state'do
let(:verifiable_model_record){build(:cool_widget)}# add extra params if needed to make sure the record is in `Geo::ReplicableModel.verifiables` scope
let(:unverifiable_model_record){build(:cool_widget)}# add extra params if needed to make sure the record is NOT included in `Geo::ReplicableModel.verifiables` scope
end
```
- [ ] Create `ee/app/replicators/geo/cool_widget_replicator.rb`. Implement the `#carrierwave_uploader` method which should return a `CarrierWave::Uploader`, and implement the class method `.model` to return the `CoolWidget` class:
```ruby
# frozen_string_literal: true
moduleGeo
classCoolWidgetReplicator<Gitlab::Geo::Replicator
include::Geo::BlobReplicatorStrategy
extend::Gitlab::Utils::Override
defself.model
::CoolWidget
end
defcarrierwave_uploader
model_record.file
end
override:verification_feature_flag_enabled?
defself.verification_feature_flag_enabled?
# We are adding verification at the same time as replication, so we
# don't need to toggle verification separately from replication. When
# the replication feature flag is off, then verification is also off
# (see `VerifiableReplicator.verification_enabled?`)
true
end
end
end
```
- [ ] Generate the feature flag definition file by running the feature flag commands and following the command prompts:
```shell
bin/feature-flag --ee geo_cool_widget_replication --type development --group'group::geo'
```
- [ ] Add this replicator class to the method `replicator_classes` in
`ee/lib/gitlab/geo.rb`:
```ruby
REPLICATOR_CLASSES=[
::Geo::PackageFileReplicator,
::Geo::CoolWidgetReplicator
]
```
- [ ] Create `ee/spec/replicators/geo/cool_widget_replicator_spec.rb` and perform the necessary setup to define the `model_record` variable for the shared examples:
# Geo::VerifiableReplicator#after_verifiable_update tries to verify
# the replicable async and marks it as verification started when the
# model record is created/updated.
#
after(:create)do|instance,_|
instance.verification_failed!
end
end
end
end
```
If there is not an existing factory for the object in `spec/factories/cool_widgets.rb`, wrap the traits in `FactoryBot.create` instead of `FactoryBot.modify`
[ ] Make sure the factory supports the `:remote_store` trait. If not, add something like
- [ ] Make sure the factory also allows setting a `project` attribute. If the model does not have a direct relation to a project, you can use a `transient` attribute. Check out `spec/factories/merge_request_diffs.rb` for an example.
-[ ] Following [the example of Merge Request Diffs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63309) add a `Geo::CoolWidgetState` model in `ee/app/models/geo/cool_widget_state.rb`:
- [ ] Add integration tests for providing CoolWidget registry data to the frontend via the GraphQL API, by duplicating and modifying the following shared examples in `ee/spec/requests/api/graphql/geo/registries_spec.rb`:
```ruby
it_behaves_like'gets registries for',{
field_name: 'coolWidgetRegistries',
registry_class_name: 'CoolWidgetRegistry',
registry_factory: :geo_cool_widget_registry,
registry_foreign_key_field_name: 'coolWidgetId'
}
```
To allow the new replicable to resync and reverify via GraphQL:
- [ ] Add the `CoolWidgetRegistryType` to the `GEO_REGISTRY_TYPE` constant in `ee/app/graphql/types/geo/registrable_type.rb`:
Individual Cool Widget replication and verification data should now be available via the GraphQL API.
#### Step 4. Handle batch destroy
If batch destroy logic is implemented for a replicable, then that logic must be "replicated" by Geo secondaries. The easiest way to do this is use `Geo::BatchEventCreateWorker` to bulk insert a delete event for each replicable.
For example, if `FastDestroyAll` is used, then you may be able to [use `begin_fast_destroy` and `finalize_fast_destroy` hooks, like we did for uploads](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69763).
Or if a special service is used to batch delete records and their associated data, then you probably need to [hook into that service, like we did for job artifacts](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79530).
As illustrated by the above two examples, batch destroy logic cannot be handled automatically by Geo secondaries without restricting the way other teams perform batch destroys. It is up to you to produce `Geo::BatchEventCreateWorker` attributes before the records are deleted, and then enqueue `Geo::BatchEventCreateWorker` after the records are deleted.
- [ ] Ensure that any batch destroy of this replicable is replicated to secondary sites
- [ ] Regardless of implementation details, please verify in specs that when the parent object is removed, the new `Geo::Event` records are created:
```ruby
describe'#destroy'do
subject{create(:cool_widget)}
context'when running in a Geo primary node'do
let_it_be(:primary){create(:geo_node,:primary)}
let_it_be(:secondary){create(:geo_node)}
it'logs an event to the Geo event log when bulk removal is used',:sidekiq_inlinedo
- [ ] Include a comment mentioning that the change is based on a documented template.
-[ ] `replicables_for_current_secondary` and `available_replicables` may differ per Model. If their queries are new, then add [query plans](https://docs.gitlab.com/ee/development/database_review.html#query-plans) to the MR description. An easy place to gather SQL queries is your GDK's `log/test.log` when running tests of these methods.
### Release Geo support of Cool Widgets
- [ ] In the rollout issue you created when creating the feature flag, modify the Roll Out Steps:
- [ ] Cross out any steps related to testing on production GitLab.com, because Geo is not running on production GitLab.com at the moment.
- [ ] Add a step to `Test replication and verification of Cool Widgets on a non-GDK-deployment. For example, using GitLab Environment Toolkit`.
- [ ] Add a step to `Ping the Geo PM and EM to coordinate testing`. For example, you might add steps to generate Cool Widgets, and then a Geo engineer may take it from there.
- [ ] In `ee/config/feature_flags/development/geo_cool_widget_replication.yml`, set `default_enabled: true`
- [ ] In `ee/app/graphql/types/geo/geo_node_type.rb`, remove the `alpha` option for the released type:
description: 'Find Cool Widget registries on this Geo node. '\
'Ignored if `geo_cool_widget_replication` feature flag is disabled.',
alpha: {milestone: '15.5'}# Update the milestone
```
- [ ] Run `bundle exec rake gitlab:graphql:compile_docs` after the step above to regenerate the GraphQL docs.
-[ ] Add a row for Cool Widgets to the `Data types` table in [Geo data types support](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/datatypes.md#data-types)
-[ ] Add a row for Cool Widgets to the `Limitations on replication/verification` table in [Geo data types support](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/datatypes.md#limitations-on-replicationverification). If the row already exists, then update it to show that Replication and Verification is released in the current version.
<!-- What problem do we solve? Try to define the who/what/why of the opportunity as a user story. For example, "As a (who), I want (what), so I can (why/value)." -->
## Proposal
<!-- Use this section to explain the feature and how it will work. It can be helpful to add technical details, design proposals, and links to related epics or issues. -->
<!-- Please add a label for the type of feature as per https://about.gitlab.com/handbook/engineering/metrics/#work-type-classification -->
<!-- Use this section to explain the feature and how it will work. It can be helpful to add technical details, design proposals, and links to related epics or issues. -->
<!-- Please add a label for the type of maintenance as per https://about.gitlab.com/handbook/engineering/metrics/#work-type-classification -->
Clearly state the scope of the problem, and how it affects GitLab.com
-->
## Impact
<!--
- Quantify the effect of the problem to help ensure that correct prioritization occurs.
- Include costs to availability. The Incident Budget Explorer dashboard can help here.
- Include the number of times alerts have fired owing to the problem, how much time was spent dealing with the problem, and how many people were involved.
- Link to affected incidents, and cross-reference them as related issues.
- Include screenshots of visualization from Grafana or Kibana.
- Always include a permalink to the source of the screenshot so that others can investigate further.
-->
## Recommendation
<!--
Provide a clear, unambiguous, self-contained solution to the problem.
-->
## Verification
<!--
Provide a method for validating that the original issue still exists.
Having a way of checking validity can save on a great deal of back-and-forth discussion between Infradev Triage participants including Engineering Managers, Directors and Product Managers and make space for other non-resolved issues to get scheduled sooner.
Ideally, provide a link to a Thanos query or an ELK query and clear instructions on how to interpret the results to determine whether the problem is still occurring.
<!-- This template is used for proposing changes to the left sidebar contextual navigation. This could include additions, removals, or general changes to overall hierarchy.-->
### Proposal
<!-- Use this section to explain the proposed changes, including details around usage and business drivers. -->
#### Other locations that were considered
<!-- Include other design patterns or places you considered for this feature besides navigation. -->
### Checklist
-[ ] Review the [handbook page for navigation](https://handbook.gitlab.com/handbook/product/ux/navigation/).
- [ ] Add relevant information to the issue description detailing your proposal, including usage and business drivers.
- [ ] List at least two other places you considered to introduce your feature.
- [ ] Add relevant designs to the Design Management area of the issue.
-[ ] Ensure your UI suggestion align with the [Documentation Style Guide](https://docs.gitlab.com/ee/development/documentation/styleguide/).
- [ ] Engage ~"Technical Writing". They can help craft a term that best describes the feature(s) you’re proposing.
-[ ] Follow the [product development workflow](https://handbook.gitlab.com/handbook/product-development-flow/#validation-phase-2-problem-validation) validation process to ensure you are solving a well understood problem and that the proposed change is understandable and non-disruptive to users. Navigation-specific research is mandatory for additions or when restructuring.
-[ ] Engage the [Foundations Product Manager](https://handbook.gitlab.com/handbook/product/categories/#foundations-group) for approval. The Foundations DRI (@jtucker_gl) will work with UX partners in product design, research, and technical writing, as applicable.
-[ ] Consider whether you need to [communicate the change somehow](https://design.gitlab.com/patterns/navigation#messaging-changes-to-users), or if you will have an interim period in the UI where your item will live in more than one place.
-[ ] Ensure engineers are familiar with the [implementation steps for navigation](https://docs.gitlab.com/ee/development/navigation_sidebar.html#navigation-sidebar).
<!-- title format: Operational Readiness Review - {`new component name`}
When we add a new component to our platform, we should keep in mind the non-functional requirements and operational needs we are adding to our platform. While
we want to move quickly, we also want to ensure:
- We know what is being added.
- If we can operate it.
- The it meets our general legal, compliance, and operational standards.
-->
## Links
<!-- Provide Links to the Epic, issue, handbook page, and/or blueprint. -->
## Type of new component
<!-- List the type of new component from one of following values:
- New third party SaaS service
- New data store (that is not a SaaS service)
- New service
- New software dependency
- New programming language
- New development and testing framework
-->
## Review process
To help us to make concise and sustainable decision when converting the prototype to a product, it is highly recommended that the PM and EM start with a
self assessment with this checklist, and then engage the appropriate groups and/or departments to review if anything in doubt. This is **NOT** a gating
process, rather a friendly checklist to ensure the success of the new component.
The review should be quick and with the least number of steps. The review will likely have 2 DRIs as reviewers for each component to ensure we can move
quickly and handle any out of office (OOO).
## Checklist
Complete common and the appropriate checklists per the type of new component mentioned above (Skip any item if not applicable).
### Common
- [ ] Definition and Goals
- [ ] What the component does and what values it provides from the external and internal customer's perspective?
- [ ] Is any existing component capable for the same use case? If so, why is the new component required?
- [ ] What is the usage estimation in both .com and self-managed?
- [ ] Who are the development and operation DRI groups?
- [ ] Legal and Security
- [ ] Are you conducting a legal and compliance review with legal department?
- [ ] Are you conducting an in-depth security review of the component with security department?
- [ ] What type of license do they use?
- [ ] What is the data classification this component will process?
- [ ] Support
-[ ] Have you involved the Customer Support Team by drafting a [Support Readiness Issue](https://gitlab.com/gitlab-com/support/support-team-meta/-/issues/new?issuable_template=Support%20Readiness)? And complete it before releasing to customers.
- [ ] Business
- [ ] Margin impact - (sheet to be created)
- [ ] What is the estimated cost of the component and associated support including infrastructure operations if any?
- [ ] Architecture
- [ ] Does the component support auto-scaling? If not, how does it handle a sudden traffic increasing?
- [ ] What are the dependencies between existing GitLab services and this component?
- [ ] What is the infrastructure requirement?
- [ ] Is this SaaS only, or will it also be supported for Self-Managed and Dedicated?
- [ ] Development, Testing, Deployment, and Operation
-[ ] Complete the [production readiness review](https://about.gitlab.com/handbook/engineering/infrastructure/production/readiness/).
- [ ] As the owner, are you confident to manage and maintain the new component end to end (E2E)? You can review below typical considerations as a guidance.
- [ ] Talent pool, e.g. existing engineers, maintainers, and future hiring opportunities. <br/>
- [ ] Testing, e.g. end-to-end, dependencies, performance. <br/>
- [ ] Operational considerations, e.g. observability, hosting knowledge, etc. <br/>
</details>
### New GitLab service
- [ ] Legal and Security
- [ ] Is there any specific security standard and compliance required before deploying to production? If so, what needs to be done?
- [ ] Architecture
- [ ] Complete `New data store, third party dependency` checklist as well if a new dependency is used
- [ ] Complete `New programming language, development, and testing framework` checklist as well if a new programming language, development, and/or testing framework is used
### New data store, third party dependency
- [ ] Legal and Security
-[ ] What is the [classification](https://about.gitlab.com/handbook/security/data-classification-standard.html#data-classification-levels) of data stored in the data store?
- [ ] Have they got any security standards to meet our and/or our customers' requirements? (i.e. FIPS and/or Fed-RAMP) If not, what needs to be done?
- [ ] Development, Testing, Deployment, and Operation
- [ ] What integration types do they provide, e.g. SaaS and/or self-hosting?
- [ ] Is rate limit possible?
- [ ] What is the cadence of version upgrades?
- [ ] What is their defect fix and security patch turnaround time?
### New programming language, development, and testing framework
- [ ] Is there a mature ecosystem that provides tooling (profiling, debugging, etc.) and 3rd party libraries?
<!-- Custom Roles documentation: See https://docs.gitlab.com/ee/user/custom_roles.html -->
<!-- Available Permissions: https://docs.gitlab.com/ee/user/custom_roles/abilities.html -->
<!-- Example of Permission Request: See https://gitlab.com/gitlab-org/gitlab/-/issues/442851 -->
## Proposed Permission
<!-- Describe the real-world use case for the permissions you want to introduce, including why you need the requested level of granularity, and why the available default roles are not sufficient.
Example: Group Owners have the ability to manage team members. This leads to organizations elevating a subset of users who need to manage these settings to Owners, so as a consequence these users can edit other group or project settings without needing to. Adding the `manage team member` custom permission will allow an organization to create a custom role, such as Developer + this permission, which reduces unneeded Owners and Maintainers in their organizations.
-->
## Proposal and User Experience
<!-- State what actions a user with this permission can take at a group and project level. -->
| Group Actions | Project Actions |
| ------------- | --------------- |
| Actions | Actions |
| Actions | Actions |
### Views+Workflows include:
<!-- State what a user with this permission can see in terms of workflows from a UI perspective. For example, for Runners, a user can see:
Base + permission: Group-> Build -> Runners
Base + permission: Projects -> Settings > CI/CD > Runners
-->
- [ ] Base + Permission
### Impacted APIs
<!-- Include a list of API's impacted for the permission -->
#### Documentation
<!-- Permissions for Custom Roles is auto-generated. A title and description should be included for the proposal. Also if the feature has documentation, there is a "Prerequisities" section under a feature that highlight required permissions. The permission for custom role should be documented and appended next to the required default role.
Example:
- Permission Title: "Manage Variables"
- Permission Description: "Create, read, update, and delete Variables"
Prerequisites:
You must be a project member with the Maintainer role or have a [custom role](link).
(_NOTE: This section can be removed when the issue is ready for creation_)
- [ ] Ensure that issue title is concise yet descriptive
-[ ] Add `Frontend :` or `Backend: ` per group [naming conventions](https://about.gitlab.com/handbook/engineering/development/ops/verify/pipeline-authoring/#splitting-issues)
- [ ] Ensure the issue containing the feature or change proposal and related discussions is linked as related to this implementation issue.
- [ ] Aside from default labeling, please make sure to include relevant labels for `type::`, `workflow::`, and `~frontend`/`~backend` labeling.
- [ ] Issues with user-facing changes should include the `~UX` label.
-->
## Summary
## Proposal
## Confirm purpose and User Reception (how does this benefit the user?)
## Additional details
<!--
_NOTE: If the issue has addressed all of these questions, this separate section can be removed._
-->
Some relevant technical details, if applicable, such as:
- Does this need a ~"feature flag"?
- Does there need to be an associated ~"instrumentation" issue created related to this work?
- Is there an example response showing the data structure that should be returned (new endpoints only)?
- What permissions should be used?
- Is this EE or CE?
- [ ] EE
- [ ] CE
- Additional comments:
## Implementation Table
<!--
_NOTE: If the issue is not part of an epic, the implementation table can be removed. If it is part of an epic, make sure that the implementation table below mirrors the corresponding epic's implementation table content._
-->
| Group | Issue Link |
| ------ | ------ |
| ~backend | :point_left: You are here |
| ~frontend | [#123123](url) |
<!--
## Documentation
_NOTE: This section is optional, but can be used for easy access to any relevant documentation URLs._
(_NOTE: This section can be removed when the issue is ready for creation_)
- [ ] Ensure that issue title is concise yet descriptive
- [ ] Add "~frontend", "~backend", or "~documentation" labels as appropriate
- [ ] Ensure the issue containing the feature or change proposal and related discussions is linked as related to this implementation issue.
- [ ] Aside from default labeling, please make sure to include relevant labels for `type::`, `workflow::`, and `~frontend`/`~backend` labeling.
- [ ] Issues with user-facing changes should include the `~UX` label.
-->
## Summary
## Proposal
## Additional details
<!--
_NOTE: If the issue has addressed all of these questions, this separate section can be removed._
-->
Some relevant technical details, if applicable, such as:
- Does this need a ~"feature flag"?
- Does there need to be an associated ~"instrumentation" issue created related to this work?
- Is there an example response showing the data structure that should be returned (new endpoints only)?
- What permissions should be used?
- Is this EE or CE?
- [ ] EE
- [ ] CE
- Additional comments:
## Implementation Table
<!--
_NOTE: If the issue is not part of an epic, the implementation table can be removed. If it is part of an epic, make sure that the implementation table below mirrors the corresponding epic's implementation table content._
-->
| Group | Issue Link |
| ------ | ------ |
| ~backend | :point_left: You are here |
| ~frontend | [#123123](url) |
<!--
## Documentation
_NOTE: This section is optional, but can be used for easy access to any relevant documentation URLs._
(_NOTE: This section can be removed when the issue is ready for creation_)
- [ ] Ensure that the issue title is concise yet descriptive.
-[ ] Add `Frontend :` or `Backend :` to the title per group [naming conventions](https://about.gitlab.com/handbook/engineering/development/ops/verify/pipeline-security/#splitting-issues)
- [ ] Ensure the issue containing the feature or change proposal and related discussions is linked as related to this implementation issue.
- [ ] Aside from default labeling, please make sure to include relevant labels for `~type::`, `~workflow::`, and `~frontend` or `~backend`.
- [ ] Issues with user-facing changes should include the `~UX` label, and `~documentation` if docs changes will be required.
*This template is meant to be a reference tool. Not all sections are applicable to each feature, bug, or maintenance item. Use your best judgment when completion the sections below.*
-->
## Summary
<!-- Briefly describe the issue. -->
### Why this matters and how we measure
<!-- What is the value to the customer or our business? Does this align with our OKRs? If we need to create or update existing instrumentation, please note here. -->
### User Stories
<!--
A user story is a requirement for any functionality or feature and follows this format:
- _As a `<user role/customer>`, I want to `<JTBD>` so that I can `<achieve a benefit or result>`._
Please try to include one user story for the main [persona](https://handbook.gitlab.com/handbook/product/personas/#list-of-user-personas) who needs this feature.
-->
## Proposal
<!-- Try to keep the proposal limited in scope. Plan for iterations, create follow up issues as required and add them as related. -->
## Performance Considerations
<!-- Performance concerns to be aware of and monitor when implementing the issue.-->
## Out of Scope
<!-- Include this section for specific use cases that are out of scope / out of bounds for this specific issue. -->
## Acceptance Criteria
<!-- This needs to be true or demonstrable to consider this specific issue complete. Keep this dependent on other issues when possible -->
## Additional details
<!--
_NOTE: If the issue has addressed all of these questions, this separate section can be removed._
-->
Some relevant technical details, if applicable, such as:
- Does this need a ~"feature flag"?
- Does there need to be an associated ~"instrumentation" issue created related to this work?
- Is there an example response showing the data structure that should be returned (new endpoints only)?
- What permissions should be used?
- Which tier(s) is this for?
- [ ] ~"GitLab Ultimate"
- [ ] ~"GitLab Premium"
- [ ] ~"GitLab Free"
- Additional comments:
## Implementation Table
<!--
_NOTE: Use this to indicate all dependent issues related to this one which are required for launch._
-->
| Group | Issue Link |
| ------ | ------ |
| ~backend | :point_left: You are here |
| ~frontend | [Issue Title](url) |
| ~documentation | [Issue Title](url) |
| Instrumentation | [Issue Title](url) |
<!--
## Documentation
_NOTE: This section is optional, but can be used for easy access to any relevant documentation URLs._
-->
## Links/References
/label ~"group::pipeline security"
/milestone %Backlog
<!-- select the correct category (and feature label if applicable) below:
/label ~"category:Build Artifacts"
/label ~"category:Secrets Management"
/label ~"ci variables"
/label ~"ci job token"
-->
<!-- select the appropriate licence below (Use the highest tier applicable):
<!-- This template is used as a starting point for understanding and articulating a customer problem.
Learn more about it in the handbook: https://about.gitlab.com/handbook/product-development-flow/#validation-phase-2-problem-validation
-->
## Problem Statement
<!-- What is the problem we hope to validate? Reference how to write a real customer problem statement at https://productcoalition.com/how-to-write-a-good-customer-problem-statement-a815f80189ba for guidance. -->
## Reach
<!-- Please describe who suffers from this problem. Consider referring to our personas, which are described at https://handbook.gitlab.com/handbook/product/personas/ -->
<!-- Please also quantify the problem's reach using the following values, considering an aggregate across GitLab.com and self-managed:
10.0 = Impacts the vast majority (~80% or greater) of our users, prospects, or customers.
6.0 = Impacts a large percentage (~50% to ~80%) of the above.
3.0 = Significant reach (~25% to ~50%).
1.5 = Small reach (~5% to ~25%).
0.5 = Minimal reach (Less than ~5%). -->
## Impact
<!-- How do we positively impact the users above and GitLab's business by solving this problem? Please describe briefly, and provide a numerical assessment:
3.0 = Massive impact
2.0 = High impact
1.0 = Medium impact
0.5 = Low impact
0.25 = Minimal impact -->
## Confidence
<!-- How do we know this is a problem? Please provide and link to any supporting information (e.g. data, customer verbatims) and use this basis to provide a numerical assessment on our confidence level in this problem's severity:
100% = High confidence
80% = Medium confidence
50% = Low confidence -->
## Effort
<!-- How much effort do we think it will be to solve this problem? Please include all counterparts (Product, UX, Engineering, etc) in your assessment and quantify the number of person-months needed to dedicate to the effort.
For example, if the solution will take a product manager, designer, and engineer two weeks of effort - you may quantify this as 1.5 (based on 0.5 months x 3 people). -->
## Definition of Done
- [ ] The problem is well understood by the PM to have an understanding summarized in a RICE score
- [ ] The problem is well understood by the PM to decide if they want to move forward with this idea or drop it
- [ ] The problem is well described and detailed with necessary requirements for product design to understand the problem
- [ ] The problem is well described and detailed with necessary requirements for engineering to understand the problem
## Research Issue
<!-- Link to the Problem Validation Research issue that will be executed by the UX Researcher. https://gitlab.com/gitlab-org/ux-research/ -->
- [ ] The root cause of the problem is identified.
- [ ] The surface of the problem is as small as possible.
## What are the potential solutions?
<!--
Please provide potential solutions here. Example solutions could be:
- Dogfood a feature.
- Refactor/improve some workflow code.
- Throw more money at the problem.
Please provide pros/cons and a weight estimate for each solution.
-->
- [ ] All potential solutions are listed.
- [ ] A solution has been chosen for the first iteration: `PUT THE CHOSEN SOLUTION HERE`
## Verify that the solution has improved the situation
<!--
Ideally, looking at the charts from the first part, we should see an improvement
after the implementation is merged/deployed/released.
-->
- [ ] The solution improved the situation.
- If yes, check this box and close the issue. Well done! :tada:
- Otherwise, create a new "Productivity Improvement" issue. You can re-use the description from this issue, but another solution should be chosen this time.
- The relative path of the failing spec file in the title, e.g. if the login
test fails, include `qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb` in the title.
This is required so that existing issues can easily be found by searching for the spec file.
- If the issue is about multiple test failures, include the path for each failing spec file in the description.
- A link to the failing job.
- The stack trace from the job's logs in the "Stack trace" section below.
- A screenshot (if available), and HTML capture (if available), in the "Screenshot / HTML page" section below.
- A link to the corresponding test case(s) in the summary.
--->
### Summary
Failing job(s):
Failing spec(s):
Corresponding test case(s):
### Stack trace
```ruby
PUTSTACKTRACEHERE
```
### Screenshot / HTML page
<!--
Attach the screenshot and HTML snapshot of the page from the job's artifacts:
1. Download the job's artifacts and unarchive them.
1. Open the `gitlab-qa-run-2020-*/gitlab-{ce,ee}-qa-*/{,ee}/{api,browser_ui}/<path to failed test>` folder.
1. Select the `.png` and `.html` files that appears in the job logs (look for `HTML screenshot: /path/to/html/page.html` / `Image screenshot: `/path/to/html/page.png`).
1. Drag and drop them here.
Note: You don't need to include a screenshot if the information it contains can be included as text. Include the text instead.
E.g., error 500/404, "Retry later" errors, etc.
If you include multiple screenshots it can be helpful to hide all but the first in a details/summary element, to avoid excessive scrolling:
<details><summary>Expand for screenshot</summary>
drag and drop the screenshot here
</details>
-->
### Possible fixes
<!-- Default due date. -->
/due in 2 weeks
<!-- Base labels. -->
/label ~Quality ~QA ~test
<!-- Work classification type label, please apply ignore type label until the investigation is complete and an [issue type](https://about.gitlab.com/handbook/engineering/metrics/#work-type-classification) is determined.-->
/label ~"type::ignore"
<!-- Test failure type label, please use just one.-->
