name:review/${CI_COMMIT_REF_SLUG}${SCHEDULE_TYPE}# No separator for SCHEDULE_TYPE so it's compatible as before and looks nice without it
action:stop
resource_group:review/${CI_COMMIT_REF_SLUG}${SCHEDULE_TYPE}# CI_ENVIRONMENT_SLUG is not available here and we want this to be the same as the environment
resource_group:review/${CI_COMMIT_REF_SLUG}${SCHEDULE_TYPE}# CI_ENVIRONMENT_SLUG is not available here and we want this to be the same as the environment
stage:review
needs:
-job:rails-production-server-boot-puma-example
optional:true
-job:rails-production-server-boot-puma-cng
optional:true
-job:build-assets-image
artifacts:false
# We do not want to have ALL global variables passed as trigger variables,
# as they cannot be overridden. See this issue for more context:
# When DB schema changes, many migrations spec may be affected. However, the test mapping from Crystalball does not map db change to a specific migration spec well.
# See https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68840.
# When DB schema changes, many migrations spec may be affected. However, the test mapping from Crystalball does not map db change to a specific migration spec well.
# See https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68840.
# If the MR isn't approved, only run when jest predictive jobs would
-<<:*if-merge-request-not-approved
changes:*frontend-predictive-patterns
-<<:*if-merge-request-not-approved
when:never
# From .frontend:rules:default-frontend-jobs
-<<:*if-default-refs
changes:*code-backstage-patterns
.frontend:rules:upload-frontend-fixtures:
rules:
# The new strategy to upload fixtures as generic packages is experimental and can be disabled by removing the `REUSE_FRONTEND_FIXTURES_ENABLED` variable
# When DB schema changes, many migrations spec may be affected. However, the test mapping from Crystalball does not map db change to a specific migration spec well.
# See https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68840.
# When DB schema changes, many migrations spec may be affected. However, the test mapping from Crystalball does not map db change to a specific migration spec well.
# See https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68840.
# The template generates jobs that trigger child pipelines for gems vendored in the main GitLab project under `gems/`.
#
# Inputs:
# - `gem_name`: The name of the gem, i.e. if the gem is located at `gems/gitlab-rspec`, `gem_name` should be set to `gitlab-rspec`.
# - `gem_path_prefix`: The prefix of the gem path, i.e. if the gem is located at `vendor/gems/gitlab-rspec`, `gem_path_prefix` should be set to `vendor/gems/`. Defaults to `gems/`.
To propose an AI experiment, focus on completing the “Experiment” section first. As you refine the idea and gather feedback on your experiment, progress to the Beta section to define how it will evolve, when ready, progress to the “Generally Available release” section to define how it will evolve GA capability. It's important that we link Experiment to Beta to GA release. Feel free to add sections, but the existing ones must be kept and completed.
You can choose how to get started with this template. For example, the proposal can start as an issue, and then be promoted to an epic to house all the work related to the Experiment, Beta, and GA release. If you prefer to start with an epic, you have to manually apply the proposal template. Regardless, if the experiment is eventually prioritized for development, the template content will need to appear in a top-level epic so it can be tracked alongside other prioritized AI experiments.
The title should be something that is easily understood that quickly communicates the intent of the project allowing team members to easily understand and recognize the expected work that will be done. A proposal title should combine the beneficiary of the feature/UI, the job it will allow them to accomplish (see https://about.gitlab.com/handbook/product/ux/jobs-to-be-done/#how-to-write-a-jtbd), and their expected outcome when the work is delivered. Well-defined statements are concise without sacrificing the substance of the proposal so that anyone can understand it at a glance. (e.g. {Reduce the effort} {for security teams} {when prioritizing business-critical risks in their assets}).
| [Problem validation](https://about.gitlab.com/handbook/product/ai/ux-maturity/#validation-problem-validation)<br>How well do we understand the problem? | [Mix of evidence and assumptions](https://about.gitlab.com/handbook/product/ai/ux-maturity/#questions-to-ask) | <!-- Acceptable answers: Yes, Somewhat or Somewhat, Somewhat --> |
| [Solution validation](https://about.gitlab.com/handbook/product/ai/ux-maturity/#validation-solution-validation)<br>How usable is the solution? | [Usability testing](https://about.gitlab.com/handbook/product/ux/ux-scorecards/#option-b-perform-a-formative-evaluation), Grade C | <!-- Acceptable: >80% and grade C --> |
| [Improve](https://about.gitlab.com/handbook/product/ai/ux-maturity/#build-improve)<br>How successful is the solution? | Quality goals set by the team are reached. | <!-- Acceptable answers: :white_check_mark: Reached all quality goals for this phase. --> |
| [Design standards](https://about.gitlab.com/handbook/product/ai/ux-maturity/#design-standards) adherence<br>How compliant is the solution with our design standards? | Should adhere to ([Pajamas](https://design.gitlab.com/), [checklist](https://docs.gitlab.com/ee/development/contributing/design.html#checklist)) | <!-- Acceptable: Mostly adheres to design standards --> |
_This section should be completed prior to beginning work on the Beta experience._
<!-- DO NOT REMOVE THIS SECTION
Although the initial focus is on the “Experiment” section, do not remove this “Beta” section. It's important that we link Experiment to Beta release. Fill this section in as you progress.
<!-- What is the [Main Job story](https://about.gitlab.com/handbook/product/ux/jobs-to-be-done/#how-to-write-a-jtbd) that this proposal was derived from? (e.g. When I am on triage rotation, I want to address all the business-critical risks in my assets, So I can minimize the likelihood of my organization being compromised by a security breach.) -->
_What are the small jobs this feature is solving for?_
### Assumption
_What assumptions are you making about this problem and the solution?_
### Proposal updates/additions
<!-- Explain any changes or updates to the original proposal from the Experiment, including details around usage, business drivers, and reasonings that drove the updates/additions. -->
### Problem validation
_What validation exists that customers have this problem?_
<!-- Refer to https://about.gitlab.com/handbook/product/ux/ux-research/research-in-the-AI-space/#guideline-1-problem-validation---identify-and-understand-user-needs --- to help identify and understand user needs -->
### Business objective
_What business objective will be achieved with this proposal?_
<!-- Objectives (from a business point of view) that will be achieved upon completion. (For instance, Increase engagement by making the experience efficient while reducing the chances of users overlooking high-priority items. -->
### Requirements
_What tasks or actions should the user be capable of performing with this feature?_
<!-- Requirements can be taken from existing features or design issues used to build this proposal. Any related issues should be linked with this issue in the Feature/solution issues section below. They are more granular validated needs, goals, and additional details that the proposal encompasses. -->
### The user needs to be able to:
- ...
- ...
#### Success
_How will you measure whether this Beta is a success?_
<!-- Consider how successful the solution is by looking beyond feature usage as the success metric. Instead consider how useful, efficient, effective, satisfying, and learnable was the feature. The Product Development Flow recommends outcomes and potential activities to create a combined and ongoing quantitative and qualitative feedback loop to evaluate feature success. -->
**UX maturity requirements** _[Beta to GA](https://about.gitlab.com/handbook/product/ai/ux-maturity/#criteria-and-requirements)_
| Criteria | Minimum Requirement | Assessment for GA |
| [Problem validation](https://about.gitlab.com/handbook/product/ai/ux-maturity/#validation-problem-validation)<br>How well do we understand the problem? | [Mix of evidence and assumptions](https://about.gitlab.com/handbook/product/ai/ux-maturity/#questions-to-ask) | <!-- Acceptable answers: Yes, Yes --> |
| [Solution validation](https://about.gitlab.com/handbook/product/ai/ux-maturity/#validation-solution-validation)<br>How usable is the solution? | [Usability testing](https://about.gitlab.com/handbook/product/ux/ux-scorecards/#option-b-perform-a-formative-evaluation) and [Heuristic evaluation](https://about.gitlab.com/handbook/product/ux/ux-scorecards/#option-a-conduct-a-heuristic-evaluation), Avg. task pass rate >80%, Grade B | <!-- Acceptable: >80% and grade B --> |
| [Improve](https://about.gitlab.com/handbook/product/ai/ux-maturity/#build-improve)<br>How successful is the solution? | Quality goals set by the team are reached. | <!-- Acceptable answers: :white_check_mark: Reached all quality goals for this phase. --> |
| [Design standards](https://about.gitlab.com/handbook/product/ai/ux-maturity/#design-standards) adherence<br>How compliant is the solution with our design standards? | Should adhere to ([Pajamas](https://design.gitlab.com/), [checklist](https://docs.gitlab.com/ee/development/contributing/design.html#checklist)) | <!-- Acceptable: Completely adheres to design standards --> |
Although the initial focus is on the “Experiment” section, do not remove this “Generally Available” section. It's important that we link Beta to GA release. Fill this section in as you progress.
-->
### Assumption
_What assumptions are you making about this problem and the solution?_
### Proposal updates/additions
<!-- Explain any changes or updates to the original proposal from the experiment, including details around usage, business drivers, and reasonings that drove the updates/additions. -->
### Problem validation
_What validation exists that customers have this problem?_
<!-- Refer to https://about.gitlab.com/handbook/product/ux/ux-research/research-in-the-AI-space/#guideline-1-problem-validation --- to help identify and understand user needs -->
### Requirements
_What tasks or actions should the user be capable of performing with this feature?_
<!-- Requirements can be taken from existing features or design issues used to build this proposal. Any related issues should be linked with this issue in the Feature/solution issues section below. They are more granular validated needs, goals, and additional details that the proposal encompasses. -->
> ⚠️ Related feature and research issues should be linked in the related issues section (Delete this line when this is done)
#### The user needs to be able to:
- ...
- ...
## Checklist
### Experiment
<details><summary> Issue information </summary>
- [ ] Add information to the issue body about:
- [ ] The user problem being solved
- [ ] Why the solution hypothesis solves this problem
- [ ] Your assumptions have been defined
- [ ] Who it's for, list of personas impacted
- [ ] Your proposal has been defined
- [ ] Your success metrics have been defined
- [ ] UX maturity requirements have been measured
- [ ] Add relevant designs to the Design Management area of the issue if available
- [ ] Confirm that an unexpected outage of this feature will not negatively impact the application or other features
- [ ] Add a feature flag so that this feature can be quickly disabled if/when needed
-[ ] If this experiment introduces a new service or data store, ensure it is not processing or storing [red data](https://about.gitlab.com/handbook/security/data-classification-standard.html#data-classification-levels) without a security and if needed legal review
-*NOTE*: We recommend using one of the already adopted models or data stores. If you need to use something else, be aware that using other models or data stores will require additional review during the feature stage for operational fitness and compliance.
- [ ] Completed the necessary steps to move from Experiment to Beta
- [ ] Ensure this issue has the ~wg-ai-integration label to ensure visibility to various teams working on this
</details>
### Beta
<details><summary> Issue information </summary>
- [ ] Add information to the issue body about:
- [ ] The Main Job story and Small Jobs it's expected to satisfy have been stated
- [ ] Your assumptions have been defined
- [ ] Proposal has been updated as necessary
- [ ] Problem validation inforamtion has been added
- [ ] Business objective has been defined
- [ ] Requirements have been defined
- [ ] Success metrics have been defined
- [ ] UX maturity requirements have been measured
- [ ] Add all related feature issues to the Linked items section
- [ ] Add all relevant solution validation issues to the Linked items section that shows this proposal will solve the customer problem, or details explaining why it's not possible to provide that validation.
- [ ] Add relevant designs to the Design Management area of the issue.
-[ ] You have adhered to our [Definition of Done](https://docs.gitlab.com/ee/development/contributing/merge_request_workflow.html#definition-of-done) standards
- [ ] Completed the necessary steps to move from Beta to GA
</details>
#### Generally available
<details><summary> Issue information </summary>
- [ ] Add information to the issue body about:
- [ ] Your assumptions have been defined
- [ ] Your proposal has been defined
- [ ] Problem validation inforamtion has been added
- [ ] Business objective has been defined
- [ ] Confidence about this feature has been assessed and defined
- [ ] Requirements have been defined
- [ ] Add all relevant solution validation issues to the Linked items section that shows this proposal will solve the customer problem, or details explaining why it's not possible to provide that validation.
- [ ] Add relevant designs to the Design Management area of the issue.
-[ ] You have adhered to our [Definition of Done](https://docs.gitlab.com/ee/development/contributing/merge_request_workflow.html#definition-of-done) standards
- [ ] Ensure this issue has the ~wg-ai-integration label to ensure visibility to various teams working on this
</details>
<details><summary> Technical needs </summary>
- [ ] Please consider the operational aspects of the feature you are creating. A list of things to think about is in: https://gitlab.com/gitlab-org/gitlab/-/issues/403859. We will be improving this process in the future: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117637#note_1353253349.
-[ ] @ mention your [AppSec Stable Counterpart](https://about.gitlab.com/handbook/product/categories/) and read the [AI secure coding guidelines](https://docs.gitlab.com/ee/development/secure_coding_guidelines.html#artificial-intelligence-ai-features)
1. Work estimate and skills needs to build an ML viable feature: To build any ML feature depending on the work, there are many personas that contribute including Data Scientist, NLP engineer, ML Engineer, MLOps Engineer, ML Infra engineers, Fullstack engineer to integrate the ML Services with Gitlab. Post-prototype we would assess the skills needed to build a production-grade ML feature for the prototype.
2. Data Limitation: We would like to upfront validate if we have viable data for the feature including whether we can use the DataOps pipeline of ModelOps or create a custom one. We would want to understand the training data, test data, and feedback data to dial up the accuracy and the limitations of the data.
3. Model Limitation: We would want to understand if we can use an open-source pre-trained model, tune and customize it or start a model from scratch as well. Further, we would assess based on the ModelOps model evaluation framework which would be the right model to use based on the use case.
4. Cost, Scalability, Reliability: We would want to estimate the cost of hosting, serving, inference of the model, and the full end-to-end infrastructure including monitoring and observability.
5. Legal and Ethical Framework: We would want to align with legal and ethical framework like any other ModelOps features to cover the nine principles of responsible ML and any legal support needed.
</details>
<details><summary> Dependency needs </summary>
- [ ] Please consider the operational aspects of the service you are creating. A list of things to think about is in: https://gitlab.com/gitlab-org/gitlab/-/issues/403859. We will be improving this process in the future: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117637#note_1353253349.
</details>
<details><summary> Legal needs </summary>
- [ ] TBD
</details>
## Additional resources
- If you'd like help with technical validation, or would like to discuss UX considerations for AI mention the AI Assisted group using `@gitlab-org/modelops/applied-ml`.
- Read about our [AI Integration strategy](https://internal-handbook.gitlab.io/handbook/product/ai-strategy/ai-integration-effort/)
-`#wg_ai_integration` - Slack channel for the working group and the high-level alignment on getting AI ready for Production (Development, Product, UX, Legal, etc.) But from the other channels feel free to reach out and post progress here
-`#ai_integration_dev_lobby` - Channel for all implementation-related topics and discussions of actual AI features (e.g. explain the code)
-`#ai_enablement_team` - Channel for the AI Enablement Team which is building the base for all features (experimentation API, Abstraction Layer, Embeddings, etc.)
/label ~"AI Feature Proposal" ~"AI-Seeking community feedback"
<!-- Actionable insights must recommend an action that needs to take place. An actionable insight both defines the insight and clearly calls out action or next step required to improve based on the result of the research observation or data. Actionable insights are tracked over time and will include follow-up. Please follow the tasks outlined in this issue for best results. Learn more in the handbook here: https://about.gitlab.com/handbook/product/ux/ux-research-training/research-insights/#actionable-insights
This issue template is for an actionable insight that requires further exploration.-->
### Insight
<!-- Describe the insight itself: often the problem, finding, or observation.-->
### Supporting evidence
<!-- Describe why the problem is happening, or more details behind the finding or observation. Try to include quotes or specific data collected. Feel free to link the Actionable insight from Dovetail here if applicable instead of retyping details. -->
### Action
<!--Since this is an actionable insight that requires further exploration, ensure the action is algned to that. Describe the next step or action that needs to take place as a result of the research. The action should be clearly defined, achievable, and directly tied back to the insight. Make sure to use directive terminology, such as: conduct, explore, redesign, etc. -->
### Resources
<!--Add resources as links below or as related issues. -->
- :dove: [Dovetail project](Paste URL for Dovetail project here)
- :mag: [Research issue](Paste URL for research issue here)
- :footprints: [Follow-up issue or epic](Paste URL for follow-up issue or epic here)
### Tasks
<!--Fill out these tasks in order to consider an Actionable Insight complete. Actionable Insights are created as confidential by default, but can be made non-confidential if the insight does not include information about competitors from a Competitor Evaluation or any other confidential information. -->
- [ ] Assign this issue to the appropriate Product Manager, Product Designer, or UX Researcher.
- [ ] Add the appropriate `Group` (such as `~"group::source code"`) label to the issue. This helps identify and track actionable insights at the group level.
- [ ] Link this issue back to the original research issue in the GitLab UX Research project and the Dovetail project.
- [ ] Adjust confidentiality of this issue if applicable
<!-- Actionable insights must recommend an action that needs to take place. An actionable insight both defines the insight and clearly calls out action or next step required to improve based on the result of the research observation or data. Actionable insights are tracked over time and will include follow-up. Please follow the tasks outlined in this issue for best results. Learn more in the handbook here: https://about.gitlab.com/handbook/product/ux/ux-research-training/research-insights/#actionable-insights
This issue template is for an actionable insight that requires a change in the product.-->
### Insight
<!-- Describe the insight itself: often the problem, finding, or observation.-->
### Supporting evidence
<!-- Describe why the problem is happening, or more details behind the finding or observation. Try to include quotes or specific data collected. Feel free to link the Actionable insight from Dovetail here if applicable instead of retyping details. -->
### Action
<!--Since this is an actionable insight that requires a change in the product, ensure the action is algned to that. Describe the next step or action that needs to take place as a result of the research. The action should be clearly defined, achievable, and directly tied back to the insight. Make sure to use directive terminology, such as: change, update, add/remove, etc. -->
### Resources
<!--Add resources as links below or as related issues. -->
- :dove: [Dovetail project](Paste URL for Dovetail project here)
- :mag: [Research issue](Paste URL for research issue here)
- :footprints: [Follow-up issue or epic](Paste URL for follow-up issue or epic here)
### Tasks
<!--Fill out these tasks in order to consider an Actionable Insight complete. Actionable Insights are created as confidential by default, but can be made non-confidential if the insight does not include information about competitors from a Competitor Evaluation or any other confidential information. -->
- [ ] Assign this issue to the appropriate Product Manager, Product Designer, or UX Researcher.
- [ ] Add the appropriate `Group` (such as `~"group::source code"`) label to the issue. This helps identify and track actionable insights at the group level.
- [ ] Link this issue back to the original research issue in the GitLab UX Research project and the Dovetail project.
- [ ] Adjust confidentiality of this issue if applicable
<!-- Please complete the template below as best as you can. Make sure to check if this issue has already been raised by someone else first to avoid duplication.
For each section below, please add screenshots or links or anything that may help visual learners understand the problem better, even if this takes you an extra minute or two this is a great help to some folks.
<!-- How did we learn of the problem? System Error? User? -->
## Impact
<!-- Who was impacted and how were they impacted?
Please include Teams, Users, Automations, Processes -->
## Additional information
<!-- Please fill out any additonal information that may help with resolution -->
## Checklist
<!-- Please ensure all of the below steps are taken -->
*[ ] Assigned severity tags based on this [guidance](https://about.gitlab.com/handbook/engineering/development/analytics/analytics-instrumentation/#incident-detection)
* [ ] Assigned to PM and EM of ~"group::analytics instrumentation"
* [ ] Posted link to incident in `g_analyze_analytics_instrumentation` and tagged both PM and EM of the group
<----TOBEFILLEDBYASSIGNEE/RESOLUTIONDRI---->
<!-- The intention of this section is to give stakeholders a high level summary of the issue with whatever information we have at the point of closing the issue or providing a quick fix. For a deeper analysis of the root cause and how we can avoid such incidents in the future please use the RCA template as outlined in the handbook incident resolution guidance and link the issue here -->
## Summary
<!-- Overall summary of the issue -->
## Root Cause
<!-- To be filled by the assignee. What caused this incident? Link to RCA issue if needed -->
## Resolution
<!-- To be filled by assignee.Has the issue been resolved? How? Please include any relevant issue and/or MR links in the description -->
<!-- The problem being addressed by this issue -->
## Desired Outcome
<!-- The desired state of the affected system after this issue has been completed -->
## Potential Solution
<!-- An outline of potential solutions to get to the desired outcome. These solution(s) can still be adjusted throughout the implementation as long as the desired outcome is achieved. -->
## How to verify
<!-- How can we verify that the desired outcome has been achieved? The instructions from this section should be used to move the issue from ~“worfklow::verification”. -->
## Further actions needed
<!-- Any further tasks that need to be completed after the main work of the issue is done, such as announcing the changes or updating documentation. -->
<!-- This section is optional. -->
<!-- Make sure to add one of the type labels (as per https://handbook.gitlab.com/handbook/engineering/metrics/#work-type-classification):-->
<!-- Link to the failing master build and add the build failure output in the below code block section. -->
### Steps to reproduce
<!-- If the pipeline failure is reproducible, provide steps to recreate the issue locally. Please use an ordered list. -->
Please refer to [Flaky tests documentation](https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html) to
learn more about how to reproduce them.
### Proposed Resolution
<!-- Describe the proposed change to restore master stability. -->
Please refer to the [Resolution guidance](https://about.gitlab.com/handbook/engineering/workflow/#resolution-of-broken-master) to learn more about resolution of broken master.
Once the flaky failure has been fixed on the default branch, open merge requests to cherry-pick the fix to the active stable branches.
