add docs for new ManuallyPromote upgrade rollout strategy

Author: jubrad

doc/user/content/self-managed-deployments/upgrading/_index.md

@@ -24,9 +24,7 @@ name="upgrade-major-version-restriction" >}}
 {{< /note >}}
 
 
-## Upgrading
-
-### Upgrade guides
+## Upgrade guides
 
 The following upgrade guides are available as examples:
 
@@ -39,7 +37,7 @@ name="upgrade-landing-guides-unified" %}}
 {{% include-from-yaml data="self_managed/upgrades"
 name="upgrade-landing-guides-legacy" %}}
 
-### Upgrading the Helm Chart and Materialize Operator
+## Upgrading the Helm Chart and Materialize Operator
 
 {{< important >}}
 
@@ -48,7 +46,7 @@ Operator first.
 
 {{</ important >}}
 
-#### Update the Helm Chart repository
+### Update the Helm Chart repository
 
 To update your Materialize Helm Chart repository:
 
@@ -62,7 +60,7 @@ View the available chart versions:
 helm search repo materialize/materialize-operator --versions
 ```
 
-#### Upgrade your Materialize Operator
+### Upgrade your Materialize Operator
 
 The Materialize Kubernetes Operator is deployed via Helm and can be updated
 through standard `helm upgrade` command:
@@ -93,7 +91,7 @@ helm upgrade -n materialize my-demo materialize/operator \
   --version {{< self-managed/versions/get-latest-version >}}
 ```
 
-### Upgrading Materialize Instances
+## Upgrading Materialize Instances
 
 **After** you have upgraded your Materialize Operator, upgrade your Materialize
 instance(s) to the **APP Version** of the Operator. To find the version of your
@@ -121,7 +119,7 @@ periods for your application, the upgrade process involves two steps:
 
 - Second, roll out the changes by specifying a new UUID for `requestRollout`.
 
-#### Stage the Materialize instance version change
+### Stage the Materialize instance version change
 
 To stage the Materialize instances version upgrade, update the
 `environmentdImageRef` field in the Materialize custom resource spec to the
@@ -143,7 +141,7 @@ does not roll out the changes.
 {{< /note >}}
 
 
-#### Applying the changes via `requestRollout`
+### Applying the changes via `requestRollout`
 
 To apply chang Materialize instance upgrade, you must update the `requestRollout` field in the Materialize custom resource spec to a new UUID.
 Be sure to consult the [Rollout Configurations](#rollout-configuration) to ensure you've selected the correct rollout behavior.
@@ -155,7 +153,7 @@ kubectl patch materialize <instance-name> \
   -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\"}}"
 ```
 
-#### Staging and applying in a single command
+### Staging and applying in a single command
 
 Although separating the staging and rollout of the changes into two steps can
 minimize unexpected downtime and avoid connection drops at critical periods, you
@@ -168,7 +166,7 @@ kubectl patch materialize <instance-name> \
   -p "{\"spec\": {\"environmentdImageRef\": \"docker.io/materialize/environmentd:{{< self-managed/versions/get-latest-version >}}\", \"requestRollout\": \"$(uuidgen)\"}}"
 ```
 
-##### Using YAML Definition
+#### Using YAML Definition
 
 Alternatively, you can update your Materialize custom resource definition directly:
 
@@ -193,9 +191,9 @@ Apply the updated definition:
 kubectl apply -f materialize.yaml
 ```
 
-### Rollout Configuration
+## Rollout Configuration
 
-#### `requestRollout`
+### `requestRollout`
 
 Specify a new `UUID` value for the `requestRollout` to roll out the changes to
 the Materialize instance.
@@ -227,15 +225,47 @@ kubectl patch materialize <instance-name> \
   -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\", \"forceRollout\": \"$(uuidgen)\"}}"
 ```
 
-#### Rollout strategies
-
+### Rollout strategies
 The behavior of the new version rollout follows your `rolloutStrategy` setting:
 
-| `rolloutStrategy` | Description                        |
-| ----------------- | -----------------------------------|
-| `WaitUntilReady`  | *Default*. New instances are created and all dataflows are determined to be ready before cutover and terminating the old version, temporarily requiring twice the resources during the transition. |
-| `ImmediatelyPromoteCausingDowntime`| Tears down the prior version before creating and promoting the new version. This causes downtime equal to the duration it takes for dataflows to hydrate, but does not require additional resources. |
-| `inPlaceRollout`| *Deprecated*.  The setting is ignored. |
+#### *WaitUntilReady* - ***Default***.
+Creates a new generation of pods and automatically cuts over to them as soon as they catch up to the old generation and become `ReadyToPromote`.
+
+{{<note>}}
+This strategy temporarily doubles the required hardware capacity.
+{{</note>}}
+
+#### *ImmediatelyPromoteCausingDowntime*
+Tears down the prior generation before creating and immediately promoting the new generation without waiting for it to hydrate.
+This causes downtime equal to the duration it takes for dataflows to hydrate, but does not require additional resources.
+
+#### *ManuallyPromote* - ***Public Preview***
+Creates a new generation of pods, cuts over to them, only when  the user
+manually promotes the new generation by updating the `forcePromote` field to
+match the `requestRollout` field. This allows the user to stage an upgrade and
+cut over to it at a precise time.
+
+In order to minimize downtime, it is recommended to wait until all dataflows are
+caught up. This can be determined by checking for the `ReadyToPromote` reason in
+the `UpToDate` status condition in the Materialize Kubernetes custom resource.
+
+{{<note>}}
+This strategy temporarily doubles the required hardware capacity.
+{{</note>}}
+
+{{<warning>}}
+It is not recommended to leave generations unpromoted for over 6 hours.
+
+Unpromoted generations keeps read holds open, preventing compaction until they
+are promoted or cancelled. If left unpromoted for an extended period, this data
+can build up and cause significant load on the metadata backend database
+resulting in service degradation.
+{{</warning>}}
+
+#### *inPlaceRollout* - ***Deprecated***
+The setting is ignored.
+
+
 
 ## Verifying the Upgrade
 

src/cloud-resources/src/crd/materialize.rs

@@ -42,9 +42,15 @@ pub mod v1alpha1 {
         /// Create a new generation of pods, leaving the old generation as the serving generation
         /// until the user manually promotes the new generation.
         ///
-        /// Users can promote the new generation at any time, even if the new generation pods are
-        /// not fully caught up, by setting `forcePromote` to the same value as `requestRollout` in
-        /// the Materialize spec.
+        /// When using `ManuallyPromote`, the new generation can be promoted at any
+        /// time, even if it has dataflows that are not fully caught up, by setting
+        /// `forcePromote` to the same value as `requestRollout` in the Materialize spec.
+        ///
+        /// To minimize downtime, promotion should occur when the new generation
+        /// has caught up to the prior generation. To determine if the new
+        /// generation has caught up, consult the `UpToDate` condition in the
+        /// status of the Materialize Resource. If the condition's reason is
+        /// `ReadyToPromote` the new generation is ready to promote.
         ///
         /// {{<warning>}}
         /// Do not leave new generations unpromoted indefinitely.