From a5f4ba63181abfdd1c3f2d3660eae90cb256ec6e Mon Sep 17 00:00:00 2001
From: Bogdan Nechyporenko <bnechyporenko@bol.com>
Date: Thu, 6 Feb 2025 18:19:24 +0100
Subject: [PATCH] Documenting experimental feature of "Retries and Recovery" in
 Scaffolder

Signed-off-by: Bogdan Nechyporenko <bnechyporenko@bol.com>
---
 .../software-templates/experimental.md        | 59 ++++++++++++++++++-
 1 file changed, 58 insertions(+), 1 deletion(-)

diff --git a/docs/features/software-templates/experimental.md b/docs/features/software-templates/experimental.md
index e37971ad82..8a3a29e9f1 100644
--- a/docs/features/software-templates/experimental.md
+++ b/docs/features/software-templates/experimental.md
@@ -13,7 +13,64 @@ Please leave feedback on these features in the [Backstage Discord](https://disco
 
 ## Retries and Recovery
 
-### TODO
+Do you have long-running tasks, and can they recover and proceed after Backstage redeploy?
+
+Then you definitely will benefit from enabling the experimental feature called EXPERIMENTAL_recovery.
+
+Whenever you do redeploy, on startup there will be a check of all tasks in "processing" state that you identified in your template as they are capable of starting over.
+
+This is an example of how you can do it:
+
+```yaml
+apiVersion: scaffolder.backstage.io/v1beta3
+kind: Template
+metadata:
+  name: recoverable-template
+spec:
+  EXPERIMENTAL_recovery:
+    EXPERIMENTAL_strategy: startOver
+```
+
+And to enable the recovery feature itself, you can do it by adding this snippet into your app-config.yaml file:
+
+```yaml
+scaffolder:
+  EXPERIMENTAL_recoverTasks: true
+```
+
+By default, all the tasks that are in a processing state and have a heartbeat of more than 30 seconds will be automatically recovered.
+This implies that the task's status will shift to "Open," initiating its execution from the beginning. So you have to be sure that your tasks can run multiple times.
+You can look at how to incorporate [checkpoints](https://backstage.io/docs/features/software-templates/writing-custom-actions#using-checkpoints-in-custom-actions-experimental) into your custom actions to achieve that.
+
+In case 30 seconds of no heartbeat time is not appropriate for your case to restart the task, you can customize it for your needs with the configuration:
+
+```yaml
+scaffolder:
+  EXPERIMENTAL_recoverTasksTimeout: { minutes: 1 }
+```
+
+If your task works with the filesystem, you might require serialization of your workspace.
+
+You can enable this feature with:
+
+```yaml
+scaffolder:
+  EXPERIMENTAL_workspaceSerialization: true
+```
+
+By default, the serialized workspace will be stored in your database.
+
+If you work with large files, it might not be the best option for you.
+
+At this moment we support the integration with the GCP bucket; to switch the serialization to this provider, you can with:
+
+```yaml
+scaffolder:
+  EXPERIMENTAL_workspaceSerializationProvider: gcpBucket
+  EXPERIMENTAL_workspaceSerializationGcpBucketName: name-of-your-bucket
+```
+
+You don't need to provide any extra configuration, but you have to be sure that you are using [workload identity](https://cloud.google.com/iam/docs/workload-identity-federation).
 
 ## Form Decorators