• Rancher v2.13.0

• Using the Rancher Backup & Restore Operator
  使用 Rancher 备份与恢复操作员

There is a known bug faced when performing a rollback from Rancher v2.13.0 to Rancher v2.12.3 using BRO (the Backup & Restore Operator), which prevents the Restore from completing successfully.
在使用 BRO（备份与恢复操作符）从 Rancher v2.13.0 回滚到 Rancher v2.12.3 时，存在已知的 bug，导致恢复无法成功完成。

The Backup & Restore Operator logs look similar to this:
备份与恢复操作员的日志大致如下：

<span style="color:#000000"><span style="background-color:#ffffff"><span style="background-color:#efefef"><code>ERRO[2025/11/17 16:26:58] Error restoring cluster-scoped resources [error restoring cattle-globalrole-user-base of type <a data-cke-saved-href="http://rbac.authorization.k8s.io/v1" href="http://rbac.authorization.k8s.io/v1">rbac.authorization.k8s.io/v1</a>, Resource=clusterroles: restoreResource: err updating resource admission webhook "<a data-cke-saved-href="http://rancher.cattle.io.clusterroles.rbac.authorization.k8s.io/" href="http://rancher.cattle.io.clusterroles.rbac.authorization.k8s.io/">rancher.cattle.io.clusterroles.rbac.authorization.k8s.io</a>" denied the request: cannot modify or remove label <a data-cke-saved-href="http://authz.management.cattle.io/gr-owner" href="http://authz.management.cattle.io/gr-owner">authz.management.cattle.io/gr-owner</a> error restoring cattle-globalrole-users-manage of type <a data-cke-saved-href="http://rbac.authorization.k8s.io/v1" href="http://rbac.authorization.k8s.io/v1">rbac.authorization.k8s.io/v1</a>, Resource=clusterroles: restoreResource: err updating resource admission webhook "<a data-cke-saved-href="http://rancher.cattle.io.clusterroles.rbac.authorization.k8s.io/" href="http://rancher.cattle.io.clusterroles.rbac.authorization.k8s.io/">rancher.cattle.io.clusterroles.rbac.authorization.k8s.io</a>" denied the request: cannot modify or remove label <a data-cke-saved-href="http://authz.management.cattle.io/gr-owner" href="http://authz.management.cattle.io/gr-owner">authz.management.cattle.io/gr-owner</a> error restoring cattle-globalrole-user of type <a data-cke-saved-href="http://rbac.authorization.k8s.io/v1" href="http://rbac.authorization.k8s.io/v1">rbac.authorization.k8s.io/v1</a>, Resource=clusterroles: restoreResource: err updating resource admission webhook "<a data-cke-saved-href="http://rancher.cattle.io.clusterroles.rbac.authorization.k8s.io/" href="http://rancher.cattle.io.clusterroles.rbac.authorization.k8s.io/">rancher.cattle.io.clusterroles.rbac.authorization.k8s.io</a>" denied the request: cannot modify or remove label <a data-cke-saved-href="http://authz.management.cattle.io/gr-owner" href="http://authz.management.cattle.io/gr-owner">authz.management.cattle.io/gr-owner</a> error restoring cattle-globalrole-clusters-create of type <a data-cke-saved-href="http://rbac.authorization.k8s.io/v1" href="http://rbac.authorization.k8s.io/v1">rbac.authorization.k8s.io/v1</a>, Resource=clusterroles: restoreResource: err updating resource admission webhook "<a data-cke-saved-href="http://rancher.cattle.io.clusterroles.rbac.authorization.k8s.io/" href="http://rancher.cattle.io.clusterroles.rbac.authorization.k8s.io/">rancher.cattle.io.clusterroles.rbac.authorization.k8s.io</a>" denied the request: cannot modify or remove label <a data-cke-saved-href="http://authz.management.cattle.io/gr-owner" href="http://authz.management.cattle.io/gr-owner">authz.management.cattle.io/gr-owner</a>]
ERRO[2025/11/17 16:26:58] error syncing 'restore-migration': handler restore: error restoring cluster-scoped resources, check logs for exact error, requeuing</code></span></span></span>

The bug will be officially fixed in the v2.14.0 version of Rancher as part of improvements tracked by this Github issue. There is, however, a workaround solution to prevent this bug from happening.
该漏洞将在 Rancher v2.14.0 版本中正式修复，作为本次 GitHub 问题跟踪改进的一部分。不过，有一个解决办法可以防止这个 bug 发生。

Disabling the Rancher Webhook during a Restore
在恢复过程中禁用牧场主的 Webhook

The official documentation on how to perform a rollback can be found here. Essentially, the usual steps will look like this (summarized from the documentation):
关于如何进行回滚的官方文档可以在这里找到 。基本上，通常的步骤如下（从文档中总结）：

1. In a cluster running Rancher v2.13.0, make sure you have access to a Backup taken for Rancher v2.12.3
   在运行 Rancher v2.13.0 的集群中，确保你能访问 Rancher v2.12.3 的备份

2. Create a BRO Restore CR referencing the desired Backup, wait for it to be completed
   创建一个 BRO Restore CR，引用所需的备份，等待备份完成

3. Perform the Helm Rollback as referenced in the docs, usually helm rollback rancher -n cattle-system
   按照文档中提到的进行 Helm Rollback，通常是 Helm Rollback Rancher -n cattle-system（Helm Rollback Rancher -n cattle-system）

To prevent the bug from happening
为了防止这个 bug 发生

Two extra steps are needed (added as points 2 and 3 below):
还需要两个额外步骤（作为下文第2点和第3点加进）：

1. In a cluster running Rancher v2.13.0, make sure you have access to a Backup taken for Rancher v2.12.3
   在运行 Rancher v2.13.0 的集群中，确保你能访问 Rancher v2.12.3 的备份

2. Scale down Rancher to 0 replicas kubectl scale deploy/rancher -n cattle-system --replicas=0
   将 Rancher 缩减至 0 个复制品 kubectl scale deploy/rancher -n cattle-system --replicas=0

3. Uninstall the Rancher webhook with helm uninstall helm uninstall rancher-webhook -n cattle-system. Make sure it was uninstalled correctly with helm list -n cattle-system
   用 helm uninstall helm uninstall rancher-webhook -n cattle-system 卸载 Rancher webhook。确保它已经正确卸载，使用 helm list -n cattle-system

4. Create a BRO Restore CR (via kubectl, as Rancher is down) referencing the desired Backup, wait for it to be completed
   创建一个 BRO 还原 CR（通过 kubectl，因为 Rancher 宕机），引用所需的备份，等待备份完成

5. Perform the Helm Rollback as referenced in the docs, usually helm rollback rancher -n cattle-system
   按照文档中提到的进行 Helm Rollback，通常是 Helm Rollback Rancher -n cattle-system（Helm Rollback Rancher -n cattle-system）

Rancher Documentation - Backup, Restore, and Disaster Recovery牧场主文档——备份、恢复与灾难恢复

GitHub - backup-restore-operator