How to Drain and Reboot an OpenShift Worker Node Safely

Planned worker-node maintenance in OpenShift is mostly a scheduling and disruption problem, not a reboot problem. The risky part is what happens to workloads while a node becomes unschedulable, pods are evicted, the host restarts, and the node later rejoins the cluster.

For OpenShift 4.14, Red Hat documents a graceful restart flow built around oc adm cordon, oc adm drain, oc debug node/..., chroot /host, systemctl reboot, and oc adm uncordon in Working with nodes. Kubernetes separately documents that drain marks a node unschedulable, uses eviction when supported, does not delete DaemonSet-managed pods, and waits for graceful termination before you operate on the machine in kubectl drain.

This guide assumes a multi-node OpenShift cluster and focuses on worker nodes. It is not a control-plane procedure. It is also intentionally conservative: if drain blocks, that usually means the cluster is exposing a real capacity, placement, or PodDisruptionBudget problem rather than a broken command.

Prerequisites

Before starting maintenance, make sure all of the following are true:

• You have the OpenShift CLI installed.
• You have cluster-admin or equivalent permissions for node administration. The OpenShift CLI administrator reference states that oc adm administrator commands require those permissions, and the oc debug examples document node debugging as an administrator in CLI tools.
• The cluster has enough spare capacity to reschedule workload pods away from the target worker.
• You are draining one worker at a time.
• You understand whether any workloads on that node use emptyDir, because --delete-emptydir-data allows drain to continue by deleting that local ephemeral data, as described in kubectl drain.
• You have reviewed any PodDisruptionBudgets that protect important workloads. A normal drain respects PodDisruptionBudgets and waits for graceful termination before you touch the node, as documented in Safely Drain a Node and kubectl drain.

What cordon, drain, and uncordon actually do

These commands are related, but they are not interchangeable.

Cordon stops new scheduling

oc adm cordon <worker-node>

A cordoned node becomes unschedulable. Existing pods keep running, but new pods are not placed there. Red Hat shows this as the first step in node evacuation and graceful restart procedures in Working with nodes.

Drain evicts movable pods

oc adm drain <worker-node> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force

Kubernetes documents that drain evicts or deletes pods, refuses to proceed past DaemonSet-managed pods unless you add --ignore-daemonsets, and requires --force for certain unmanaged pods in kubectl drain. Red Hat documents the same operational pattern for OpenShift worker maintenance in Working with nodes.

Uncordon returns the node to service

oc adm uncordon <worker-node>

Uncordon makes the node schedulable again after the reboot. Red Hat places this after the node has restarted and returned cleanly in Working with nodes.

Prechecks before you touch the node

A safe reboot starts with inspection.

Check what is running on the worker

oc get pods -A --field-selector spec.nodeName=<worker-node> -o wide

Expected result:

• most application pods are managed by a controller such as a Deployment, StatefulSet, or Job
• critical services have replicas elsewhere
• nothing important depends on that node as a singleton

If a single-replica workload cannot tolerate disruption, fix the application design or maintenance plan before you drain the node.

Review PodDisruptionBudgets

oc get pdb -A

If a drain blocks on a PDB, treat that as a useful signal, not an inconvenience. Kubernetes recommends draining through the normal eviction path and explains that drain can wait when disruption policy or unhealthy pods prevent safe eviction in Safely Drain a Node.

Check whether this worker also hosts OpenShift infrastructure pods

This is where an ordinary Kubernetes drain becomes an OpenShift-specific maintenance decision.

Red Hat warns that when a node hosts critical OpenShift infrastructure such as router, image registry, or monitoring pods, you should ensure that at least three nodes are available to run those components in Working with nodes.

Check the obvious infrastructure namespaces on the target worker:

oc get pods -n openshift-ingress -o wide \
  --field-selector spec.nodeName=<worker-node>

oc get pods -n openshift-image-registry -o wide \
  --field-selector spec.nodeName=<worker-node>

oc get pods -n openshift-monitoring -o wide \
  --field-selector spec.nodeName=<worker-node>

If those commands return critical infrastructure pods, do not treat this as an ordinary application-only worker reboot. First decide one of these paths:

• Prefer the simple path: choose a different worker that does not host critical infrastructure.
• If you must continue: verify there are at least three eligible nodes for those components, and verify the placement policy spreads replicas correctly.

For router pods, Red Hat notes that most router pods expose a host port, which prevents same-port router pods from landing on the same node. It also notes rare cases where a router pod might not use a host port, which is why the stricter infrastructure-node guidance still matters in Working with nodes.

Be deliberate about emptyDir

--delete-emptydir-data is not a harmless default. Kubernetes is explicit that this flag lets drain continue even when pods use emptyDir, and that local data is deleted when the node is drained in kubectl drain.

Practical runbook for one worker node

This runbook is appropriate for a worker whose application pods can move safely and which is not the last eligible host for critical OpenShift infrastructure.

1. Cordon the worker

oc adm cordon <worker-node>
oc get node <worker-node>

Expected result:

• the node reports Ready,SchedulingDisabled
• current pods stay up for the moment
• no new pods are scheduled there

2. Drain the worker

oc adm drain <worker-node> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force

Expected result:

• OpenShift evicts movable workload pods
• DaemonSet-managed pods are ignored rather than deleted
• replacement pods land on other nodes if policy and capacity allow
• the command returns only after drain is complete

Kubernetes documents that once drain returns without error, you can power down the node, and if the node will stay in the cluster afterward you should uncordon it later in Safely Drain a Node.

3. Reboot through a debug session

oc debug node/<worker-node>
chroot /host
systemctl reboot

Red Hat documents this sequence for a graceful node restart in Working with nodes.

Expected result:

• the debug session ends as the node restarts
• the node transitions through NotReady
• previously evicted workload pods continue elsewhere

4. Wait for the node to return

oc get nodes

Expected result:

• the worker comes back as Ready
• cluster services can schedule to it again once uncordoned

Do not uncordon just because the VM or server answers the network. Wait for the cluster view to recover.

5. Uncordon the worker

oc adm uncordon <worker-node>
oc get node <worker-node>

Expected result:

• SchedulingDisabled disappears
• the worker becomes eligible for new pods again

If you are patching several workers, finish validation on this one before starting the next.

When drain does not finish

A PodDisruptionBudget blocks eviction

A typical error here means the workload does not currently have enough safe redundancy for this maintenance step.

Red Hat documents a stronger fallback:

oc adm drain <worker-node> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force \
  --disable-eviction

Kubernetes documents that --disable-eviction forces delete behavior instead of using the eviction API, which bypasses PodDisruptionBudget checks. Use this only when you understand the availability impact and have explicit approval, as described in kubectl drain and shown in Red Hat's restart guidance in Working with nodes.

DaemonSet pods are still present

That is normal. Drain does not delete DaemonSet-managed pods, because those pods would be recreated immediately and the DaemonSet controller ignores the unschedulable marking, as documented in kubectl drain.

Replacement pods remain Pending

When replacement pods cannot schedule elsewhere, the problem is usually outside the drain command:

• not enough free CPU or memory
• restrictive node selectors
• taints without matching tolerations
• anti-affinity rules
• storage placement limits

If the blocked pods are infrastructure replicas, revisit the earlier check for eligible nodes and anti-affinity. Kubernetes explains how required and preferred pod anti-affinity influence placement in Assigning Pods to Nodes.

Final thoughts

A safe OpenShift worker reboot is mostly about sequence and restraint: identify what is on the node, confirm it is not the wrong place to risk critical infrastructure, cordon first, drain carefully, reboot through a supported path, and uncordon only after the cluster says the node is healthy again. If drain argues with you, treat that as feedback from workload policy or cluster capacity rather than a cue to reach for stronger flags immediately.

For structured practice with OpenShift day-2 operations such as RBAC, network policies, quotas, operators, and workload security, see OpenShift Administration: Configuring a Production Cluster.

Sources

• Working with nodes
• CLI tools
• kubectl drain
• Safely Drain a Node
• Assigning Pods to Nodes