:_mod-docs-content-type: PROCEDURE

// Module included in the following assemblies:
//
// assembly-config.adoc

[id='proc-moving-node-pools-{context}']
= Moving nodes between node pools

[role="_abstract"]
This procedure describes how to move nodes between source and target Kafka node pools without downtime.
You create a new node on the target node pool and reassign partitions to move data from the old node on the source node pool.
When the replicas on the new node are in-sync, you can delete the old node.

In this procedure, we start with two node pools:

* `pool-a` with three replicas is the target node pool
* `pool-b` with four replicas is the source node pool

We scale up `pool-a`, and reassign partitions and scale down `pool-b`, which results in the following:

* `pool-a` with four replicas
* `pool-b` with three replicas

Currently, scaling is only possible for broker-only node pools containing nodes that run as dedicated brokers.

NOTE: During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.

.Prerequisites

* xref:deploying-cluster-operator-str[The Cluster Operator must be deployed.]
* xref:proc-configuring-deploying-cruise-control-str[Cruise Control is deployed with Kafka.]
* (Optional) xref:proc-automating-rebalances-{context}[Auto-rebalancing is enabled]. +
If auto-rebalancing is enabled, partition reassignment happens automatically during the node scaling process, so you don't need to manually initiate the reassignment through Cruise Control.
* (Optional) For scale up and scale down operations, xref:proc-managing-node-pools-ids-{context}[you can specify the range of node IDs to use]. +
If you have assigned node IDs for the operation, the ID of the node being added or removed is determined by the sequence of nodes given. 
Otherwise, the lowest available node ID across the cluster is used when adding nodes; and the node with the highest available ID in the node pool is removed. 

.Procedure

. Create a new node in the target node pool.
+
For example, node pool `pool-a` has three replicas. We add a node by increasing the number of replicas:
+
[source,shell]
----
kubectl scale kafkanodepool pool-a --replicas=4
----

. Check the status of the deployment and wait for the pods in the node pool to be created and ready (`1/1`).
+
[source,shell]
----
kubectl get pods -n <my_cluster_operator_namespace>
----
+
.Output shows four Kafka nodes in the source and target node pools
[source,shell]
----
NAME                 READY  STATUS   RESTARTS
my-cluster-pool-a-0  1/1    Running  0
my-cluster-pool-a-1  1/1    Running  0
my-cluster-pool-a-4  1/1    Running  0
my-cluster-pool-a-7  1/1    Running  0
my-cluster-pool-b-2  1/1    Running  0
my-cluster-pool-b-3  1/1    Running  0
my-cluster-pool-b-5  1/1    Running  0
my-cluster-pool-b-6  1/1    Running  0
----
+
Node IDs are appended to the name of the node on creation.
We add node `my-cluster-pool-a-7`, which has a node ID of `7`.
+
If auto-rebalancing is enabled, partitions are reassigned to new nodes and moved off brokers that are going to be removed automatically, so you can skip the next step.

. If auto-rebalancing is not enabled, reassign partitions before decreasing the number of nodes in the source node pool.
+
Use the Cruise Control `remove-brokers` mode to move partition replicas off the brokers that are going to be removed.
+
.Using Cruise Control to reassign partition replicas
[source,shell,subs="+attributes"]
----
apiVersion: {KafkaRebalanceApiVersion}
kind: KafkaRebalance
metadata:
  # ...
spec:
  mode: remove-brokers
  brokers: [6]
----
+
We are reassigning partitions from node `my-cluster-pool-b-6`. 
The reassignment can take some time depending on the number of topics and partitions in the cluster.

. After the reassignment process is complete, reduce the number of Kafka nodes in the source node pool.
+
For example, node pool `pool-b` has four replicas. We remove a node by decreasing the number of replicas:
+
[source,shell]
----
kubectl scale kafkanodepool pool-b --replicas=3
----
+
The node with the highest ID (`6`) within the pool is removed.
+
.Output shows three Kafka nodes in the source node pool
[source,shell]
----
NAME                       READY  STATUS   RESTARTS
my-cluster-pool-b-kafka-2  1/1    Running  0
my-cluster-pool-b-kafka-3  1/1    Running  0
my-cluster-pool-b-kafka-5  1/1    Running  0
----

