
# Troubleshouting the Compliance-controller of the Klusterlet

## Klusterlet overall check

The first step is to make sure that the Klusterlet is installed and running.
The Klusterelet is installed by a helm chart as a `k8s deployment`. We will first examine this deployment

```bash
 DEPLOYMENT=$(kubectl get deployments --all-namespaces | grep klusterlet | grep -v controller | grep -v weave | awk '{print $2}')
 DEPLOYMENT_NS=$(kubectl get deployments --all-namespaces | grep klusterlet | grep -v controller | grep -v weave | awk '{print $1}')
 kubectl get deployment $DEPLOYMENT -n $DEPLOYMENT_NS -owide
 ```

Next we can take a closer look at the Klusterlet deployment and look into its events and status

 ```bash
 kubectl describe deployment $DEPLOYMENT -n $DEPLOYMENT_NS

```

## Klusterlet pod's container status

If everything so far looks normal, the next step is to examine the Klusterlet pod.

```bash
 POD=$(kubectl get pods --all-namespaces | grep klusterlet | grep -v controller | grep -v weave | awk '{print $2}')
 POD_NS=$(kubectl get pods --all-namespaces | grep klusterlet | grep -v controller | grep -v weave | awk '{print $1}')
 kubectl get pod $POD -n $POD_NS -owide
 ```

 Expected output:

 ```bash
 NAME                                                READY     STATUS    RESTARTS   AGE       IP            NODE
mcm-ibm-mcm-chart-dev-klusterlet-5f5f877644-p8t7f    3/3       Running   0          9d        10.1.103.39   9.12.235.140
```

Seeing that all the containers are ready (e.g. 3/3 or 4/4) is a good sign.

if any of the pods are not ready, then we need to check the unhealthy container log.
To get the list of container names within the pod use:

```bash
kubectl get pod $POD -n $POD_NS -o jsonpath="{.spec.containers[*].name}";echo
```

You can describe the pod and see which containers have the `ready=false` state

```bash
kubectl describe pod $POD -n $POD_NS
```

then look at the log of the pods that are not ready:

```bash
kubectl logs $POD -n $POD_NS -c sync-controller #or work-manager, policy-controller, compliance-controller
```

if you see a message of `unauthorized` then you have an installation error, the Klusterlet cannot connect properly to the `MCM-hub`

Note: if for some reason the container is missing, you can try the `--previous` flag

```bash
kubectl logs --previous  $POD -n $POD_NS -c sync-controller #or work-manager, policy-controller, compliance-controller
```

Other error messages might be related to not having a required `namespace` created in your cluster.
Get a list of the cluster namespaces:

```bash
kubectl get namespaces
```

Then check the namespace that the Klusterlet is listening to:

```bash
kubectl describe pod $POD -n $POD_NS | grep "watch-ns"
```

Example result:

```bash
kubectl describe pod $POD -n $POD_NS | grep "watch-ns"
      --watch-ns=mcm-icp2
      --watch-ns=mcm-icp2
```

If all seems well, the containers are running, the logs show no major errors, and the namespaces exist, yet still the policies (you created in a <some_compliance_file.yaml> in the hub cluster) are not enforced in the remote cluster, then we check the compliance and policy files.

## Ensuring the compliance have propagated to the cluster

first we will check the compliance objects in the namespace the compliance is configured to listen to, i.e. this one:

```bash
kubectl describe pod $POD -n $POD_NS | grep "watch-ns"
      --watch-ns=mcm-icp2
      --watch-ns=mcm-icp2
```

then we use the found namespace to list the compliance

```bash
kubectl get compliance -n `the namespace we found in the previous command`
```

for more details:

```bash
kubectl get compliance -n `the namespace we found in the previous command` -o yaml
```

If no compliance is found here, then it has not propagated to the remote cluster from the MCM-hub cluster.

### Check compliance objects on the hub cluster
Note: depends on the deployment, the hub server can be one of the managed servers as well.


First we will check that the `Compliance` object exsists in the IBM Multi-cloud Manager namespace on the hub cluster.
Check the MCM namespace name:

```bash
POD=$(kubectl get pods --all-namespaces | grep mcm | grep controller | awk '{print $2}')
POD_NS=$(kubectl get pods --all-namespaces | grep mcm | grep controller | awk '{print $1}')
kubectl describe pod $POD -n $POD_NS | grep "mcm-ns"
```

Example result:
```bash
kubectl describe pod $POD -n $POD_NS | grep "mcm-ns"
      -mcm-ns=mcm
```
Now, check that your compliance objects exsist in the mcm namespace, which name we just found, and in the namespace for 
the managed cluster. Its name is the same on the managed cluster and on the hub.

```bash
kubectl get compliance --all-namespaces
```

If the checked `Compliance` object doesn't exsist in the mcm namespace, your compliance deploymend failed, or you deploy
it into a wrong namesapce. Redeploy it if required.

Now check that the Compliance object exsists in the namespace asigned to the destiantion cluster. You can discover the 
namespace by listing all clusters, and see the relevant namespaces.

```bash
kubectl get cluster --all-namespaces
```
If the  `Compliance` object exsists in the mcm namespace but doesn't exsist in the managed cluater namespace, check `PlacementPolicy` and `PlacementBinding` objects in the mcm namespace.
First list PlacementBindings 

Example output:

```bash
kubectl get PlacementBinding -n mcm
NAME       PLACEMENTPOLICY   SUBJECTS                                AGE
binding1   placement1        compliance0(Compliance.)                27m
binding2   placement2        deployable-simple(DestinationPolicy.)   19m
binding3   placement3        compliance3(Compliance.)                2s
```

Find `PlacementPolicies` objects that bound to your `Compliance` object. For example, if your `Complaince` is named `compliance0`, its placement is defined by the 
`PlacementPolicy` object named `placement1`.

Note: can be several `PlacementPolicy` objects wich manage teh same `Compliance` object.

Check `clusterNames` and `clusterLabels` in the `PlacementPolicy` object.  
Sometimes there can be a name-mismatch or a label specified that does not exist on the remote cluster object in the cluster registry

```yaml
apiVersion: mcm.ibm.com/v1alpha1
kind: PlacementPolicy
metadata:
  name: placement1
  namespace: mcm
spec:
  clusterNames:
  - "icp3"
  - "cl1"
  clusterLabels:
    matchLabels:
      cloud: "IBM"
```

Side Note: you can see the clusters registered to MCM by executing this command on the MCM-hub cluster:

```bash
kubectl get clusters --all-namespaces
```

## Check compliance policies on the cluster

If you find the compliance exists on the remote cluster, and all seems normal, then we look at the policies

```bash
kubectl get policy -n `the namespace we found in the previous command`
```

for more details:

```bash
kubectl get policy -n `the namespace we found in the previous command` -o yaml
```

Make sure to the check the policy `spec` the following attributes:

```bash
        remediationAction: "enforce" # enforce or inform 
        complianceType: "musthave" # used as default, when missing in a particular sub-template
        namespaces:
          include: ["default"]
          exclude: ["kube*"]
```

You might have specified to include a namespace that does not exist, or you excluded all. also note that `inform` only checks but does not remdiate a violation
