AWS-EKS-19--Autoscaling 之 Karpenter

发表于 更新于 分类于 ， ， 置顶 精品
阅读次数： 本文字数： 4k 阅读时长 ≈ 15 分钟

摘要

Karperter是什么？

  • Karpenter 是一个开源集群自动缩放器，可以自动为不可安排的pod提供新节点。Karpenter评估了挂起的pod的聚合资源需求，并选择运行它们的最佳实例类型。它将自动扩展或终止没有任何非daemonset pod的实例，以减少浪费。它还支持整合功能，该功能将积极移动pod，并用更便宜的版本删除或替换节点，以降低集群成本。

  • Karpenter 是aws为 k8s 构建的能用于生产环境的开源的工作节点动态调度控制器。

  • 在Karpenter推出之前，Kubernetes用户主要依靠Amazon EC2 Auto Scaling组和Kubernetes Cluster Autoscaler（CAS）来动态调整其集群的计算容量。

  • 相较于传统的 Cluster Autoscaler 工具，Karpenter 具有调度速度快、更灵活、资源利用率高等众多优势，另外，Karpenter与Kubernetes版本没有那么紧密耦合（像CAS那样），所以其是 EKS 自动扩缩容的首选方案，两者的比较可以参考下图。

特 性 Cluster Autoscaler Karpenter
资源管理 Cluster Autoscaler基于现有节点的资源利用率采用反应性方法来扩展节点。 Karpenter基于未调度的Pod的当前资源需求采取主动方法来进行节点预配。
节点管理 Cluster Autoscaler根据当前工作负载的资源需求来管理节点，使用预定义的自动缩放组。 Karpenter根据自定义预配程序的配置来扩展、预配和管理节点。
扩展 Cluster Autoscaler更专注于节点级别的扩展，这意味着它可以有效地添加更多的节点以满足需求的增加。 但这也意味着它在缩减资源方面可能不太有效。 Karpenter根据特定的工作负载需求提供更有效和精细的扩展功能。换句话说，它根据实际使用情况进行扩展。它还允许用户指定特定的扩展策略或规则以满足其需求。
调度 使用Cluster Autoscaler进行调度更简单，因为它是根据工作负载的当前需求设计的进行扩展或缩减。 Karpenter可以根据可用区和资源需求有效地调度工作负载。它可以尝试通过Spot实例来优化成本，但它不会知道你已经在aws帐号中做的任何承诺，如RI（预留实例）或Savings Plans（储蓄计划）。

Karpenter 运行环境准备

设置环境变量

1
2
3
4
5
6
7
8
9
10
11
12
13
# aws认证profile
$ AWS_PROFILE=eks-us-west-2
# EKS集群名称
$ CLUSTER_NAME=eks-lexing
# AWS分区
# if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov
$ AWS_PARTITION="aws"
# EKS所在的Region
$ AWS_REGION="$(aws configure list | grep region | tr -s " " | cut -d" " -f3)"
# EKS配置OIDC的Endpoint
$ OIDC_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.identity.oidc.issuer" --output text)"
# aws帐号ID
$ AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)

创建 Karpenter 的 node 需要的 role

  • 1.创建role

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ echo '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}' > node-trust-policy.json

$ aws iam create-role --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --assume-role-policy-document file://node-trust-policy.json
{
    "Role": {
        "Path": "/",
        "RoleName": "KarpenterNodeRole-eks-lexing",
        "RoleId": "AROA22DP3G4GH4RZQFGR7",
        "Arn": "arn:aws:iam::743263909644:role/KarpenterNodeRole-eks-lexing",
        "CreateDate": "2023-07-19T07:20:17+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "ec2.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

  • 2.给这个 role 添加 policy

1
2
3
4
5
6
7
8
9
10
11
$ aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKSWorkerNodePolicy

$ aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKS_CNI_Policy

$ aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

$ aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonSSMManagedInstanceCore

  • 3.把 role 授予 EC2 的 instance profile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ aws iam create-instance-profile \
    --instance-profile-name "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"
{
    "InstanceProfile": {
        "Path": "/",
        "InstanceProfileName": "KarpenterNodeInstanceProfile-eks-lexing",
        "InstanceProfileId": "AIPA22DP3G4GBMSNSBEQF",
        "Arn": "arn:aws:iam::743263909644:instance-profile/KarpenterNodeInstanceProfile-eks-lexing",
        "CreateDate": "2023-07-19T07:24:19+00:00",
        "Roles": []
    }
}

$ aws iam add-role-to-instance-profile \
    --instance-profile-name "KarpenterNodeInstanceProfile-${CLUSTER_NAME}" \
    --role-name "KarpenterNodeRole-${CLUSTER_NAME}"

创建 Karpenter controller 需要的 role

  • 1.创建role

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
$ cat << EOF > controller-trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "${OIDC_ENDPOINT#*//}:aud": "sts.amazonaws.com",
                    "${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:karpenter:karpenter"
                }
            }
        }
    ]
}
EOF

$ aws iam create-role --role-name KarpenterControllerRole-${CLUSTER_NAME} \
    --assume-role-policy-document file://controller-trust-policy.json
{
    "Role": {
        "Path": "/",
        "RoleName": "KarpenterControllerRole-eks-lexing",
        "RoleId": "AROA22DP3G4GLRLLAPW6T",
        "Arn": "arn:aws:iam::743263909644:role/KarpenterControllerRole-eks-lexing",
        "CreateDate": "2023-07-19T07:28:09+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "arn:aws:iam::743263909644:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/1029FF88CB872B6B7A1CC65D44191A56"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "oidc.eks.us-west-2.amazonaws.com/id/1029FF88CB872B6B7A1CC65D44191A56:aud": "sts.amazonaws.com",
                            "oidc.eks.us-west-2.amazonaws.com/id/1029FF88CB872B6B7A1CC65D44191A56:sub": "system:serviceaccount:karpenter:karpenter"
                        }
                    }
                }
            ]
        }
    }
}

  • 2.为role配置policy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
$ cat << EOF > controller-policy.json
{
    "Statement": [
        {
            "Action": [
                "ssm:GetParameter",
                "ec2:DescribeImages",
                "ec2:RunInstances",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeLaunchTemplates",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeInstanceTypeOfferings",
                "ec2:DescribeAvailabilityZones",
                "ec2:DeleteLaunchTemplate",
                "ec2:CreateTags",
                "ec2:CreateLaunchTemplate",
                "ec2:CreateFleet",
                "ec2:DescribeSpotPriceHistory",
                "pricing:GetProducts"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": "Karpenter"
        },
        {
            "Action": "ec2:TerminateInstances",
            "Condition": {
                "StringLike": {
                    "ec2:ResourceTag/karpenter.sh/provisioner-name": "*"
                }
            },
            "Effect": "Allow",
            "Resource": "*",
            "Sid": "ConditionalEC2Termination"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}",
            "Sid": "PassNodeIAMRole"
        },
        {
            "Effect": "Allow",
            "Action": "eks:DescribeCluster",
            "Resource": "arn:${AWS_PARTITION}:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}",
            "Sid": "EKSClusterEndpointLookup"
        }
    ],
    "Version": "2012-10-17"
}
EOF

$ aws iam put-role-policy --role-name KarpenterControllerRole-${CLUSTER_NAME} \
    --policy-name KarpenterControllerPolicy-${CLUSTER_NAME} \
    --policy-document file://controller-policy.json

为所有子网和安全组添加标签

  • 1.为节点组内的子网打标签

1
2
3
4
5
6
$ for NODEGROUP in $(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} --query 'nodegroups' --output text)
  do aws ec2 create-tags \
        --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \
        --resources $(aws eks describe-nodegroup --cluster-name ${CLUSTER_NAME} \
        --nodegroup-name $NODEGROUP --query 'nodegroup.subnets' --output text )
  done

  • 2.给托管节点组的运行模版的安全组打标签

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 获取节点组
$ NODEGROUP=$(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} \
    --query 'nodegroups[0]' --output text)

# 获取节点组的启动模板及其版本号
$ LAUNCH_TEMPLATE=$(aws eks describe-nodegroup --cluster-name ${CLUSTER_NAME} \
    --nodegroup-name ${NODEGROUP} --query 'nodegroup.launchTemplate.{id:id,version:version}' \
    --output text | tr -s "\t" ",")

# 获取启动模板的安全组
$ SECURITY_GROUPS=$(aws ec2 describe-launch-template-versions \
    --launch-template-id ${LAUNCH_TEMPLATE%,*} --versions ${LAUNCH_TEMPLATE#*,} \
    --query 'LaunchTemplateVersions[0].LaunchTemplateData.[NetworkInterfaces[0].Groups||SecurityGroupIds]' \
    --output text)

# 为安全组打标签
$ aws ec2 create-tags \
    --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \
    --resources ${SECURITY_GROUPS}

更新 aws-auth ConfigMap

  • 将上面为node创建的role加入到集群权限

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ kubectl edit configmap aws-auth -n kube-system

# 在 mapRoles 下添加如下内容
- groups:
  - system:bootstrappers
  - system:nodes
  rolearn: arn:aws:iam::743263909644:role/KarpenterNodeRole-eks-lexing
  username: system:node:{{EC2PrivateDNSName}}

# 或者通过命令行直接添加
$ eksctl create iamidentitymapping --cluster eks-lexing \
  --arn arn:aws:iam::743263909644:role/KarpenterNodeRole-eks-lexing \
  --username system:node:{{EC2PrivateDNSName}} \
  --group system:bootstrappers \
  --group system:nodes \
  --no-duplicate-arns

  • 编辑后完整的内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::743263909644:role/eksctl-eks-lexing-nodegroup-ng-4d-NodeInstanceRole-Y28EPJO9XYDG
      username: system:node:{{EC2PrivateDNSName}}
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::743263909644:role/KarpenterNodeRole-eks-lexing
      username: system:node:{{EC2PrivateDNSName}}
  mapUsers: |
    - groups:
      - eks-console-dashboard-full-access-group
      userarn: arn:aws:iam::743263909644:user/ekstest
kind: ConfigMap
metadata:
  creationTimestamp: "2023-06-28T06:30:56Z"
  name: aws-auth
  namespace: kube-system
  resourceVersion: "7544319"
  uid: 5e488d60-6ce0-4657-9a08-bbd17d048f0c

  • 查看授权信息

1
2
3
4
5
$ eksctl get iamidentitymapping --cluster eks-lexing
ARN												USERNAME				GROUPS					ACCOUNT
arn:aws:iam::743263909644:role/KarpenterNodeRole-eks-lexing					system:node:{{EC2PrivateDNSName}}	system:bootstrappers,system:nodes
arn:aws:iam::743263909644:role/eksctl-eks-lexing-nodegroup-ng-4d-NodeInstanceRole-Y28EPJO9XYDG	system:node:{{EC2PrivateDNSName}}	system:bootstrappers,system:nodes
arn:aws:iam::743263909644:user/ekstest													eks-console-dashboard-full-access-group

部署 Karpenter

  • 1.设置环境变量

1
2
# 当前最新版是 v0.29.1 , https://github.com/aws/karpenter/releases
export KARPENTER_VERSION=v0.29.1

  • 2.创建 karpenter.yaml 模版

1
2
3
4
5
6
7
8
9
$ helm template karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} \
    --namespace karpenter \
    --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
    --set settings.aws.clusterName=${CLUSTER_NAME} \
    --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}" \
    --set controller.resources.requests.cpu=1 \
    --set controller.resources.requests.memory=1Gi \
    --set controller.resources.limits.cpu=1 \
    --set controller.resources.limits.memory=1Gi > karpenter.yaml

  • 3.设置节点亲和性，编辑karpenter.yaml，找到karpenter deployment的亲和性配置，修改为如下内容，注意这里ng-4d9024eb要替换为你的${NODEGROUP}。关节K8s节点亲和性的介绍可以参考官方文档亲和性与反亲和性

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: karpenter.sh/provisioner-name
            operator: DoesNotExist
        - matchExpressions:
          - key: eks.amazonaws.com/nodegroup
            operator: In
            values:
            - ng-4d9024eb
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: "kubernetes.io/hostname"

为其他关键集群工作负载设置nodeAffinity

  • 例如 CoreDNS，Controller，CNI，CSI 和 Operator 等，这些 workload 对弹性要求不高但是稳定性要求比较高，建议部署在创建EKS时的节点组运行。
  • 为这些关键负载设置nodeAffinity
1
2
3
4
5
6
7
8
9
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: eks.amazonaws.com/nodegroup
            operator: In
            values:
            - ${NODEGROUP}
  • 比如设置k edit deploy ebs-csi-controller -n kube-system，添加好nodeAffinity后保存，然后查看对应的pod是否重启成功，如果一只处于pending状态，可以试着重启
1
2
$ k scale deploy ebs-csi-controller --replicas 0 -n kube-system
$ k scale deploy ebs-csi-controller --replicas 2 -n kube-system
  • 设置好所有关键负载后，可以重启一下 karpenter
1
2
$ k scale deploy karpenter --replicas 0 -n karpenter
$ k scale deploy karpenter --replicas 2 -n karpenter

  • 查看 karpenter 日志是否正常

1
$ k logs deployments/karpenter -f -n karpenter

  • 4.部署 karpenter 及其 相关资源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 创建Namespace
$ kubectl create namespace karpenter

# 下载karpenter 相关资源
# karpenter的版本分支，正常karpenter发布时都会打branch，比如：release-v0.28.1，不过我在github上没有找到v0.29.1的branch，所以这里就指定 main 了
$ KARPENTER_BRANCH=main
$ curl -O "https://raw.githubusercontent.com/aws/karpenter/${KARPENTER_BRANCH}/pkg/apis/crds/karpenter.sh_provisioners.yaml"
$ curl -O "https://raw.githubusercontent.com/aws/karpenter/${KARPENTER_BRANCH}/pkg/apis/crds/karpenter.k8s.aws_awsnodetemplates.yaml"
$ curl -O "https://raw.githubusercontent.com/aws/karpenter/${KARPENTER_BRANCH}/pkg/apis/crds/karpenter.sh_machines.yaml "

# 部署karpenter 相关资源
$ kubectl apply -f karpenter.sh_provisioners.yaml
$ kubectl apply -f karpenter.k8s.aws_awsnodetemplates.yaml
$ kubectl apply -f karpenter.sh_machines.yaml

# 部署karpenter
$ k apply -f karpenter.yaml

  • 5.创建默认的供应者(provisioner)

    • Provisioner与AWSNodeTemplate是karpenter在K8s中的自定义资源。
    • Provisioner 对 Karpenter 可创建的节点以及可在这些节点上运行的 Pod 设置约束。如果没有配置至少一个Provisioner，Karpenter 将不会执行任何操作。
    • 关于Provisioner支持的配置项可以参考官方文档，比如下面就限制了被管控的节点必须符合两个条件：

    1.实例类别必须在[c, m, r]中
    2.实例的生成代次必须大于2。比如 实例类型为 c1.xxx，m1.xxx，m2.xxx就不符合要求

    • AWSNodeTemplate节点模板启用AWS特定设置的配置。关于AWSNodeTemplate支持的配置项可以参考官方文档，比如默认节点关联的存储为20G gp3，如果要修改为40G可以在spec下指定如下内容，先创建后编辑也可以，但只有修改后新创建的节点才会使用新的配置。
    1
    2
    3
    4
    5
    6
    blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeType: gp3
        volumeSize: 40Gi
        deleteOnTermination: true

    • Provisioner需要与AWSNodeTemplate关联使用，通过在Provisioner的providerRef中指定AWSNodeTemplate的name进行关联，一个AWSNodeTemplate可以被多个Provisioner关联。

    • 配置的每个 Provisioner 均由 Karpenter 循环遍历。在 Provisioner 中定义污点以限制可以在 Karpenter 创建的节点上运行的 Pod。建议创建互斥的 Provisioner。因此任何 Pod 都不应该匹配多个 Provisioner。如果匹配多个Provisioner，Karpenter将使用权重最高的Provisioner 。关于K8s中污点的介绍可以参考官方文档污点和容忍度

    • 下面是一个最基本的Provisioner定义，你可以根据需要创建自己的Provisioner，可以参考官方文档或者查看provisioner examples中的示例。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 创建 default Provisioner
$ cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: karpenter.k8s.aws/instance-category
      operator: In
      values: [c, m, r]
    - key: karpenter.k8s.aws/instance-generation
      operator: Gt
      values: ["2"]
  providerRef:
    name: default
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector:
    karpenter.sh/discovery: "${CLUSTER_NAME}"
  securityGroupSelector:
    karpenter.sh/discovery: "${CLUSTER_NAME}"
EOF

  • 查看 karpenter 状态

1
2
3
4
$ k get pod -n karpenter
NAME                         READY   STATUS    RESTARTS   AGE
karpenter-789bbcbfd7-htj6z   1/1     Running   0          101s
karpenter-789bbcbfd7-rlxfp   1/1     Running   0          101s

  • 查看 karpenter 日志是否正常

1
$ k logs deployments/karpenter -f -n karpenter

  • karpenter创建新的节点时不会在原有的节点组中进行，所以为了摆脱从节点组添加的实例，我们可以将节点组缩小到最小大小

1
2
3
4
# 如果您有一个多AZ节点组，我们建议至少2个实例。
$ aws eks update-nodegroup-config --cluster-name ${CLUSTER_NAME} \
    --nodegroup-name ${NODEGROUP} \
    --scaling-config "minSize=2,maxSize=2,desiredSize=2"

  • 关停Cluster Autoscaler(CAS)
    如果EKS中已经开启了CAS，则安装Karpenter后需要关闭CAS

1
$ kubectl scale deploy/cluster-autoscaler -n kube-system --replicas=0

测试

  • 查看当前node信息

1
2
3
4
$ k get node
NAME                                           STATUS   ROLES    AGE   VERSION
ip-192-168-16-155.us-west-2.compute.internal   Ready    <none>   19d   v1.26.4-eks-0a21954
ip-192-168-48-14.us-west-2.compute.internal    Ready    <none>   19d   v1.26.4-eks-0a21954
1
2
3
4
5
6
# 可以看到node数量已经变为3了，说明扩容成功
$ k get node
NAME                                           STATUS   ROLES    AGE   VERSION
ip-192-168-16-155.us-west-2.compute.internal   Ready    <none>   19d   v1.26.4-eks-0a21954
ip-192-168-34-126.us-west-2.compute.internal   Ready    <none>   47s   v1.26.6-eks-a5565ad
ip-192-168-48-14.us-west-2.compute.internal    Ready    <none>   19d   v1.26.4-eks-0a21954

  • 缩容测试用的deploy，副本数设置为1，过一会发现node节点并没有被终止，这是为什么呢？

默认情况下，Karpenter不会主动终止节点，需要为其设置终止节点的方式，参考Karpenter官方文档Deprovisioning部分

  • 在Provisioner中设置节点终止的方式

    • spec.ttlSecondsAfterEmpty: 当最后一个工作负载（非守护程序集）pod停止在节点上运行时，Karpenter会注意到。从那时起，Karpenter在提供程序中等待ttlSecondsAfterEmpty设置的秒数，然后Karpenter请求删除节点。此功能可以通过删除不再用于工作负载的节点来降低成本。
    • spec.ttlSecondsUntilExpired: Karpenter 将根据Provisioner的ttlSecondsUntilExpired值将节点注释为过期，并在节点生存了设定秒数后取消配置节点。节点过期的一种用例是定期回收节点。
    • spec.consolidation.enabled: 实现整合，通过删除不需要的节点和缩减无法删除的节点的规模来降低集群成本。与ttlSecondsAfterEmpty参数互斥。
1
2
# 编辑 default provisioner，并为其指定 spec.ttlSecondsAfterEmpty: 30，表示空闲超过30秒则终止节点。
$ k edit provisioners.karpenter.sh default
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"karpenter.sh/v1alpha5","kind":"Provisioner","metadata":{"annotations":{
  creationTimestamp: "2023-07-19T09:49:43Z"
  generation: 3
  name: default
  resourceVersion: "7987578"
  uid: 5716142b-2b9f-493c-a049-e4304cdf82d4
spec:
  providerRef:
    name: default
  requirements:
  - key: karpenter.k8s.aws/instance-category
    operator: In
    values:
    - c
    - m
    - r
  - key: karpenter.k8s.aws/instance-generation
    operator: Gt
    values:
    - "2"
  - key: kubernetes.io/os
    operator: In
    values:
    - linux
  - key: kubernetes.io/arch
    operator: In
    values:
    - amd64
  - key: karpenter.sh/capacity-type
    operator: In
    values:
    - on-demand
  ttlSecondsAfterEmpty: 30
status:
  resources:
    cpu: "2"
    ephemeral-storage: 20959212Ki
    memory: 3900360Ki
    pods: "29"

  • 等待30秒后查看node情况，新创建的node已经成功终止

1
2
3
4
$ k get node
NAME                                           STATUS   ROLES    AGE   VERSION
ip-192-168-16-155.us-west-2.compute.internal   Ready    <none>   19d   v1.26.4-eks-0a21954
ip-192-168-48-14.us-west-2.compute.internal    Ready    <none>   19d   v1.26.4-eks-0a21954

  • 创建和终止节点的过程可以通过日志进行观察

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ k logs deployments/karpenter -f -n karpenter

# 以下创建节点的日志
2023-07-20T09:03:54.763Z	INFO	controller.provisioner	found provisionable pod(s)	{"commit": "5b469b8-dirty", "pods": 17}
2023-07-20T09:03:54.763Z	INFO	controller.provisioner	computed new machine(s) to fit pod(s)	{"commit": "5b469b8-dirty", "machines": 1, "pods": 17}
2023-07-20T09:03:54.788Z	INFO	controller.provisioner	created machine	{"commit": "5b469b8-dirty", "provisioner": "default", "requests": {"cpu":"155m","memory":"120Mi","pods":"21"}, "instance-types": "c3.2xlarge, c3.large, c3.xlarge, c4.2xlarge, c4.large and 95 other(s)"}
2023-07-20T09:03:54.978Z	DEBUG	controller.machine.lifecycle	created launch template	{"commit": "5b469b8-dirty", "machine": "default-c6jkx", "provisioner": "default", "launch-template-name": "karpenter.k8s.aws/2892028901667059566", "id": "lt-0ef53ab2c4909f493"}
2023-07-20T09:03:55.108Z	DEBUG	controller.machine.lifecycle	created launch template	{"commit": "5b469b8-dirty", "machine": "default-c6jkx", "provisioner": "default", "launch-template-name": "karpenter.k8s.aws/1810058814439854165", "id": "lt-02cda7252a2d29e16"}
2023-07-20T09:03:57.095Z	INFO	controller.machine.lifecycle	launched machine	{"commit": "5b469b8-dirty", "machine": "default-c6jkx", "provisioner": "default", "provider-id": "aws:///us-west-2d/i-0bb92e04b7192cafd", "instance-type": "c6a.large", "zone": "us-west-2d", "capacity-type": "on-demand", "allocatable": {"cpu":"1930m","ephemeral-storage":"17Gi","memory":"3114Mi","pods":"29"}}
2023-07-20T09:04:14.473Z	DEBUG	controller.machine.lifecycle	registered machine	{"commit": "5b469b8-dirty", "machine": "default-c6jkx", "provisioner": "default", "provider-id": "aws:///us-west-2d/i-0bb92e04b7192cafd", "node": "ip-192-168-79-236.us-west-2.compute.internal"}
2023-07-20T09:04:28.620Z	DEBUG	controller.machine.lifecycle	initialized machine{"commit": "5b469b8-dirty", "machine": "default-c6jkx", "provisioner": "default", "provider-id": "aws:///us-west-2d/i-0bb92e04b7192cafd", "node": "ip-192-168-79-236.us-west-2.compute.internal"}


# 以下是终止节点的日志
2023-07-20T09:05:46.420Z	DEBUG	controller.machine.disruption	marking machine as empty	{"commit": "5b469b8-dirty", "machine": "default-c6jkx"}
2023-07-20T09:06:20.642Z	INFO	controller.deprovisioning	deprovisioning via emptiness delete, terminating 1 machines ip-192-168-79-236.us-west-2.compute.internal/c6a.large/on-demand	{"commit": "5b469b8-dirty"}
2023-07-20T09:06:20.699Z	INFO	controller.termination	cordoned node	{"commit": "5b469b8-dirty", "node": "ip-192-168-79-236.us-west-2.compute.internal"}
2023-07-20T09:06:21.026Z	INFO	controller.termination	deleted node	{"commit": "5b469b8-dirty", "node": "ip-192-168-79-236.us-west-2.compute.internal"}
2023-07-20T09:06:21.288Z	INFO	controller.machine.termination	deleted machine	{"commit": "5b469b8-dirty", "machine": "default-c6jkx", "node": "ip-192-168-79-236.us-west-2.compute.internal", "provisioner": "default", "provider-id": "aws:///us-west-2d/i-0bb92e04b7192cafd"}

小贴士
当Karpenter管理的Node节点由于某种原因不可用时(比如我们在AWS控制台终止了EC2或通过命令行删除节点k delete node nodeName)，Karpenter会立刻为我们创建一个新的Node节点，并在其上重启Pod。

参考资料

从Cluster Autoscaler迁移
EKS Cluster Autoscaler 迁移 Karpenter 实践
Karpenter Best Practices