[ Kans 3 Study - 3w ] 1. Calico CNI & Mode 실습 환경 설정

CloudNetaStudy - Kubernets Networtk 3기 실습 스터디 게시글입니다.

 

 

Calico 실습을 위한 K8S 배포 

 

• AWS 환경에서는 k8s-rtr 은 없고, AWS 내부 라우터가 대신 라우팅을 처리합니다.

 

실습 환경 : K8S v1.30.X, 노드 OS(Ubuntu 22.04 LTS) , CNI(Calico v3.28.1, IPIP, NAT enable) , IPTABLES proxy mode

# YAML 파일 다운로드
curl -O https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/kans/kans-3w.yaml

# CloudFormation 스택 배포
# aws cloudformation deploy --template-file kans-3w.yaml --stack-name mylab --parameter-overrides KeyName=<My SSH Keyname> SgIngressSshCidr=<My Home Public IP Address>/32 --region ap-northeast-2
예시) aws cloudformation deploy --template-file kans-3w.yaml --stack-name mylab --parameter-overrides KeyName=kp-gasida SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32 --region ap-northeast-2

## Tip. 인스턴스 타입 변경 : MyInstanceType=t2.micro
예시) aws cloudformation deploy --template-file kans-3w.yaml --stack-name mylab --parameter-overrides MyInstanceType=t2.micro KeyName=kp-gasida SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32 --region ap-northeast-2

# CloudFormation 스택 배포 완료 후 k8s-m EC2 IP 출력
aws cloudformation describe-stacks --stack-name mylab --query 'Stacks[*].Outputs[0].OutputValue' --output text --region ap-northeast-2
43.203.220.23

# [모니터링] CloudFormation 스택 상태 : 생성 완료 확인
while true; do 
  date
  AWS_PAGER="" aws cloudformation list-stacks \
    --stack-status-filter CREATE_IN_PROGRESS CREATE_COMPLETE CREATE_FAILED DELETE_IN_PROGRESS DELETE_FAILED \
    --query "StackSummaries[*].{StackName:StackName, StackStatus:StackStatus}" \
    --output table
  sleep 1
done

Fri Sep 20 22:20:16 KST 2024
----------------------------------
|           ListStacks           |
+------------+-------------------+
|  StackName |    StackStatus    |
+------------+-------------------+
|  mylab     |  CREATE_COMPLETE  |
+------------+-------------------+

# k8s-m EC2 SSH 접속
ssh -i ~/.ssh/[pem키] ubuntu@$(aws cloudformation describe-stacks --stack-name mylab --query 'Stacks[*].Outputs[0].OutputValue' --output text --region ap-northeast-2)

 

• kans-3w.yaml

kans-3w.yaml

0.02MB

 

 

 

---

 

 

기본 설정 확인

# (참고) control-plane
## kubeadm init --token 123456.1234567890123456 --token-ttl 0 --pod-network-cidr=172.16.0.0/16 --apiserver-advertise-address=192.168.10.10 --service-cidr 10.200.1.0/24 --cri-socket=unix:///run/containerd/containerd.sock
# worker
## kubeadm join --token 123456.1234567890123456 --discovery-token-unsafe-skip-ca-verification 192.168.10.10:6443

#
kubectl config rename-context "kubernetes-admin@kubernetes" "HomeLab"
kubens default

#
kubectl cluster-info
kubectl get node -owide
kubectl get service,ep
kubectl get pod -A -owide

#z
tree /opt/cni/bin/
ls -l /opt/cni/bin/

#
ip -c route
ip -c addr
iptables -t filter -L
iptables -t nat -L
iptables -t filter -L | wc -l
iptables -t nat -L | wc -l

k8s-w0 인스턴스만 192.268.20.100으로 대역대 다른것 확인

 

CNI 배포 전으로 NotReady 상태

--service-cidr를 10.200.1.0/24 로 설정했기 때문에 생성되는 모든 서비스의 cluster-ip는 이 대역대 안에서 생성됨.

 

Calico 설치 후 pending 상태였던 coredns도 application ip를 할당할 수 있게되어 running 상태

 

---

 

 

• (참고) Calico CNI v3.28.1 설치 - Install , Release , IP pool(subnet)

# 모니터링
watch -d 'kubectl get pod -A -owide'

# calico cni install
## kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/calico.yaml - 서브넷 24bit 추가
# 기본 yaml 에 4946줄 이동 후 아래 내용 추가 해둠
vi calico.yaml
...
            # Block size to use for the IPv4 POOL created at startup. Block size for IPv4 should be in the range 20-32. default 24
            - name: CALICO_IPV4POOL_BLOCK_SIZE
              value: "24"
kubectl apply -f https://raw.githubusercontent.com/gasida/KANS/main/kans3/calico-kans.yaml

#
tree /opt/cni/bin/
ls -l /opt/cni/bin/
ip -c route
ip -c addr
iptables -t filter -L
iptables -t nat -L
iptables -t filter -L | wc -l
iptables -t nat -L | wc -l

# calicoctl install
curl -L https://github.com/projectcalico/calico/releases/download/v3.28.1/calicoctl-linux-amd64 -o calicoctl
chmod +x calicoctl && mv calicoctl /usr/bin
calicoctl version

# CNI 설치 후 파드 상태 확인
kubectl get pod -A -o wide

Calico 바이너리를 깔기 전, 후 /opt/cni/bin

 

---

 

K8s ops view

 

# helm show values geek-cookbook/kube-ops-view
helm repo add geek-cookbook https://geek-cookbook.github.io/charts/
helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=NodePort,service.main.ports.http.nodePort=30000 --set env.TZ="Asia/Seoul" --namespace kube-system

# 설치 확인
kubectl get deploy,pod,svc,ep -n kube-system -l app.kubernetes.io/instance=kube-ops-view

# kube-ops-view 접속 URL 확인 (1.5 , 2 배율) : [실습환경 A Type]
echo -e "KUBE-OPS-VIEW URL = http://$(curl -s ipinfo.io/ip):30000/#scale=1.5"
echo -e "KUBE-OPS-VIEW URL = http://$(curl -s ipinfo.io/ip):30000/#scale=2"

# kube-ops-view 접속 URL 확인 (1.5 , 2 배율) : [실습환경 B Type]
echo -e "KUBE-OPS-VIEW URL = http://192.168.10.10:30000/#scale=1.5"
echo -e "KUBE-OPS-VIEW URL = http://192.168.10.10:30000/#scale=2"

# (참고) 삭제
helm uninstall -n kube-system kube-ops-view

 

---

 

 metric-server

# metrics-server
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm upgrade --install metrics-server metrics-server/metrics-server --set 'args[0]=--kubelet-insecure-tls' -n kube-system

kubectl get all -n kube-system -l app.kubernetes.io/instance=metrics-server
kubectl get apiservices |egrep '(AVAILABLE|metrics)'

# 확인
kubectl top node
kubectl top pod -A --sort-by='cpu'
kubectl top pod -A --sort-by='memory'

# (참고) 삭제
helm uninstall -n kube-system metrics-server