CloudNetaStudy - Kubernets Networtk 3기 실습 스터디 게시글입니다.
[ 요약 ]
쿠버네티스는 네트워크 모델의 요건을 만족하는 CNI 플러그인이 있고, 대표적으로 'Calico, Cilium 등'이 있습니다.
- K8S 버전은 1.30.4 : CNI 는 flannel(v0.25.6)
- kind & Flannel 배포 - Docs Blog Github : Windows 사용자 → bridge 플러그인 바이러니 파일은 intel cpu PC에서 생성 후 첨부하기
# cat <<EOF> kind-cni.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane labels: mynode: control-plane extraPortMappings: - containerPort: 30000 hostPort: 30000 - containerPort: 30001 hostPort: 30001 - containerPort: 30002 hostPort: 30002 kubeadmConfigPatches: - | kind: ClusterConfiguration controllerManager: extraArgs: bind-address: 0.0.0.0 etcd: local: extraArgs: listen-metrics-urls: http://0.0.0.0:2381 scheduler: extraArgs: bind-address: 0.0.0.0 - | kind: KubeProxyConfiguration metricsBindAddress: 0.0.0.0 - role: worker labels: mynode: worker - role: worker labels: mynode: worker2 networking: disableDefaultCNI: true EOF kind create cluster --config kind-cni.yaml --name myk8s --image kindest/node:v1.30.4 # 배포 확인 kind get clusters kind get nodes --name myk8s kubectl cluster-info # 네트워크 확인 kubectl cluster-info dump | grep -m 2 -E "cluster-cidr|service-cluster-ip-range" # 노드 확인 : CRI kubectl get nodes -o wide # 노드 라벨 확인 kubectl get nodes myk8s-control-plane -o jsonpath={.metadata.labels} | jq ... "mynode": "control-plane", ... kubectl get nodes myk8s-worker -o jsonpath={.metadata.labels} | jq kubectl get nodes myk8s-worker2 -o jsonpath={.metadata.labels} | jq # 컨테이너 확인 : 컨테이너 갯수, 컨테이너 이름 확인 docker ps docker port myk8s-control-plane docker port myk8s-worker docker port myk8s-worker2 # 컨테이너 내부 정보 확인 docker exec -it myk8s-control-plane ip -br -c -4 addr docker exec -it myk8s-worker ip -br -c -4 addr docker exec -it myk8s-worker2 ip -br -c -4 addr # docker exec -it myk8s-control-plane sh -c 'apt update && apt install tree jq psmisc lsof wget bridge-utils tcpdump iputils-ping htop git nano -y' docker exec -it myk8s-worker sh -c 'apt update && apt install tree jq psmisc lsof wget bridge-utils tcpdump iputils-ping -y' docker exec -it myk8s-worker2 sh -c 'apt update && apt install tree jq psmisc lsof wget bridge-utils tcpdump iputils-ping -y' - (참고) Deploying Flannel with helm - Link
# Needs manual creation of namespace to avoid helm error kubectl create ns kube-flannel kubectl label --overwrite ns kube-flannel pod-security.kubernetes.io/enforce=privileged # helm repo add flannel https://flannel-io.github.io/flannel/ helm show values flannel/flannel # 설치 helm install flannel --set podCidr="10.244.0.0/16" --namespace kube-flannel flannel/flannel - (참고) helm show values flannel/flannel
helm show values flannel/flannel --- global: imagePullSecrets: # - name: "a-secret-name" # The IPv4 cidr pool to create on startup if none exists. Pod IPs will be # chosen from this range. podCidr: "10.244.0.0/16" podCidrv6: "" flannel: # kube-flannel image image: repository: docker.io/flannel/flannel tag: v0.25.6 image_cni: repository: docker.io/flannel/flannel-cni-plugin tag: v1.5.1-flannel2 # flannel command arguments enableNFTables: false args: - "--ip-masq" - "--kube-subnet-mgr" # Backend for kube-flannel. Backend should not be changed # at runtime. (vxlan, host-gw, wireguard, udp) # Documentation at https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md backend: "vxlan" # Port used by the backend 0 means default value (VXLAN: 8472, Wireguard: 51821, UDP: 8285) #backendPort: 0 # MTU to use for outgoing packets (VXLAN and Wiregurad) if not defined the MTU of the external interface is used. # mtu: 1500 # # VXLAN Configs: # # VXLAN Identifier to be used. On Linux default is 1. #vni: 1 # Enable VXLAN Group Based Policy (Default false) # GBP: false # Enable direct routes (default is false) # directRouting: false # MAC prefix to be used on Windows. (Defaults is 0E-2A) # macPrefix: "0E-2A" # # Wireguard Configs: # # UDP listen port used with IPv6 # backendPortv6: 51821 # Pre shared key to use # psk: 0 # IP version to use on Wireguard # tunnelMode: "separate" # Persistent keep interval to use # keepaliveInterval: 0 # # General daemonset configs # tolerations: - effect: NoExecute operator: Exists - effect: NoSchedule operator: Exists netpol: enabled: false args: - "--hostname-override=$(MY_NODE_NAME)" - "--v=2" image: repository: registry.k8s.io/networking/kube-network-policies tag: v0.4.0 - Flannel 정보 확인
# kubectl get ds,pod,cm -n kube-flannel -owide kubectl describe cm -n kube-flannel kube-flannel-cfg # iptables 정보 확인 for i in filter nat mangle raw ; do echo ">> IPTables Type : $i <<"; docker exec -it myk8s-control-plane iptables -t $i -S ; echo; done for i in filter nat mangle raw ; do echo ">> IPTables Type : $i <<"; docker exec -it myk8s-worker iptables -t $i -S ; echo; done for i in filter nat mangle raw ; do echo ">> IPTables Type : $i <<"; docker exec -it myk8s-worker2 iptables -t $i -S ; echo; done # flannel 정보 확인 : 대역, MTU for i in myk8s-control-plane myk8s-worker myk8s-worker2; do echo ">> node $i <<"; docker exec -it $i cat /run/flannel/subnet.env ; echo; done >> node myk8s-control-plane << FLANNEL_NETWORK=10.244.0.0/16 FLANNEL_SUBNET=10.244.0.1/24 # 해당 노드에 파드가 배포 시 할당 할 수 있는 네트워크 대역 FLANNEL_MTU=65485 # MTU 지정 FLANNEL_IPMASQ=true # 파드가 외부(인터넷) 통신 시 해당 노드의 마스커레이딩을 사용 ... # 노드마다 할당된 dedicated subnet (podCIDR) 확인 kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}' ;echo # 노드 정보 중 flannel 관련 정보 확인 : VXLAN 모드 정보와, VTEP 정보(노드 IP, VtepMac) 를 확인 kubectl describe node | grep -A3 Annotations # 각 노드(?) 마다 bash 진입 후 아래 기본 정보 확인 : 먼저 worker 부터 bash 진입 후 확인하자 docker exec -it myk8s-worker bash docker exec -it myk8s-worker2 bash docker exec -it myk8s-control-plane bash ---------------------------------------- # 호스트 네트워크 NS와 flannel, kube-proxy 컨테이너의 네트워크 NS 비교 : 파드의 IP와 호스트(서버)의 IP를 비교해보자! lsns -p 1 lsns -p $(pgrep flanneld) lsns -p $(pgrep kube-proxy) # 기본 네트워크 정보 확인 ip -c -br addr ip -c link | grep -E 'flannel|cni|veth' -A1 ip -c addr ip -c -d addr show cni0 # 네트워크 네임스페이스 격리 파드가 1개 이상 배치 시 확인됨 ip -c -d addr show flannel.1 5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether 1e:81:fc:d5:8a:42 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 vxlan id 1 local 192.168.100.10 dev enp0s8 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 172.16.0.0/32 brd 172.16.0.0 scope global flannel.1 brctl show bridge name bridge id STP enabled interfaces cni0 8000.663bf746b6a8 no vethbce1591c vethc17ba51b vethe6540260 # 라우팅 정보 확인 : 다른 노드의 파드 대역(podCIDR)의 라우팅 정보가 업데이트되어 있음을 확인 ip -c route default via 172.18.0.1 dev eth0 10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink 10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1 10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink 172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.3 # flannel.1 인터페이스를 통한 ARP 테이블 정보 확인 : 다른 노드의 flannel.1 IP와 MAC 정보를 확인 ip -c neigh show dev flannel.1 # 브리지 fdb 정보에서 해당 MAC 주소와 통신 시 각 노드의 enp0s8 bridge fdb show dev flannel.1 # 다른 노드의 flannel.1 인터페이스로 ping 통신 : VXLAN 오버레이를 통해서 통신 ping -c 1 10.244.0.0 ping -c 1 10.244.1.0 ping -c 1 10.244.2.0 # iptables 필터 테이블 정보 확인 : 파드의 10.244.0.0/16 대역 끼리는 모든 노드에서 전달이 가능 iptables -t filter -S | grep 10.244.0.0 # iptables NAT 테이블 정보 확인 : 10.244.0.0/16 대역 끼리 통신은 마스커레이딩 없이 통신을 하며, # 10.244.0.0/16 대역에서 동일 대역(10.244.0.0/16)과 멀티캐스트 대역(224.0.0.0/4) 를 제외한 나머지 (외부) 통신 시에는 마스커레이딩을 수행 iptables -t nat -S | grep 'flanneld masq' | grep -v '! -s' ---------------------------------------- - 파드 2개 생성
# [터미널1,2] 워커 노드1,2 - 모니터링 docker exec -it myk8s-worker bash docker exec -it myk8s-worker2 bash ----------------------------- watch -d "ip link | egrep 'cni|veth' ;echo; brctl show cni0" ----------------------------- # [터미널3] cat & here document 명령 조합으로 즉석(?) 리소스 생성 cat <<EOF | kubectl create -f - apiVersion: v1 kind: Pod metadata: name: pod-1 labels: app: pod spec: nodeSelector: kubernetes.io/hostname: myk8s-worker containers: - name: netshoot-pod image: nicolaka/netshoot command: ["tail"] args: ["-f", "/dev/null"] terminationGracePeriodSeconds: 0 --- apiVersion: v1 kind: Pod metadata: name: pod-2 labels: app: pod spec: nodeSelector: kubernetes.io/hostname: myk8s-worker2 containers: - name: netshoot-pod image: nicolaka/netshoot command: ["tail"] args: ["-f", "/dev/null"] terminationGracePeriodSeconds: 0 EOF # 파드 확인 : IP 확인 kubectl get pod -o wide - 파드 Shell 접속 후 확인 & 패킷 캡처 : Windows 사용자
kubectl exec -it pod-1 -- zsh ----------------------------- ip -c addr show eth0 # GW IP는 어떤 인터페이스인가? (1) flannel.1 (2) cni0 ip -c route route -n ping -c 1 <GW IP> ping -c 1 <pod-2 IP> # 다른 노드에 배포된 파드 통신 확인 ping -c 1 8.8.8.8 # 외부 인터넷 IP 접속 확인 curl -s wttr.in/Seoul # 외부 인터넷 도메인 접속 확인 ip -c neigh exit ----------------------------- # 패킷 캡처 후 확인 # [터미널1,2] 워커 노드1,2 docker exec -it myk8s-worker bash docker exec -it myk8s-worker2 bash ----------------------------- tcpdump -i cni0 -nn icmp tcpdump -i flannel.1 -nn icmp tcpdump -i eth0 -nn icmp tcpdump -i eth0 -nn udp port 8472 -w /root/vxlan.pcap # CTRL+C 취소 후 확인 : ls -l /root/vxlan.pcap conntrack -L | grep -i icmp ----------------------------- # [터미널3] docker cp -a myk8s-worker:/root/vxlan.pcap . # ubuntu 가상머신 내부에 파일을 호스트(윈도우)에 복사하기 : cmd 창(관리자 권한) 실행 # scp 실행(root 계정) 암호 qwe123 입력 scp root@192.168.50.10:/root/vxlan.pcap . root@192.168.50.10's password: qwe123 # D:\Vagrant-Lab\kind> dir Volume in drive D is NEW Volume Serial Number is 5263-D701 Directory of D:\Vagrant-Lab\kind 2024-09-03 오후 02:57 <DIR> . 2024-09-03 오후 02:57 <DIR> .. 2024-09-02 오후 10:12 <DIR> .vagrant 2024-09-03 오후 02:38 2,828 init_cfg.sh 2024-09-02 오후 10:08 727 Vagrantfile 2024-09-03 오후 02:57 680 vxlan.pcap 3 File(s) 4,235 bytes 3 Dir(s) 176,679,444,480 bytes free - Wireshark 실행
- Wireshark 에서 vxlan 기본 udp port 를 4789 를 사용. Flannel 은 8472 를 사용하니 udp port 정보 맞추기
- 옵션 설정 - Protocols - VXLAN : 4789 → 8472
실습 도움 툴 설치 : kube-ops-view , metrics-server , prometheus-stack ← 이후 스터디에서도 자주 사용 예정
- kube-ops-view
# helm show values geek-cookbook/kube-ops-view helm repo add geek-cookbook https://geek-cookbook.github.io/charts/ helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=NodePort,service.main.ports.http.nodePort=30000 --set env.TZ="Asia/Seoul" --namespace kube-system # 설치 확인 kubectl get deploy,pod,svc,ep -n kube-system -l app.kubernetes.io/instance=kube-ops-view # myk8s-control-plane 노드?에 배치 kubectl get nodes myk8s-control-plane -o jsonpath={.metadata.labels} | jq kubectl describe node myk8s-control-plane | grep Taints kubectl -n kube-system get deploy kube-ops-view -o yaml | k neat kubectl -n kube-system edit deploy kube-ops-view --- spec: ... template: ... spec: nodeSelector: mynode: control-plane tolerations: - key: "node-role.kubernetes.io/control-plane" operator: "Equal" effect: "NoSchedule" --- kubectl -n kube-system get pod -o wide -l app.kubernetes.io/instance=kube-ops-view # kube-ops-view 접속 URL 확인 (1.5 , 2 배율) : macOS 사용자 echo -e "KUBE-OPS-VIEW URL = http://localhost:30000/#scale=1.5" echo -e "KUBE-OPS-VIEW URL = http://localhost:30000/#scale=2" # kube-ops-view 접속 URL 확인 (1.5 , 2 배율) : Windows 사용자 echo -e "KUBE-OPS-VIEW URL = http://192.168.50.10:30000/#scale=1.5" echo -e "KUBE-OPS-VIEW URL = http://192.168.50.10:30000/#scale=2" # (참고) 삭제 helm uninstall -n kube-system kube-ops-view - metrics-server
# metrics-server helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/ helm upgrade --install metrics-server metrics-server/metrics-server --set 'args[0]=--kubelet-insecure-tls' -n kube-system kubectl get all -n kube-system -l app.kubernetes.io/instance=metrics-server kubectl get apiservices |egrep '(AVAILABLE|metrics)' # 확인 kubectl top node kubectl top pod -A --sort-by='cpu' kubectl top pod -A --sort-by='memory' # (참고) 삭제 helm uninstall -n kube-system metrics-server - prometheus-stack
# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts # 파라미터 파일 생성 cat <<EOT > monitor-values.yaml prometheus: service: type: NodePort nodePort: 30001 prometheusSpec: podMonitorSelectorNilUsesHelmValues: false serviceMonitorSelectorNilUsesHelmValues: false nodeSelector: mynode: control-plane tolerations: - key: "node-role.kubernetes.io/control-plane" operator: "Equal" effect: "NoSchedule" grafana: defaultDashboardsTimezone: Asia/Seoul adminPassword: kans1234 service: type: NodePort nodePort: 30002 nodeSelector: mynode: control-plane tolerations: - key: "node-role.kubernetes.io/control-plane" operator: "Equal" effect: "NoSchedule" defaultRules: create: false alertmanager: enabled: false EOT # 배포 kubectl create ns monitoring helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 62.3.0 -f monitor-values.yaml --namespace monitoring # 확인 helm list -n monitoring # Grafana 접속 계정 : admin / kans1234 : macOS 사용자 echo -e "Prometheus URL = http://localhost:30001" echo -e "Grafana URL = http://localhost:30002" # Grafana 접속 계정 : admin / kans1234 : Windows 사용자 echo -e "Prometheus URL = http://192.168.50.10:30001" echo -e "Grafana URL = http://192.168.50.10:30002" # (참고) helm 삭제 helm uninstall -n monitoring kube-prometheus-stack - Prometheus Target connection refused 관련 정상 동작 설정 : kind k8s yaml 파일에 이미 적용되어 있음
- kube-controller-manager , kube-scheduler , etcd , kube-proxy
# Prometheus Target connection refused 관련 정상 동작 설정 : kube-proxy kubectl edit cm -n kube-system kube-proxy ... metricsBindAddress: "0.0.0.0:10249" ## 추가 ... kubectl rollout restart daemonset -n kube-system kube-proxy # Prometheus Target connection refused 관련 정상 동작 설정 : kube-controller-manager , kube-scheduler , etcd docker exec -it myk8s-control-plane bash ---------------------------------------- cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep bind-address sed -i "s/bind-address=127.0.0.1/bind-address=0.0.0.0/g" /etc/kubernetes/manifests/kube-controller-manager.yaml cat /etc/kubernetes/manifests/kube-scheduler.yaml | grep bind-address sed -i "s/bind-address=127.0.0.1/bind-address=0.0.0.0/g" /etc/kubernetes/manifests/kube-scheduler.yaml cat /etc/kubernetes/manifests/etcd.yaml | grep 127.0.0.1:2381 sed -i "s/127.0.0.1:2381/0.0.0.0:2381/g" /etc/kubernetes/manifests/etcd.yaml ----------------------------------------
- kube-controller-manager , kube-scheduler , etcd , kube-proxy
- 설치 후 iptables 확인
# iptables 정보 확인 for i in filter nat mangle raw ; do echo ">> IPTables Type : $i <<"; docker exec -it myk8s-control-plane iptables -t $i -S ; echo; done for i in filter nat mangle raw ; do echo ">> IPTables Type : $i <<"; docker exec -it myk8s-worker iptables -t $i -S ; echo; done for i in filter nat mangle raw ; do echo ">> IPTables Type : $i <<"; docker exec -it myk8s-worker2 iptables -t $i -S ; echo; done - 실습 완료 후 kind 삭제 : kind delete cluster --name myk8s
'Kubernetes' 카테고리의 다른 글
| [ Kans 3 Study - 3w ] 2. Calico Component (2) | 2024.09.21 |
|---|---|
| [ Kans 3 Study - 3w ] 1. Calico CNI & Mode 실습 환경 설정 (0) | 2024.09.20 |
| [ Kans 3 Study - 2w ] 3. kind 활용 (0) | 2024.09.08 |
| [ Kans 3 Study - 2w ] 2. kind 기본 사용 (0) | 2024.09.08 |
| [ Kans 3 Study - 2w ] 1. Vagrant 설정 (1) | 2024.09.08 |