技术

Appearance

需求背景 ​

k8s的集群容器需要访问宿主机的某个服务(mysql或者其他类型的服务)，或者其他外部远程设备的服务，但是服务不在集群当中

访问外部服务 ​

访问远程外部服务，如下任选一个实现

  • 外部域名映射到内部service
  • 外部 IP 映射到内部 Service

访问当前Pod所在宿主机服务，如下任选一个实现

  • 在pod中挂载环境变量表示宿主机的IP，容器内部通过环境变量映射的IP访问服务
  • 如果是只访问当前宿主机服务，通过创建linux虚拟网桥的访问，指定一个固定的网桥IP，在容器内部访问该IP来实现访问宿主机的效果，也可以叠加外部 IP 映射到内部 Service

外部域名映射到内部service ​

mysql-service.yaml

yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  externalName: mysql.example.com
  type: ExternalName
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  externalName: mysql.example.com
  type: ExternalName

应用

$ kubectl apply -f mysql-service.yaml
$ kubectl apply -f mysql-service.yaml

创建之后，Pod 就可以通过 mysql:3306 访问外部的mysql服务（服务需要存在一个可访问的域名）

外部 IP 映射到内部 Service ​

outer-ip.yaml

注意点

  • endpoints/service的metadata.name需要保持一致
  • 如果service配置了端口名称，在endpoints的端口配置也需要配置端口名称，名称保持一致
yaml
kind: Endpoints
apiVersion: v1
metadata:
  name: outer-ip
subsets:
  - addresses:
      - ip: 192.168.0.1
    ports:
      - port: 3306
        name: tcp
---
apiVersion: v1
kind: Service
metadata:
  name: outer-ip
spec:
  ports:
    - protocol: TCP
      port: 3306
      name: tcp
      targetPort: 3306
kind: Endpoints
apiVersion: v1
metadata:
  name: outer-ip
subsets:
  - addresses:
      - ip: 192.168.0.1
    ports:
      - port: 3306
        name: tcp
---
apiVersion: v1
kind: Service
metadata:
  name: outer-ip
spec:
  ports:
    - protocol: TCP
      port: 3306
      name: tcp
      targetPort: 3306

比如在IP： 192.168.0.1有一个mysql服务，但是服务不是在集群内部的，想通过k8s service访问，就可以创建一个endpoints, service的绑定关系，在集群内部访问outer-ip:3306访问数据库

应用

$ kubectl apply -f outer-ip.yaml
$ kubectl apply -f outer-ip.yaml

查看endpoints

$ kubectl describe endpoints outer-ip
Name:         outer-ip
Namespace:    default
Labels:       <none>
Annotations:  <none>
Subsets:
  Addresses:          192.168.0.1
  NotReadyAddresses:  <none>
  Ports:
    Name  Port  Protocol
    ----  ----  --------
    tcp   3306  TCP

Events:  <none>
$ kubectl describe endpoints outer-ip
Name:         outer-ip
Namespace:    default
Labels:       <none>
Annotations:  <none>
Subsets:
  Addresses:          192.168.0.1
  NotReadyAddresses:  <none>
  Ports:
    Name  Port  Protocol
    ----  ----  --------
    tcp   3306  TCP

Events:  <none>

查看service

$ kubectl describe svc outer-ip   
Name:              outer-ip
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.1.83
IPs:               10.43.1.83
Port:              tcp  3306/TCP
TargetPort:        3306/TCP
Endpoints:         192.168.0.1:3306
Session Affinity:  None
Events:            <none>
$ kubectl describe svc outer-ip   
Name:              outer-ip
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.1.83
IPs:               10.43.1.83
Port:              tcp  3306/TCP
TargetPort:        3306/TCP
Endpoints:         192.168.0.1:3306
Session Affinity:  None
Events:            <none>

挂载Pod环境变量 ​

对于容器来说，在不与 Kubernetes 过度耦合的情况下，拥有关于自身的信息有时是很有用的，**Downward API** 允许容器在不使用 Kubernetes 客户端或 API 服务器的情况下获得自己或集群的信息

创建test.yaml

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-env
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-env
  template:
    metadata:
      labels:
        app: test-env
    spec:
      containers:
        - name: test-env
          image: alpine
          command: [ "sleep" ]
          args: [ "infinity" ]
          env:
            - name: NODE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-env
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-env
  template:
    metadata:
      labels:
        app: test-env
    spec:
      containers:
        - name: test-env
          image: alpine
          command: [ "sleep" ]
          args: [ "infinity" ]
          env:
            - name: NODE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP

上面通过环境变量NODE把k8s的status.hostIP（Pod 所在节点的主 IP 地址）挂载进入容器

应用配置

shell
$ kubectl apply -f test.yaml
$ kubectl apply -f test.yaml

查询生成的pod

shell
$ kubectl get pods|grep test-env
test-env-5f55d9d96f-kj69p           2/2     Running            0                 3m43s
$ kubectl get pods|grep test-env
test-env-5f55d9d96f-kj69p           2/2     Running            0                 3m43s

测试环境变量，宿主机IP是10.30.6.88

shell
$ kubectl exec -it test-env-5f55d9d96f-kj69p -- sh
/ # env|grep NODE
NODE=10.30.6.88
$ kubectl exec -it test-env-5f55d9d96f-kj69p -- sh
/ # env|grep NODE
NODE=10.30.6.88

更多可挂载参数查阅k8s官方文档的Downward API文档

https://kubernetes.io/zh-cn/docs/concepts/workloads/pods/downward-api/
https://kubernetes.io/zh-cn/docs/concepts/workloads/pods/downward-api/

创建linux网桥 ​

该方案参考docker服务的docker0网桥实现

Docker 服务默认会创建一个 docker0 网桥（其上有一个 docker0 内部接口），它在内核层连通了其他的物理或虚拟网卡，这就将所有容器和本地主机都放到同一个物理网络

Docker 默认指定了docker0 接口 的 IP 地址(默认172.17.0.1)和子网掩码，让主机和容器之间可以通过网桥相互通信

在Ubuntu上创建网桥采用netplan实现

（其他发行版本创建自行搜索，保证创建网桥之后可以持久化，网桥不需要连接其他网口，有网桥IP即可）

创建/etc/netplan/k8s-bridge.yaml

yaml
network:
  version: 2
  renderer: networkd
  bridges:
    k8s-bridge:
      addresses: [172.172.0.1/28]
network:
  version: 2
  renderer: networkd
  bridges:
    k8s-bridge:
      addresses: [172.172.0.1/28]

配置应用

$ netplan apply
$ netplan apply

查看网口状态信息

$ ip address show k8s-bridge
65: k8s-bridge: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 56:97:87:ba:a8:97 brd ff:ff:ff:ff:ff:ff
    inet 172.172.0.1/28 brd 172.172.0.15 scope global k8s-bridge
       valid_lft forever preferred_lft forever
$ ip address show k8s-bridge
65: k8s-bridge: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 56:97:87:ba:a8:97 brd ff:ff:ff:ff:ff:ff
    inet 172.172.0.1/28 brd 172.172.0.15 scope global k8s-bridge
       valid_lft forever preferred_lft forever

创建一个容器测试一下网络访问，容器内部ping 172.172.0.1，该IP值是固定的，即刚刚创建的网桥的IP，即使主机IP修改了，也不会影响容器的通信

$ kubectl run -it alpine-test --image=alpine -- sh
If you don't see a command prompt, try pressing enter.
/ # ping 172.172.0.1
PING 172.172.0.1 (172.172.0.1): 56 data bytes
64 bytes from 172.172.0.1: seq=0 ttl=64 time=0.209 ms
$ kubectl run -it alpine-test --image=alpine -- sh
If you don't see a command prompt, try pressing enter.
/ # ping 172.172.0.1
PING 172.172.0.1 (172.172.0.1): 56 data bytes
64 bytes from 172.172.0.1: seq=0 ttl=64 time=0.209 ms

创建endponints, service的绑定

k8s-bridge-service.yaml

yaml
kind: Endpoints
apiVersion: v1
metadata:
  name: k8s-bridge
subsets:
  - addresses:
      - ip: 172.172.0.1
    ports:
      - port: 3306
        name: tcp
---
apiVersion: v1
kind: Service
metadata:
  name: k8s-bridge
spec:
  ports:
    - protocol: TCP
      port: 3306
      name: tcp
      targetPort: 3306
kind: Endpoints
apiVersion: v1
metadata:
  name: k8s-bridge
subsets:
  - addresses:
      - ip: 172.172.0.1
    ports:
      - port: 3306
        name: tcp
---
apiVersion: v1
kind: Service
metadata:
  name: k8s-bridge
spec:
  ports:
    - protocol: TCP
      port: 3306
      name: tcp
      targetPort: 3306

如果宿主机存在外部mysql服务，则容器内部就可以通过k8s-bridge:3306访问该服务

如果存在一个宿主机上的HTTP服务，则可以用这种方式配置一个service，这样就可以在ingress配置路由访问该service

参考阅读 ​

k8s downwward API

Defining a Service for an External Database

Docker0网络及原理探究

Last updated:

Released under the MIT License.

Copyright © 2023 gongzhengyang