美文网首页
istio流量路由小试牛刀

istio流量路由小试牛刀

作者: go4it | 来源:发表于2023-07-26 22:16 被阅读0次

    序

    本文主要演示一下如何在mac m2安装istio，及基于header的流量路由。

    安装istio

    下载https://github.com/istio/istio/releases/download/1.18.2/istio-1.18.2-osx-arm64.tar.gz
    解压，并将istio-1.18.2/bin添加到PATH中

    istioctl install --set profile=demo -y
✔ Istio core installed
✔ Istiod installed
✔ Egress gateways installed
✔ Ingress gateways installed
✔ Installation complete                                                                               Making this installation the default for injection and validation.

kubectl get ns
NAME                 STATUS   AGE
default              Active   75m
istio-system         Active   104s
kube-node-lease      Active   75m
kube-public          Active   75m
kube-system          Active   75m
local-path-storage   Active   75m
➜  ~ kubectl get pods -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
istio-egressgateway-75db994b58-jdztx    1/1     Running   0          61s
istio-ingressgateway-79bb75ddbb-965t8   1/1     Running   0          61s
istiod-68cb9f5cb6-jksg5                 1/1     Running   0          115s

    给default命名空间注入istio

    kubectl label namespace default istio-injection=enabled
namespace/default labeled

    部署bookinfo示例

    image.png 
    kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v1 created
deployment.apps/reviews-v2 created
deployment.apps/reviews-v3 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created

kubectl get services
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
details       ClusterIP   10.96.79.135   <none>        9080/TCP   74s
kubernetes    ClusterIP   10.96.0.1      <none>        443/TCP    78m
productpage   ClusterIP   10.96.86.114   <none>        9080/TCP   74s
ratings       ClusterIP   10.96.26.22    <none>        9080/TCP   74s
reviews       ClusterIP   10.96.26.69    <none>        9080/TCP   74s


kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
details-v1-7c7dbcb4b5-vdlkt       2/2     Running   0          59m
productpage-v1-664d44d68d-lmbz8   2/2     Running   0          59m
ratings-v1-844796bf85-vzf7g       2/2     Running   0          59m
reviews-v1-5cf854487-brgpt        2/2     Running   0          59m
reviews-v2-955b74755-8rdww        2/2     Running   0          59m
reviews-v3-797fc48bc9-xzzq8       2/2     Running   0          59m

    访问应用

    kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -sS productpage:9080/productpage | grep -o "<title>.*</title>"

    开启网关

    kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

gateway.networking.istio.io/bookinfo-gateway created
virtualservice.networking.istio.io/bookinfo created

    验证配置

    istioctl analyze

✔ No validation issues found when analyzing namespace: default.

    确定入站 IP 和端口

    kubectl get service -n istio-system
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                      AGE
istio-egressgateway    ClusterIP      10.96.85.74     <none>        80/TCP,443/TCP                                                               91m
istio-ingressgateway   LoadBalancer   10.96.159.108   <pending>     15021:32027/TCP,80:32707/TCP,443:30486/TCP,31400:32398/TCP,15443:32482/TCP   91m
istiod                 ClusterIP      10.96.8.225     <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP                                        92m

    这里没有external-ip，通过cluster-ip访问

    kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -sS 10.96.159.108/productpage | grep "Reviews served by" -A2
        <dt>Reviews served by:</dt>
        <u>reviews-v2-955b74755-twbrw</u>

    多访问几次，会发现有时候是路由到reviews的V1版本，有时候是V2版本，有时候是V3版本

    流量路由示例

    创建默认DestinationRule

    kubectl apply -f samples/bookinfo/networking/destination-rule-all.yaml
destinationrule.networking.istio.io/productpage created
destinationrule.networking.istio.io/reviews created
destinationrule.networking.istio.io/ratings created
destinationrule.networking.istio.io/details created

kubectl get destinationrules -o yaml
apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: DestinationRule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"details","namespace":"default"},"spec":{"host":"details","subsets":[{"labels":{"version":"v1"},"name":"v1"},{"labels":{"version":"v2"},"name":"v2"}]}}
    creationTimestamp: "2023-07-27T13:39:56Z"
    generation: 1
    name: details
    namespace: default
    resourceVersion: "22396"
    uid: ec433861-b9ce-40e3-96de-3ac4e0851c86
  spec:
    host: details
    subsets:
    - labels:
        version: v1
      name: v1
    - labels:
        version: v2
      name: v2
- apiVersion: networking.istio.io/v1beta1
  kind: DestinationRule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"productpage","namespace":"default"},"spec":{"host":"productpage","subsets":[{"labels":{"version":"v1"},"name":"v1"}]}}
    creationTimestamp: "2023-07-27T13:39:56Z"
    generation: 1
    name: productpage
    namespace: default
    resourceVersion: "22393"
    uid: 070bf079-610f-4066-a99d-d0fad8e4150e
  spec:
    host: productpage
    subsets:
    - labels:
        version: v1
      name: v1
- apiVersion: networking.istio.io/v1beta1
  kind: DestinationRule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"ratings","namespace":"default"},"spec":{"host":"ratings","subsets":[{"labels":{"version":"v1"},"name":"v1"},{"labels":{"version":"v2"},"name":"v2"},{"labels":{"version":"v2-mysql"},"name":"v2-mysql"},{"labels":{"version":"v2-mysql-vm"},"name":"v2-mysql-vm"}]}}
    creationTimestamp: "2023-07-27T13:39:56Z"
    generation: 1
    name: ratings
    namespace: default
    resourceVersion: "22395"
    uid: 931b7b84-f0f4-4c99-b81a-dfb004521e4b
  spec:
    host: ratings
    subsets:
    - labels:
        version: v1
      name: v1
    - labels:
        version: v2
      name: v2
    - labels:
        version: v2-mysql
      name: v2-mysql
    - labels:
        version: v2-mysql-vm
      name: v2-mysql-vm
- apiVersion: networking.istio.io/v1beta1
  kind: DestinationRule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"reviews","namespace":"default"},"spec":{"host":"reviews","subsets":[{"labels":{"version":"v1"},"name":"v1"},{"labels":{"version":"v2"},"name":"v2"},{"labels":{"version":"v3"},"name":"v3"}]}}
    creationTimestamp: "2023-07-27T13:39:56Z"
    generation: 1
    name: reviews
    namespace: default
    resourceVersion: "22394"
    uid: e11a97ae-a62c-4fbd-bcb6-990e2bc47abf
  spec:
    host: reviews
    subsets:
    - labels:
        version: v1
      name: v1
    - labels:
        version: v2
      name: v2
    - labels:
        version: v3
      name: v3
kind: List
metadata:
  resourceVersion: ""

    指定路由到v1版本

    kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml

virtualservice.networking.istio.io/productpage created
virtualservice.networking.istio.io/reviews created
virtualservice.networking.istio.io/ratings created
virtualservice.networking.istio.io/details created

kubectl get virtualservices -o yaml
apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: DestinationRule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"details","namespace":"default"},"spec":{"host":"details","subsets":[{"labels":{"version":"v1"},"name":"v1"},{"labels":{"version":"v2"},"name":"v2"}]}}
    creationTimestamp: "2023-07-27T13:39:56Z"
    generation: 1
    name: details
    namespace: default
    resourceVersion: "22396"
    uid: ec433861-b9ce-40e3-96de-3ac4e0851c86
  spec:
    host: details
    subsets:
    - labels:
        version: v1
      name: v1
    - labels:
        version: v2
      name: v2
- apiVersion: networking.istio.io/v1beta1
  kind: DestinationRule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"productpage","namespace":"default"},"spec":{"host":"productpage","subsets":[{"labels":{"version":"v1"},"name":"v1"}]}}
    creationTimestamp: "2023-07-27T13:39:56Z"
    generation: 1
    name: productpage
    namespace: default
    resourceVersion: "22393"
    uid: 070bf079-610f-4066-a99d-d0fad8e4150e
  spec:
    host: productpage
    subsets:
    - labels:
        version: v1
      name: v1
- apiVersion: networking.istio.io/v1beta1
  kind: DestinationRule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"ratings","namespace":"default"},"spec":{"host":"ratings","subsets":[{"labels":{"version":"v1"},"name":"v1"},{"labels":{"version":"v2"},"name":"v2"},{"labels":{"version":"v2-mysql"},"name":"v2-mysql"},{"labels":{"version":"v2-mysql-vm"},"name":"v2-mysql-vm"}]}}
    creationTimestamp: "2023-07-27T13:39:56Z"
    generation: 1
    name: ratings
    namespace: default
    resourceVersion: "22395"
    uid: 931b7b84-f0f4-4c99-b81a-dfb004521e4b
  spec:
    host: ratings
    subsets:
    - labels:
        version: v1
      name: v1
    - labels:
        version: v2
      name: v2
    - labels:
        version: v2-mysql
      name: v2-mysql
    - labels:
        version: v2-mysql-vm
      name: v2-mysql-vm
- apiVersion: networking.istio.io/v1beta1
  kind: DestinationRule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"reviews","namespace":"default"},"spec":{"host":"reviews","subsets":[{"labels":{"version":"v1"},"name":"v1"},{"labels":{"version":"v2"},"name":"v2"},{"labels":{"version":"v3"},"name":"v3"}]}}
    creationTimestamp: "2023-07-27T13:39:56Z"
    generation: 1
    name: reviews
    namespace: default
    resourceVersion: "22394"
    uid: e11a97ae-a62c-4fbd-bcb6-990e2bc47abf
  spec:
    host: reviews
    subsets:
    - labels:
        version: v1
      name: v1
    - labels:
        version: v2
      name: v2
    - labels:
        version: v3
      name: v3
kind: List
metadata:
  resourceVersion: ""
➜  istio-1.18.2 kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml
virtualservice.networking.istio.io/productpage created
virtualservice.networking.istio.io/reviews created
virtualservice.networking.istio.io/ratings created
virtualservice.networking.istio.io/details created
➜  istio-1.18.2 kubectl get virtualservices -o yaml
apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: VirtualService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"bookinfo","namespace":"default"},"spec":{"gateways":["bookinfo-gateway"],"hosts":["*"],"http":[{"match":[{"uri":{"exact":"/productpage"}},{"uri":{"prefix":"/static"}},{"uri":{"exact":"/login"}},{"uri":{"exact":"/logout"}},{"uri":{"prefix":"/api/v1/products"}}],"route":[{"destination":{"host":"productpage","port":{"number":9080}}}]}]}}
    creationTimestamp: "2023-07-27T13:34:16Z"
    generation: 1
    name: bookinfo
    namespace: default
    resourceVersion: "21772"
    uid: 3130ef2e-22f2-4505-983f-705f969720cd
  spec:
    gateways:
    - bookinfo-gateway
    hosts:
    - '*'
    http:
    - match:
      - uri:
          exact: /productpage
      - uri:
          prefix: /static
      - uri:
          exact: /login
      - uri:
          exact: /logout
      - uri:
          prefix: /api/v1/products
      route:
      - destination:
          host: productpage
          port:
            number: 9080
- apiVersion: networking.istio.io/v1beta1
  kind: VirtualService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"details","namespace":"default"},"spec":{"hosts":["details"],"http":[{"route":[{"destination":{"host":"details","subset":"v1"}}]}]}}
    creationTimestamp: "2023-07-27T13:41:07Z"
    generation: 1
    name: details
    namespace: default
    resourceVersion: "22528"
    uid: bb2f5625-7fd7-40c5-88a3-18d00ac92c15
  spec:
    hosts:
    - details
    http:
    - route:
      - destination:
          host: details
          subset: v1
- apiVersion: networking.istio.io/v1beta1
  kind: VirtualService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"productpage","namespace":"default"},"spec":{"hosts":["productpage"],"http":[{"route":[{"destination":{"host":"productpage","subset":"v1"}}]}]}}
    creationTimestamp: "2023-07-27T13:41:07Z"
    generation: 1
    name: productpage
    namespace: default
    resourceVersion: "22525"
    uid: 377fa7eb-00a3-4554-af95-5e438ba3e467
  spec:
    hosts:
    - productpage
    http:
    - route:
      - destination:
          host: productpage
          subset: v1
- apiVersion: networking.istio.io/v1beta1
  kind: VirtualService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"ratings","namespace":"default"},"spec":{"hosts":["ratings"],"http":[{"route":[{"destination":{"host":"ratings","subset":"v1"}}]}]}}
    creationTimestamp: "2023-07-27T13:41:07Z"
    generation: 1
    name: ratings
    namespace: default
    resourceVersion: "22527"
    uid: ef8dba74-a058-476d-b0b5-59ceb540eab2
  spec:
    hosts:
    - ratings
    http:
    - route:
      - destination:
          host: ratings
          subset: v1
- apiVersion: networking.istio.io/v1beta1
  kind: VirtualService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"reviews","namespace":"default"},"spec":{"hosts":["reviews"],"http":[{"route":[{"destination":{"host":"reviews","subset":"v1"}}]}]}}
    creationTimestamp: "2023-07-27T13:41:07Z"
    generation: 1
    name: reviews
    namespace: default
    resourceVersion: "22526"
    uid: f44879f8-a9ac-4a8b-9870-f327442134dd
  spec:
    hosts:
    - reviews
    http:
    - route:
      - destination:
          host: reviews
          subset: v1
kind: List
metadata:
  resourceVersion: ""

    将Istio配置为路由到Bookinfo微服务的v1版本，reviews服务的版本1

    kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -sS 10.96.159.108/productpage | grep "Reviews served by" -A2
        <dt>Reviews served by:</dt>
        <u>reviews-v1-5cf854487-86f8d</u>

    多执行几次发现都是路由到reviews的v1版本

    指定路由到reviews的v2版本

    kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-test-v2.yaml
virtualservice.networking.istio.io/reviews configured

kubectl get virtualservice reviews -o yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"reviews","namespace":"default"},"spec":{"hosts":["reviews"],"http":[{"match":[{"headers":{"end-user":{"exact":"jason"}}}],"route":[{"destination":{"host":"reviews","subset":"v2"}}]},{"route":[{"destination":{"host":"reviews","subset":"v1"}}]}]}}
  creationTimestamp: "2023-07-27T13:41:07Z"
  generation: 3
  name: reviews
  namespace: default
  resourceVersion: "23802"
  uid: f44879f8-a9ac-4a8b-9870-f327442134dd
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1

    可以看到名为reviews的VirtualService的generation被更新了，比其第一版新增了v2的路由，针对名为x-route-env，值为feature1的header的请求路由到v2的subset

    kubectl get svc istio-ingressgateway -n istio-system -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"istio-ingressgateway","install.operator.istio.io/owning-resource":"unknown","install.operator.istio.io/owning-resource-namespace":"istio-system","istio":"ingressgateway","istio.io/rev":"default","operator.istio.io/component":"IngressGateways","operator.istio.io/managed":"Reconcile","operator.istio.io/version":"1.18.2","release":"istio"},"name":"istio-ingressgateway","namespace":"istio-system"},"spec":{"ports":[{"name":"status-port","port":15021,"protocol":"TCP","targetPort":15021},{"name":"http2","port":80,"protocol":"TCP","targetPort":8080},{"name":"https","port":443,"protocol":"TCP","targetPort":8443},{"name":"tcp","port":31400,"protocol":"TCP","targetPort":31400},{"name":"tls","port":15443,"protocol":"TCP","targetPort":15443}],"selector":{"app":"istio-ingressgateway","istio":"ingressgateway"},"type":"LoadBalancer"}}
  creationTimestamp: "2023-07-27T09:42:26Z"
  labels:
    app: istio-ingressgateway
    install.operator.istio.io/owning-resource: unknown
    install.operator.istio.io/owning-resource-namespace: istio-system
    istio: ingressgateway
    istio.io/rev: default
    operator.istio.io/component: IngressGateways
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.18.2
    release: istio
  name: istio-ingressgateway
  namespace: istio-system
  resourceVersion: "3466"
  uid: 9699d091-4567-4c60-bc0b-366c332ca84e
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.96.159.108
  clusterIPs:
  - 10.96.159.108
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: status-port
    nodePort: 32027
    port: 15021
    protocol: TCP
    targetPort: 15021
  - name: http2
    nodePort: 32707
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 30486
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: tcp
    nodePort: 32398
    port: 31400
    protocol: TCP
    targetPort: 31400
  - name: tls
    nodePort: 32482
    port: 15443
    protocol: TCP
    targetPort: 15443
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

    可以看到http2的nodePort为32707

    kubectl get node kind-control-plane -o wide
NAME                 STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                        CONTAINER-RUNTIME
kind-control-plane   Ready    control-plane   5h28m   v1.27.3   192.168.228.2   <none>        Debian GNU/Linux 11 (bullseye)   6.3.13-orbstack-00217-g0f5ad12a9255   containerd://1.7.1

    可以看到nodeip为192.168.228.2

    于是访问http://192.168.228.2:32707/productpage，没有评分，如果以用户jason(密码jason)登录，发现有评分，Reviews served by为reviews-v2-955b74755-twbrw，路由到了v2版本

    不经过gateway，直接通过productpage的clusterip访问

    kubectl get svc productpage
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
productpage   ClusterIP   10.96.214.31   <none>        9080/TCP   31m

kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -H "Cookie:session=eyJ1c2VyIjoiamFzb24ifQ.ZMJ29Q.QzbItYVJISKKnrbii9ii0fJTU38" -sS 10.96.214.31:9080/productpage | grep "Reviews served by" -A2
        <dt>Reviews served by:</dt>
        <u>reviews-v2-955b74755-twbrw</u>

    发现也能够正确路由到reviews的v2版本

    移除jason的cookie

    kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -sS 10.96.214.31:9080/productpage | grep "Reviews served by" -A2
        <dt>Reviews served by:</dt>
        <u>reviews-v1-5cf854487-86f8d</u>

    发现路由到了reviews的v1版本

    小结

    istio于2023年7月12日正式从CNCF正式毕业了，是时候在生产上把istio用起来了，istio的sidecar路由相比于基于特定语言sdk做路由更为省事。

    doc

    相关文章

      网友评论

          本文标题：istio流量路由小试牛刀

          本文链接：https://www.haomeiwen.com/subject/fztlpdtx.html

          延伸阅读

          深度阅读

          • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

          栏目导航

          热点阅读

          关于我们|服务条款|联系我们|istio流量路由小试牛刀|投稿指南|网站地图|RSS订阅|排版工具|手机版

          提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

          Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

          备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

          本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

          所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！