istio1.7 istio-proxy容器无法启动

最近在测试环境安装istio发现,只要pod不和istiod在同一个node上,就会导致pod中 istio-proxy容器无法启动,而普通pod是没有问题的,下面是正常启动和异常启动日志对比:

正常启动日志:

2020-08-30T01:20:15.451130Z info cache Root cert has changed, start rotating root cert for SDS clients
2020-08-30T01:20:15.451172Z info cache GenerateSecret default
2020-08-30T01:20:15.451677Z info sds resource:default pushed key/cert pair to proxy
2020-08-30T01:20:15.452189Z debug envoy router [C0][S1234166092710155880] upstream headers complete: end_stream=false
2020-08-30T01:20:15.452418Z debug envoy http async http request response headers (end_stream=false):
‘:status’, ‘200’
‘content-type’, 'application/grpc

异常启动日志:

2020-08-30T01:18:46.360509Z debug envoy config Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
2020-08-30T01:18:46.360719Z debug envoy router [C0][S4218232719059193173] cluster ‘xds-grpc’ match for URL ‘/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources’
2020-08-30T01:18:46.360760Z debug envoy upstream no healthy host for HTTP connection pool
2020-08-30T01:18:46.360833Z debug envoy http async http request response headers (end_stream=true):
‘:status’, ‘200’
‘content-type’, ‘application/grpc’
‘grpc-status’, ‘14’
‘grpc-message’, 'no healthy upstream
`

下面是非正常启动的全部日志:

2020-08-30T01:18:45.328127Z info FLAG: --concurrency=“2”
2020-08-30T01:18:45.328198Z info FLAG: --disableInternalTelemetry=“false”
2020-08-30T01:18:45.328210Z info FLAG: --domain=“default.svc.cluster.local”
2020-08-30T01:18:45.328219Z info FLAG: --help=“false”
2020-08-30T01:18:45.328227Z info FLAG: --id=""
2020-08-30T01:18:45.328235Z info FLAG: --ip=""
2020-08-30T01:18:45.328243Z info FLAG: --log_as_json=“false”
2020-08-30T01:18:45.328250Z info FLAG: --log_caller=""
2020-08-30T01:18:45.328258Z info FLAG: --log_output_level=“default:info”
2020-08-30T01:18:45.328276Z info FLAG: --log_rotate=""
2020-08-30T01:18:45.328284Z info FLAG: --log_rotate_max_age=“30”
2020-08-30T01:18:45.328292Z info FLAG: --log_rotate_max_backups=“1000”
2020-08-30T01:18:45.328307Z info FLAG: --log_rotate_max_size=“104857600”
2020-08-30T01:18:45.328317Z info FLAG: --log_stacktrace_level=“default:none”
2020-08-30T01:18:45.328328Z info FLAG: --log_target="[stdout]"
2020-08-30T01:18:45.328337Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2020-08-30T01:18:45.328345Z info FLAG: --mixerIdentity=""
2020-08-30T01:18:45.328352Z info FLAG: --outlierLogPath=""
2020-08-30T01:18:45.328361Z info FLAG: --proxyComponentLogLevel=“misc:debug”
2020-08-30T01:18:45.328368Z info FLAG: --proxyLogLevel=“debug”
2020-08-30T01:18:45.328377Z info FLAG: --serviceCluster=“proxy.default”
2020-08-30T01:18:45.328386Z info FLAG: --serviceregistry=“Kubernetes”
2020-08-30T01:18:45.328393Z info FLAG: --stsPort=“0”
2020-08-30T01:18:45.328401Z info FLAG: --templateFile=""
2020-08-30T01:18:45.328409Z info FLAG: --tokenManagerPlugin=“GoogleTokenExchange”
2020-08-30T01:18:45.328417Z info FLAG: --trust-domain=“cluster.local”
2020-08-30T01:18:45.328452Z info Version 1.7.0-2022348138e47498c4b54995b4cb5a1656817c4e-Clean
2020-08-30T01:18:45.328749Z info Obtained private IP [192.244.84.221]
2020-08-30T01:18:45.329134Z info Apply proxy config from env {“proxyMetadata”:{“DNS_AGENT”:""}}

2020-08-30T01:18:45.330752Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
DNS_AGENT: “”
serviceCluster: proxy.default
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: zipkin.istio-system:9411

2020-08-30T01:18:45.330828Z info Proxy role: &model.Proxy{Type:“sidecar”, IPAddresses:[]string{“192.244.84.221”}, ID:“proxy.default”, Locality:(*envoy_config_core_v3.Locality)(nil), DNSDomain:“default.svc.cluster.local”, ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:model.XdsResourceGenerator(nil), Active:map[string]*model.WatchedResource(nil), ActiveExperimental:map[string]*model.WatchedResource(nil), RequestedTypes:struct { CDS string; EDS string; RDS string; LDS string }{CDS:"", EDS:"", RDS:"", LDS:""}}
2020-08-30T01:18:45.330837Z info JWT policy is first-party-jwt
2020-08-30T01:18:45.330913Z info PilotSAN []string{“istiod.istio-system.svc”}
2020-08-30T01:18:45.330926Z info MixerSAN []string{“spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account”}
2020-08-30T01:18:45.330971Z info sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012
2020-08-30T01:18:45.330978Z info Using user-configured CA istiod.istio-system.svc:15012
2020-08-30T01:18:45.330983Z info istiod uses self-issued certificate
2020-08-30T01:18:45.331131Z info the CA cert of istiod is: -----BEGIN CERTIFICATE-----
MIIC3jCCAcagAwIBAgIRALa6TQQG5MKWKobm/SI6W7UwDQYJKoZIhvcNAQELBQAw
GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMDA4MjgwOTI3NDZaFw0zMDA4
MjYwOTI3NDZaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQDjym42ZaoIbKDwD/sVDfubB/kZ4tcJHG6fJETY
8f6bJp+UmQABm5UIZaedStzSSJ7+CCPQQ6CBlmG1zHqdNLJ8R42mK6FeYY/17hGF
ypFdiUqM+PSSNx7iqN4iXxcdOZBJ2BZT9UWJem3muJb9yeefYd98EdWHKtgRycFY
t9oLheseskoj1Q4QKPyzCq/vzlnmWm4ZqSz/6O628jgO0CusDeLhY0LyiWRtMQuP
Ea+gSC4XJJYrPY7u387uEQBNoDBSw+mJcWwLp2MzmhryWH7HUVfMaL5kZspB3yeJ
q47MyJJDeY+Hu+iZbS4QPJu9rsGWQdQPyNoIWkqZNI2tDcgLAgMBAAGjIzAhMA4G
A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB
AQCUVaksCt78RQ6XpmK3RDRkGPZoQ1cQAd+239/nDDgN9Eya2V7dSkQ8Pf0umCmW
/LYzzWZcLbtyBokdIQLNZQzuFH1wtdL4Xus+8jQr3Bz3PEgzc6YtZMAXRtgHzcrN
75qw3OnKTBWKkcvA5fL2P5t9mFSeGurUFJXr2dYOU72CAj4c4VS3GyIf8GF4FEM1
foLLBDkuqPgqYy/5tNdBNn19+xHRE/ZIizteW/DOEzxPUArsJx2YUgnjZn/Li6G0
+xRy5AfUejLc+BIH8A+/sVwMxgMiXDljNHM1fhvcB+it2M2Mc4TnIgwkxGnm08Wd
d0GXVCbV+E3t715L92T/uOSf
-----END CERTIFICATE-----

2020-08-30T01:18:45.383577Z info sds SDS gRPC server for workload UDS starts, listening on “./etc/istio/proxy/SDS”

2020-08-30T01:18:45.383668Z info Starting proxy agent
2020-08-30T01:18:45.383784Z info sds Start SDS grpc server
2020-08-30T01:18:45.383889Z info Opening status port 15020

2020-08-30T01:18:45.384101Z info Received new config, creating new Envoy epoch 0
2020-08-30T01:18:45.384210Z info Epoch 0 starting
2020-08-30T01:18:45.397878Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster proxy.default --service-node sidecar~192.244.84.221~proxy.default~default.svc.cluster.local --local-address-ip-version v4 --log-format-prefix-with-location 0 --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l debug --component-log-level misc:debug --concurrency 2]
2020-08-30T01:18:45.479160Z warning envoy runtime Unable to use runtime singleton for feature envoy.reloadable_features.activate_fds_next_event_loop
2020-08-30T01:18:45.479294Z info envoy main initializing epoch 0 (base id=0, hot restart version=11.104)
2020-08-30T01:18:45.479304Z info envoy main statically linked extensions:
2020-08-30T01:18:45.479314Z info envoy main envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
2020-08-30T01:18:45.479325Z info envoy main envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
2020-08-30T01:18:45.479332Z info envoy main envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
2020-08-30T01:18:45.479338Z info envoy main envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
2020-08-30T01:18:45.479344Z info envoy main envoy.dubbo_proxy.serializers: dubbo.hessian2
2020-08-30T01:18:45.479350Z info envoy main envoy.compression.decompressor: envoy.compression.gzip.decompressor
2020-08-30T01:18:45.479357Z info envoy main envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
2020-08-30T01:18:45.479363Z info envoy main envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
2020-08-30T01:18:45.479368Z info envoy main envoy.dubbo_proxy.protocols: dubbo
2020-08-30T01:18:45.479374Z info envoy main envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
2020-08-30T01:18:45.479380Z info envoy main envoy.thrift_proxy.transports: auto, framed, header, unframed
2020-08-30T01:18:45.479386Z info envoy main envoy.health_checkers: envoy.health_checkers.redis
2020-08-30T01:18:45.479403Z info envoy main envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.lua, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash, istio.alpn, istio_authn, mixer
2020-08-30T01:18:45.479457Z info envoy main envoy.filters.upstream_network: envoy.filters.network.upstream.metadata_exchange

2020-08-30T01:18:45.479543Z info envoy main envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.metadata_exchange, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_cluster_rewrite, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy, forward_downstream_sni, mixer, sni_verifier
2020-08-30T01:18:45.492183Z info envoy main request header map: 496 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-method,authorization,cache-control,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,keep-alive,origin,proxy-connection,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-proto,x-ot-span-context,x-request-id
2020-08-30T01:18:45.492206Z info envoy main request trailer map: 72 bytes:
2020-08-30T01:18:45.492212Z info envoy main response header map: 352 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,cache-control,connection,content-encoding,content-length,content-type,date,etag,grpc-message,grpc-status,keep-alive,location,proxy-connection,referer,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
2020-08-30T01:18:45.492218Z info envoy main response trailer map: 96 bytes: grpc-message,grpc-status
2020-08-30T01:18:45.494583Z info envoy main admin address: 127.0.0.1:15000
2020-08-30T01:18:45.494883Z debug envoy main No overload action is configured for envoy.overload_actions.shrink_heap.
2020-08-30T01:18:45.496599Z debug envoy main No overload action is configured for envoy.overload_actions.stop_accepting_connections.
2020-08-30T01:18:45.496647Z debug envoy main No overload action is configured for envoy.overload_actions.stop_accepting_connections.
2020-08-30T01:18:45.497242Z info envoy main runtime: layers:

  • name: deprecation
    static_layer:
    envoy.deprecated_features:envoy.config.listener.v3.Listener.hidden_envoy_deprecated_use_original_dst: true
  • name: admin
    admin_layer:
    {}
    2020-08-30T01:18:45.497588Z info envoy config loading tracing configuration
    2020-08-30T01:18:45.497601Z info envoy config validating default server-wide tracing driver: envoy.zipkin
    2020-08-30T01:18:45.498028Z info envoy config loading 0 static secret(s)
    2020-08-30T01:18:45.498043Z info envoy config loading 5 cluster(s)
    2020-08-30T01:18:45.499454Z debug envoy grpc completionThread running
    2020-08-30T01:18:45.499892Z debug envoy upstream transport socket match, socket default selected for host with address 127.0.0.1:15000
    2020-08-30T01:18:45.500180Z debug envoy upstream transport socket match, socket default selected for host with address 127.0.0.1:15020
    2020-08-30T01:18:45.500359Z debug envoy upstream transport socket match, socket default selected for host with address ./etc/istio/proxy/SDS
    2020-08-30T01:18:45.501075Z debug envoy init added target SdsApi default to init manager Cluster xds-grpc
    2020-08-30T01:18:45.517258Z debug envoy upstream adding TLS initial cluster agent
    2020-08-30T01:18:45.517464Z debug envoy upstream adding TLS initial cluster prometheus_stats
    2020-08-30T01:18:45.517494Z debug envoy upstream adding TLS initial cluster sds-grpc
    2020-08-30T01:18:45.517512Z debug envoy upstream adding TLS initial cluster xds-grpc

2020-08-30T01:18:45.518716Z debug envoy init init manager Cluster prometheus_stats contains no targets
2020-08-30T01:18:45.518721Z debug envoy init init manager Cluster prometheus_stats initialized, notifying ClusterImplBase
2020-08-30T01:18:45.518728Z debug envoy upstream membership update for TLS cluster prometheus_stats added 1 removed 0
2020-08-30T01:18:45.518736Z debug envoy upstream cm init: init complete: cluster=prometheus_stats primary=0 secondary=0
2020-08-30T01:18:45.518740Z debug envoy upstream maybe finish initialize state: 0
2020-08-30T01:18:45.518745Z debug envoy upstream cm init: adding: cluster=prometheus_stats primary=0 secondary=0
2020-08-30T01:18:45.518753Z debug envoy upstream initializing Primary cluster sds-grpc completed
2020-08-30T01:18:45.518758Z debug envoy init init manager Cluster sds-grpc contains no targets

2020-08-30T01:18:45.519387Z debug envoy router [C0][S8469622345674452569] cluster ‘xds-grpc’ match for URL ‘/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources’
2020-08-30T01:18:45.519425Z debug envoy upstream no healthy host for HTTP connection pool
2020-08-30T01:18:45.520240Z debug envoy http async http request response headers (end_stream=true):
‘:status’, ‘200’
‘content-type’, ‘application/grpc’
‘grpc-status’, ‘14’
‘grpc-message’, ‘no healthy upstream’

2020-08-30T01:18:45.520273Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-08-30T01:18:45.520292Z warning envoy config Unable to establish new stream
2020-08-30T01:18:45.520299Z info envoy config loading 2 listener(s)
2020-08-30T01:18:45.520307Z debug envoy config listener #0:
2020-08-30T01:18:45.520697Z debug envoy config begin add/update listener: name=23aac0cf-1d3d-4996-8155-3c9f8efe4776 hash=16131835685062216385
2020-08-30T01:18:45.520718Z debug envoy config use full listener update path for listener name=23aac0cf-1d3d-4996-8155-3c9f8efe4776 hash=16131835685062216385
2020-08-30T01:18:45.520794Z debug envoy config filter #0:
2020-08-30T01:18:45.520804Z debug envoy config name: envoy.http_connection_manager
2020-08-30T01:18:45.520991Z debug envoy config config: {
@type”: “type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager”,
“codec_type”: “AUTO”,
“stat_prefix”: “stats”,
“route_config”: {
“virtual_hosts”: [
{
“name”: “backend”,
“domains”: [
“*”
],
“routes”: [
{
“match”: {
“prefix”: “/stats/prometheus”
},
“route”: {
“cluster”: “prometheus_stats”
}
}
]
}
]
},
“http_filters”: [
{
“name”: “envoy.router”,
“typed_config”: {
@type”: “type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
}
}
]
}

2020-08-30T01:18:45.523588Z debug envoy config http filter #0
2020-08-30T01:18:45.526235Z debug envoy filter Called AuthnFilterConfig : createEmptyConfigProto
2020-08-30T01:18:45.526515Z debug envoy config name: envoy.router
2020-08-30T01:18:45.526550Z debug envoy config config: {
@type”: “type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
}

2020-08-30T01:18:45.526912Z debug envoy config new fc_contexts has 1 filter chains, including 1 newly built
2020-08-30T01:18:45.526940Z debug envoy init added target Listener-init-target 23aac0cf-1d3d-4996-8155-3c9f8efe4776 to init manager Server
2020-08-30T01:18:45.526973Z debug envoy config Create listen socket for listener 23aac0cf-1d3d-4996-8155-3c9f8efe4776 on address 0.0.0.0:15090
2020-08-30T01:18:45.526979Z debug envoy config Set listener 23aac0cf-1d3d-4996-8155-3c9f8efe4776 socket factory local address to 0.0.0.0:15090
2020-08-30T01:18:45.526986Z debug envoy config add active listener: name=23aac0cf-1d3d-4996-8155-3c9f8efe4776, hash=16131835685062216385, address=0.0.0.0:15090
2020-08-30T01:18:45.526994Z debug envoy config listener #1:
2020-08-30T01:18:45.527088Z debug envoy config begin add/update listener: name=122bd9cd-95af-4a23-8b70-f2d4cb738448 hash=15856556719592924960
2020-08-30T01:18:45.527093Z debug envoy config use full listener update path for listener name=122bd9cd-95af-4a23-8b70-f2d4cb738448 hash=15856556719592924960
2020-08-30T01:18:45.527119Z debug envoy config filter #0:
2020-08-30T01:18:45.527124Z debug envoy config name: envoy.http_connection_manager
2020-08-30T01:18:45.527302Z debug envoy config config: {
@type”: “type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager”,
“codec_type”: “AUTO”,
“stat_prefix”: “agent”,
“route_config”: {
“virtual_hosts”: [
{
“name”: “backend”,
“domains”: [
“*”
],
“routes”: [
{
“match”: {
“prefix”: “/healthz/ready”
},
“route”: {
“cluster”: “agent”
}
}
]
}
]
},
“http_filters”: [
{
“name”: “envoy.router”,
“typed_config”: {
@type”: “type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
}
}
]
}

2020-08-30T01:18:45.527603Z debug envoy config http filter #0
2020-08-30T01:18:45.527643Z debug envoy config name: envoy.router
2020-08-30T01:18:45.527657Z debug envoy config config: {
@type”: “type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
}

2020-08-30T01:18:45.527877Z debug envoy config new fc_contexts has 1 filter chains, including 1 newly built
2020-08-30T01:18:45.527899Z debug envoy init added target Listener-init-target 122bd9cd-95af-4a23-8b70-f2d4cb738448 to init manager Server
2020-08-30T01:18:45.527928Z debug envoy config Create listen socket for listener 122bd9cd-95af-4a23-8b70-f2d4cb738448 on address 0.0.0.0:15021
2020-08-30T01:18:45.527934Z debug envoy config Set listener 122bd9cd-95af-4a23-8b70-f2d4cb738448 socket factory local address to 0.0.0.0:15021
2020-08-30T01:18:45.527939Z debug envoy config add active listener: name=122bd9cd-95af-4a23-8b70-f2d4cb738448, hash=15856556719592924960, address=0.0.0.0:15021
2020-08-30T01:18:45.527945Z info envoy config loading stats sink configuration
2020-08-30T01:18:45.529016Z debug envoy init added target LDS to init manager Server
2020-08-30T01:18:45.529452Z info envoy main starting main dispatch loop
2020-08-30T01:18:45.530763Z debug envoy upstream transport socket match, socket default selected for host with address 10.96.145.171:15012
2020-08-30T01:18:45.530790Z debug envoy upstream DNS hosts have changed for istiod.istio-system.svc
2020-08-30T01:18:45.530813Z debug envoy upstream DNS refresh rate reset for istiod.istio-system.svc, refresh rate 3000 ms
2020-08-30T01:18:45.530819Z debug envoy upstream initializing Primary cluster xds-grpc completed
2020-08-30T01:18:45.530825Z debug envoy init init manager Cluster xds-grpc initializing
2020-08-30T01:18:45.530833Z debug envoy init init manager Cluster xds-grpc initializing target SdsApi default
2020-08-30T01:18:45.530845Z debug envoy config gRPC mux addWatch for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret
2020-08-30T01:18:45.530918Z debug envoy config No stream available to sendDiscoveryRequest for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret
2020-08-30T01:18:45.530931Z debug envoy config Establishing new gRPC bidi stream for rpc StreamSecrets(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

2020-08-30T01:18:45.530957Z debug envoy router [C0][S9006466111699690305] cluster ‘sds-grpc’ match for URL ‘/envoy.service.secret.v3.SecretDiscoveryService/StreamSecrets’
2020-08-30T01:18:45.531016Z debug envoy router [C0][S9006466111699690305] router decoding headers:
‘:method’, ‘POST’
‘:path’, ‘/envoy.service.secret.v3.SecretDiscoveryService/StreamSecrets’
‘:authority’, ‘sds-grpc’
‘:scheme’, ‘http’
‘te’, ‘trailers’
‘content-type’, ‘application/grpc’
‘x-envoy-internal’, ‘true’
‘x-forwarded-for’, ‘192.244.84.221’

2020-08-30T01:18:45.531046Z debug envoy pool queueing request due to no available connections
2020-08-30T01:18:45.531053Z debug envoy pool creating a new connection
2020-08-30T01:18:45.531133Z debug envoy client [C0] connecting
2020-08-30T01:18:45.531149Z debug envoy connection [C0] connecting to ./etc/istio/proxy/SDS
2020-08-30T01:18:45.531278Z debug envoy http2 [C0] updating connection-level initial window size to 268435456
2020-08-30T01:18:45.531580Z debug envoy upstream DNS refresh rate reset for zipkin.istio-system, (failure) refresh rate 5000 ms
2020-08-30T01:18:45.531592Z debug envoy upstream initializing Primary cluster zipkin completed
2020-08-30T01:18:45.531598Z debug envoy init init manager Cluster zipkin contains no targets
2020-08-30T01:18:45.531603Z debug envoy init init manager Cluster zipkin initialized, notifying ClusterImplBase
2020-08-30T01:18:45.531612Z debug envoy upstream cm init: init complete: cluster=zipkin primary=1 secondary=0
2020-08-30T01:18:45.531617Z debug envoy upstream maybe finish initialize state: 1
2020-08-30T01:18:45.531622Z debug envoy upstream maybe finish initialize primary init clusters empty: false
2020-08-30T01:18:45.531653Z debug envoy connection [C0] connected
2020-08-30T01:18:45.531660Z debug envoy client [C0] connected
2020-08-30T01:18:45.531670Z debug envoy pool [C0] attaching to next request
2020-08-30T01:18:45.531677Z debug envoy pool [C0] creating stream
2020-08-30T01:18:45.531719Z debug envoy router [C0][S9006466111699690305] pool ready
2020-08-30T01:18:45.536124Z info sds resource:default new connection
2020-08-30T01:18:45.536312Z info sds Skipping waiting for gateway secret
2020-08-30T01:18:46.360509Z debug envoy config Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

2020-08-30T01:18:46.360719Z debug envoy router [C0][S4218232719059193173] cluster ‘xds-grpc’ match for URL ‘/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources’
2020-08-30T01:18:46.360760Z debug envoy upstream no healthy host for HTTP connection pool
2020-08-30T01:18:46.360833Z debug envoy http async http request response headers (end_stream=true):
‘:status’, ‘200’
‘content-type’, ‘application/grpc’
‘grpc-status’, ‘14’
‘grpc-message’, ‘no healthy upstream’

2020-08-30T01:18:46.360866Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-08-30T01:18:46.360895Z warning envoy config Unable to establish new stream
2020-08-30T01:18:48.132600Z debug envoy config Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

2020-08-30T01:18:48.132806Z debug envoy router [C0][S8728669482332281681] cluster ‘xds-grpc’ match for URL ‘/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources’
2020-08-30T01:18:48.132876Z debug envoy upstream no healthy host for HTTP connection pool
2020-08-30T01:18:48.132966Z debug envoy http async http request response headers (end_stream=true):
‘:status’, ‘200’
‘content-type’, ‘application/grpc’
‘grpc-status’, ‘14’
‘grpc-message’, ‘no healthy upstream’

2020-08-30T01:18:48.133010Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-08-30T01:18:48.133044Z warning envoy config Unable to establish new stream
2020-08-30T01:18:48.435753Z debug envoy config Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

2020-08-30T01:18:48.435878Z debug envoy router [C0][S3951149882507668887] cluster ‘xds-grpc’ match for URL ‘/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources’
2020-08-30T01:18:48.435917Z debug envoy upstream no healthy host for HTTP connection pool
2020-08-30T01:18:48.435993Z debug envoy http async http request response headers (end_stream=true):
‘:status’, ‘200’
‘content-type’, ‘application/grpc’
‘grpc-status’, ‘14’
‘grpc-message’, ‘no healthy upstream’

2020-08-30T01:18:48.436033Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-08-30T01:18:48.436063Z warning envoy config Unable to establish new stream
2020-08-30T01:18:48.534265Z debug envoy upstream transport socket match, socket default selected for host with address 10.96.145.171:15012
2020-08-30T01:18:48.534566Z debug envoy upstream DNS refresh rate reset for istiod.istio-system.svc, refresh rate 30000 ms
2020-08-30T01:18:50.529553Z debug envoy main flushing stats
2020-08-30T01:18:50.529595Z debug envoy main Envoy is not fully initialized, skipping histogram merge and flushing stats
2020-08-30T01:18:50.532998Z debug envoy upstream DNS refresh rate reset for zipkin.istio-system, (failure) refresh rate 5000 ms
2020-08-30T01:18:55.529808Z debug envoy main flushing stats
2020-08-30T01:18:55.530397Z debug envoy main Envoy is not fully initialized, skipping histogram merge and flushing stats
2020-08-30T01:18:55.535581Z debug envoy upstream DNS refresh rate reset for zipkin.istio-system, (failure) refresh rate 5000 ms
2020-08-30T01:19:00.530242Z debug envoy main flushing stats
2020-08-30T01:19:00.530304Z debug envoy main Envoy is not fully initialized, skipping histogram merge and flushing stats
2020-08-30T01:19:00.538701Z debug envoy upstream DNS refresh rate reset for zipkin.istio-system, (failure) refresh rate 5000 ms
2020-08-30T01:19:05.333083Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = “transport: authentication handshake failed: context deadline exceeded”

‘content-type’, ‘application/grpc’
‘grpc-status’, ‘14’
‘grpc-message’, ‘connection error: desc = “transport: authentication handshake failed: context deadline exceeded”’

2020-08-30T01:19:05.334842Z warning envoy config StreamSecrets gRPC config stream closed: 14, connection error: desc = “transport: authentication handshake failed: context deadline exceeded”
2020-08-30T01:19:05.334860Z debug envoy config gRPC update for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret failed
2020-08-30T01:19:05.334889Z debug envoy http2 [C0] stream closed: 0
2020-08-30T01:19:05.334927Z debug envoy http2 [C0] sent reset code=0
2020-08-30T01:19:05.393550Z debug envoy config Establishing new gRPC bidi stream for rpc StreamSecrets(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

2020-08-30T01:19:05.393669Z debug envoy router [C0][S12597778247121430545] cluster ‘sds-grpc’ match for URL ‘/envoy.service.secret.v3.SecretDiscoveryService/StreamSecrets’
2020-08-30T01:19:05.393767Z debug envoy router [C0][S12597778247121430545] router decoding headers:
‘:method’, ‘POST’
‘:path’, ‘/envoy.service.secret.v3.SecretDiscoveryService/StreamSecrets’
‘:authority’, ‘sds-grpc’
‘:scheme’, ‘http’
‘te’, ‘trailers’
‘content-type’, ‘application/grpc’
‘x-envoy-internal’, ‘true’
‘x-forwarded-for’, ‘192.244.84.221’

2020-08-30T01:19:05.393794Z debug envoy pool [C0] using existing connection
2020-08-30T01:19:05.393814Z debug envoy pool [C0] creating stream
2020-08-30T01:19:05.393858Z debug envoy router [C0][S12597778247121430545] pool ready
2020-08-30T01:19:05.397643Z info sds resource:default new connection
2020-08-30T01:19:05.397904Z info sds Skipping waiting for gateway secret
2020-08-30T01:19:05.530585Z debug envoy main flushing stats
2020-08-30T01:19:05.530631Z debug envoy main Envoy is not fully initialized, skipping histogram merge and flushing stats
2020-08-30T01:19:05.541372Z debug envoy upstream DNS refresh rate reset for zipkin.istio-system, (failure) refresh rate 5000 ms
2020-08-30T01:19:05.732261Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = “transport: authentication handshake failed: context deadline exceeded”

应该是DNS解析问题

请参考该issue: