Too Many useless clusters in gateway envoy config

Istio discovers all Services and Endpoints in the cluster and put the cluster info and endpoint info to gateways

This is fixed in 1.9.x and controlled by istiod environment variable PILOT_FILTER_GATEWAY_CLUSTER_CONFIG: true (default value false)

Action

• upgrade istio 1.9.x or cherry-pick
• update istiod deploy the set PILOT_FILTER_GATEWAY_CLUSTER_CONFIG to true

Istiod Scalability

we proposed Istio primary-remote deployment mode in same AZ, which means to major changes

1. gateway router mode is set to sni-dnat, which results double the clusters in envoy
2. istiod watch all clusters in same AZ for service, pod and endpoint info, this would impact istiod

Unnecessary APIServer calls

fixed by https://github.com/istio/istio/pull/32269

add service informer filter

currently all svc changes triggers CDS, in tess most of the services are updated by service controller, dns controller, cms controller, and there is probing annotation which updated by cms regularly, such update triggered service being enqueued in Istio and it's costy, we should add filter here, e.g. only enqueue service when spec is changed or istio recognized annotations are changed

registerHandlers(c.serviceInformer, c.queue, "Services", c.onServiceEvent, nil)

should revisit all required changes for other informers like pods