[Bug]: Cannot get F5 Nginx Ingress to report node ExternalIPs to ExternalDNS integration

[artem-zinnatullin]

Version

5.2.1

What Kubernetes platforms are you running on?

Other

Steps to reproduce

1. We have k8s cluster with bus-bar ingress networking where most nodes in the cluster run ingress controller and take external HTTP(S) traffic on ports 80 and 443
2. The current setup uses now deprecated community-maintained ingress-nginx running as DaemonSet with ExternalDNS set up to source=ingress which syncs ingress-controller node external IPs to DNS and works okay
3. We're now trying to migrate to F5 Nginx Ingress (this repo), however we're facing major challenges as we can't get the F5 Ingress DaemonSet to pick up nodes' ExternalIP to get them reported to ExternalDNS

These are our Helm values for F5 Nginx Ingress (helm chart oci://ghcr.io/nginx/charts/nginx-ingress 2.3.1):

controller:
  kind: DaemonSet
  ingressClass:
    name: f5-nginx
  ingressClassResource:
    setAsDefaultIngress: true
  service:
    enabled: false
    externalTrafficPolicy: Local
  reportIngressStatus:
    enable: true
    externalService: ""
    enableLeaderElection: true
  enableCustomResources: true
  enableCertManager: true
  enableExternalDNS: true
  hostNetwork: true

Here are helm values for ingress-nginx that easily report ExternalIPs to ExternalDNS via Ingress objects (helm chart https://kubernetes.github.io/ingress-nginx 4.14.0):

controller:
  kind: DaemonSet
  ingressClass: ingress-nginx
  ingressClassResource:
    default: true
  service:
    enabled: false
  reportNodeInternalIp: false
  terminationGracePeriodSeconds: 240
  extraArgs:
    shutdown-grace-period: 180
    post-shutdown-grace-period: 1
    status-update-interval: 15
    update-status-on-shutdown: false
    election-ttl: 10s
  hostPort:
    enabled: true

Each node in the cluster has ExternalIP on eth0:

kubectl describe node x

# Real IPs changed here of course.
Addresses:
  InternalIP:  10.0.0.3
  Hostname:    xxx
  ExternalIP:  1.2.3.4

We've tried so many things, but yet to make F5 Nginx resolve the nodes' ExternalIP and report the IPs to ExternalDNS.

How does one make F5 Nginx Ingress resolve node ExternalIP and report it to ExternalDNS without using MetalLB or other LoadBalancers that add unnecessary network hops and cost?

[github-actions]

Hi @artem-zinnatullin thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

Cheers!

[AlexFenlon]

Hi @artem-zinnatullin ,

Thank you for the issue,

We will look into this soon when we get a chance.

[AlexFenlon]

Hi @artem-zinnatullin,

We have a guide that might put you in the right direction,
https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/reporting-resources-status/

[vepatel]

hey @artem-zinnatullin, can you try https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#configmap-keys external-status-address with your nodeIP and see if that works?

[artem-zinnatullin]

Hi @AlexFenlon I've checked the documentation thoroughly and it didn't really help.

@vepatel it is simply not an option for us (and many other users I'm sure) to manually specify external node ips in configs, our k8s nodes have properly exposed ExternalIPs which should be picked up:

kubectl describe node x

# Real IPs changed here of course.
Addresses:
  InternalIP:  10.0.0.3
  Hostname:    xxx
  ExternalIP:  1.2.3.4

The workaround I came up with so far is to disable ExternalDNS integration helm --set controller.enableExternalDNS=false and configure ExternalDNS to monitor Service objects helm --set sources[0]=service, then deploy Service object for each Nginx VirtualServer like so:

apiVersion: v1
kind: Service
metadata:
  name: myhost-com-dns
  # Has to match F5 Nginx Ingress Namespace, not the VirtualServer namespace.
  namespace: f5-nginx-ingress
  annotations:
    external-dns.alpha.kubernetes.io/hostname: myhost.com
    external-dns.alpha.kubernetes.io/endpoints-type: NodeExternalIP
    external-dns.alpha.kubernetes.io/ttl: "300"
spec:
  clusterIP: None
  # Select F5 Nginx Ingress Controller pods to discover their nodes ExternalIPs.
  selector:
    app.kubernetes.io/instance: f5-ingress-nginx
    app.kubernetes.io/name: nginx-ingress
  # Some port has to be declared for Service to be valid, doesn't have any effect on DNS of course.
  ports:
    - port: 80
      targetPort: 80

---

To be honest, I don't understand the idea behind built-in enableExternalDNS integration if it cannot automatically expose ExternalIPs of nodes that run ingress-controller pods for given VirtualServer objects, maybe I'm missing something?