Nginx proxy_pass DNS Cache

We use nginx to proxy to an Amazon S3 bucket to serve static content to customers. Yesterday, I noticed a high failure rate through the proxy but I couldn’t figure out why.

After some debugging, I finally discovered that the IP address nginx was hitting was different from the one that DNS was returning. It turns out that nginx resolves hostnames only once on load, meaning whatever IP address it got on load would stick around until reload.

So Amazon updated DNS for the S3 endpoint for our bucket but we were still using the old one (which was apparently failing).

Dynamic proxy_pass
Here’s how the directive might look:

server {
  listen 80;
  server_name files.example.com;

  location / {
    proxy_pass http://example.s3.amazonaws.com;
  }
}

The documentation for proxy_pass has this little gem:
​​​https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass​​

In some cases, the part of a request URI to be replaced cannot be determined:

[…]

A server name, its port and the passed URI can also be specified using variables:

[…]

In this case, the server name is searched among the described server groups, and, if not found, is determined using a resolver.

Meaning if we change proxy_pass to use a variable instead, then nginx will be forced to resolve it using a resolver which will work how we want (i.e. DNS TTLs will work).

Here’s what the “fixed” block looks like:

server {
  listen 80;
  server_name files.example.com;

  // or use whatever resolver your machine is set to use
  resolver 8.8.8.8;

  set $proxy_pass_url http://example.s3.amazonaws.com;

  location / {
    proxy_pass $proxy_pass_url;
  }
}

NGINX DNS CACHING problem

Names are resolved to IP addresses while loading a configuration. To re-resolve them on changes you have to instruct nginx to reload the configuration. Alternatively, you can:

  • use​​proxy_pass​​​ with variables and the​​resolver​​ directive to force nginx to resolve names at run-time;
  • use the​​resolve​​ parameter of the server directive in the upstream block to trigger periodic resolution of a name (available as part of the commercial subscription).

​​https://trac.nginx.org/nginx/ticket/1064​​

​​# Nginx with dynamic upstreams​​

....
write the configuration like this:

resolver 172.16.0.23;
set $upstream_endpoint http://service-1234567890.us-east-1.elb.amazonaws.com;
location / {
    proxy_pass $upstream_endpoint;
}

This will work and Nginx will honour the TTL of the DNS record and re-resolve it in case a request comes in and the cached entry has expired. But why is that?

The answer is in part found by the end of the documentation for the proxy_pass directive which states:

A server name, its port and the passed URI can also be specified using variables:

proxy_pass http://$host$uri;

or even like this:

proxy_pass $request;

In this case, the server name is searched among the described server groups, and, if not found, is determined using a resolver.

​​https://tenzer.dk/nginx-with-dynamic-upstreams/​​

Kotlin 开发者社区

Nginx proxy_pass DNS Cache_开发者社区


国内第一Kotlin 开发者社区公众号，主要分享、交流 Kotlin 编程语言、Spring Boot、Android、React.js/Node.js、函数式编程、编程思想等相关主题。