Understanding the “502 Bad Gateway” Error in NGINX

Every now and then, when you browse the web, you see an error message that says “502 Bad Gateway” like this:

It’s a rather cryptic message, and if this is something you encounter as a user, there’s nothing you can do other than move along and try again later. But if you’re a website administrator, and you receive reports of this error appearing, then you need to fix it as soon as possible. In this article, I’ll explain what the error means, what underlies its architecture, and some ways to fix it.

Can be a Hard Error to Reproduce

One of the frustrating things about this error is that it’s hard to reproduce. The error is caused by an upstream server not responding well to the gateway, and this is something that can fix itself if it’s temporary. So when debugging this issue, it’s important to rely heavily upon logs that can tell you when the error happened and see the time stamps so you can evaluate the health of the server at a given point in time with proper monitoring, instead of what the server looks like at a later date, when everything is fine.

What the “502 Bad Gateway” Error for NGINX Means

Many servers running websites don’t directly serve their content to you when you access a URL. Instead, they use a machine that sits between you and the main server called a “Proxy”. Proxies are used for many reasons – for example, they can improve the security of the main server, they can reduce the load to the server using techniques like caching, and they can even act as load-balancers, distributing traffic to many servers behind it, depending on which one is least busy.

NGINX (pronounced EN-JIN-EX) is a very popular open-source proxy server that’s robust and lightweight. As a result, site administrators use it extensively.

Under normal operating conditions, the following workflow occurs when you request a website on a server sitting behind a proxy:

1. User requests a website on their browser
2. The Proxy server receives the request
3. If necessary, the proxy server asks the actual server for the content
4. The actual server (or the upstream server responds to the proxy with the content)
5. The proxy serves the response it received back to the original user

When everything goes smoothly, you never even know that a proxy served you the request and not the actual server. In the vast majority of situations, the presence of a proxy is beneficial to both you as a user and the website in question.

However, when something goes wrong with step (4) above, you get served a “502 Bad Gateway” Error.

What Causes a Server to Not Respond

If, in step (4), when asked by the proxy server to respond to a request, a server might experience any number of problems that either prevent it from responding and sending an error message to the proxy, which then generates the “502 Bad Gateway” response. It doesn’t mean that the proxy itself is working properly, or that there’s anything wrong with the gateway itself.

The reasons why a server might send a malformed response or no response at all vary widely. Here are some of the things that can go wrong.

Server Overload

An overloaded server has simply too much going on at the moment. Either for reasons of not enough resources or too much traffic, a server can often take too long to respond or generate an error. The proxy has a timeout after which it gives up waiting for a response from the server, and when that happens, it generates the error.

This problem is often seen on shared hosting servers, which are notorious for having low resources, but even well-provisioned VPS servers can suffer from this. You can easily end up using too many resources on a VPS by enabling all the modules and hogging the RAM. For example, on my VPS, I’ve disabled all inbound mail, so I don’t have to worry about things like spam, and I can disable a bunch of modules to keep memory low.

Servers can also be subject to DDoS attacks that rob them of the resources necessary to fulfill ordinary requests. Even with a proxy server in place to shield them, servers can be accessed directly via their IP addresses, and DDoS attacks can occur with frightening efficiency. I’ve written an article on how to stop simple DDoS attacks on your server that you might find useful.

Whatever the reason, an overloaded server behind a proxy gateway like NGINX will result in a 502 Bad Gateway error when it’s unable to respond.

Network Issues

A variety of network issues can block the connection between the gateway and the incoming server. For example, it could be because of a DNS misconfiguration that prevents the proxy server from resolving the IP address of the upstream server. Or it could be as simple as high levels of traffic, which is another variation of the previous problem relating to server load.

Configuration issues can also make the server accidentally block the proxy server because of a misconfigured firewall – for example, only allowing access from certain IP addresses, and not including the one that belongs to the proxy. Hardware failures, SSL issues, and even geographic problems that block connections from certain countries can all contribute to network-related breakdowns between the gateway and the proxy.

Fixing the 502 Bad Gateway Error on NGINX

Here are some things you can try to fix the error.

Restarting the Gateway

The first thing to do is to restart the proxy server and hope the problem solves itself. There are a host of reasons why a proxy server might start behaving strangely, particularly if it hasn’t been restarted for a long time. For example, memory leaks can build up over time, which could consume increasing amounts of memory. Restarting it will fix these problems.

Another reason restarting the proxy might work is if configuration changes haven’t yet been applied. If this is the case, a restart will load the new configuration, possibly solving the problem. A restart can even fix network ports, allowing new, clean connections.

If you’re using a CentOS-based system, the command to restart NGINX is:

sudo systemctl restart nginx

As you can see, when in doubt, restart!

Troubleshooting Network Problems

Network issues can be tricky to debug. The first step is to find out where the problem occurs. To do this, you can use a command like tracert to find out where the packet loss is happening. I’ve written an article on how to use tracert and diagnose request timed-out errors.

Based on the results of the tests, you should review network logs to try and isolate the point at which the failure happened, and then examine the server’s state. If you see a pattern, you might be able to identify the failure point. Resource usage problems are hard to debug, so be prepared for some frustrating experiences.

Conclusion

A 502 Gateway Error on NGINX is easy to understand but hard to debug. So many things can go wrong in the connection between the proxy and upstream server, so isolating the problem can be a nightmare. Your best bet is to restart the proxy server and hope the problem disappears. Otherwise, you’re stuck examining network requests and server logs to find the cause of the problem. There’s no way around it!