I don't really see how this is an issue engineers need to be particularly wary of?
Firstly, your typical load balancer doesn't work this way anyway. It will just keep feeding requests to the application hosts on a round robin or random basis. Most don't keep track of how busy each instance is.
Secondly, any decent (HTTP/layer 7) load balancer will notice if an instance is returning exclusively 5xx errors and will stop routing requests to it. Would fail even the most basic of health checks.
Do you have a link to any documentation mentioning this?
As far as I’m aware - using ELBs heavily day to day - it doesn’t maintain a backlog and simply distributes requests on a round robin basis the moment they come in. There is no queue of waiting requests I don’t believe. If the request can’t be fulfilled immediately, it is rejected.
> With Classic Load Balancers, the load balancer node that receives the request selects a registered instance using the round robin routing algorithm for TCP listeners and the least outstanding requests routing algorithm for HTTP and HTTPS listeners.
Firstly, your typical load balancer doesn't work this way anyway. It will just keep feeding requests to the application hosts on a round robin or random basis. Most don't keep track of how busy each instance is.
Secondly, any decent (HTTP/layer 7) load balancer will notice if an instance is returning exclusively 5xx errors and will stop routing requests to it. Would fail even the most basic of health checks.