Re: [libmicrohttpd] unexpected "Server reached connection limit." failures

[Christian Grothoff]

Hi Frank,

I would first try adding/setting the timeout option for the MHD server.
That might help if otherwise neither your reverse proxy nor MHD close
connections and thus results them in living forever.

Second you might want to check your application logic to make sure that
it doesn't hold/suspend connections indefinitely for some reason.

happy hacking!

Christian

On 04.12.21 01:34, Frank Ch. Eigler wrote:
> Hi -
> 
> I manage a libmicrohttpd based public service, and the software
> that runs it (elfutils debuginfod & its fedora service).  Under
> some usage patterns, it can get into a state where it starts
> rejecting new incoming connections with $SUBJECT stderr message.
> But it's a mystery why.
> 
> The server is throttled to a few dozen tasks via systemd limits.  It
> uses MHD_USE_THREAD_PER_CONNECTION | MHD_USE_INTERNAL_POLLING_THREAD,
> and received front-end reverse-proxy services with keepalive etc.
> That part's fine most of the time.  So we're servicing some dozen
> concurrent connections, no problem.
> 
> But somehow, libmicrohttpd (0.9.73, fedora34) gets into a state where
> the daemon->connections number is weirdly bloated (1020), and so
> internal_add_connection reports $SUBJECT.  But there are still only a
> few dozen keepalive sockets alive (and their threads), and matching
> number of entries on the daemon->connections list, and zero or one
> entry on the daemon->cleanup list.
> 
> I wonder if the thread throttling (via linux cgroups) or other
> transient problem can let that number bloat.  In
> internal_add_connection(), ->connections++ is done, but in case of
> later error, the cleanup: path does not propagate the halfbroken
> connection object to MHD_connection_cleanup_() for a subsequent
> ->connections--.  So that ->connections number could grow and grow,
> and eventually saturate to the ->global_connection_limit.
> 
> Do I read the code right?  Other than a fix to this part of
> libmicrohttpd, do I have an option other than using
> MHD_OPTION_THREAD_POOL_SIZE instead of MHD_USE_THREAD_PER_CONNECTION?
> 
> 
> - FChE
> 
>