|
| From: | Erik Smith |
| Subject: | Re: [libmicrohttpd] epoll and memory leaks |
| Date: | Fri, 10 Dec 2021 06:29:19 -0800 |
Hi Erik,
It's hard to move forward without knowing exact MHD version used.
Please share information about your MHD version.
I suspect that your "Reply-To" header may confuse mailing list system.
Do not use "Reply-To" headers with mailing list.
--
Evgeny
-------- Original Message --------
From: Erik Smith <cruisercoder@gmail.com>
Sent: Friday, December 10, 2021, 03:29 UTC+3
Subject: [libmicrohttpd] epoll and memory leaks
> I've been able to reproduce this with a modified program from the
> examples to show the memory consumption I'm seeing. I'm using jemalloc
> to capture memory consumption and here's what it looks like for the
> program below when repeatedly hitting the endpoint:
>
> allocated: 120072, active: 163840, resident: 9150464
> allocated: 150536, active: 196608, resident: 9228288
> allocated: 181000, active: 229376, resident: 9306112
> allocated: 211464, active: 262144, resident: 9383936
> allocated: 211464, active: 262144, resident: 9383936
> allocated: 241928, active: 294912, resident: 9461760
> allocated: 272392, active: 327680, resident: 9539584
> allocated: 272392, active: 327680, resident: 9539584
> allocated: 302856, active: 360448, resident: 9617408
>
> The delay in the handler and the use of ASAN tend to inflate the memory
> growth. The key factor here seems to be the use of the thread poll
> with either poll or epoll. Without the thread pool, there is no memory
> growth at all. The growth happens on low connection rates (manual
> refreshing in the browser). I haven't yet tried compiling MHD with ASAN.
>
> I'm also not getting responses to my threads in email for some reason
> but I'm checking the archive.
>
> #include <cstring>
> #include <iostream>
> #include <jemalloc/jemalloc.h>
> #include <microhttpd.h>
> #include <sstream>
> #include <thread>
>
> static enum MHD_Result handler(void *, struct MHD_Connection *connection,
> const char *url, const char *method,
> const char *, const char *, size_t *,
> void **ptr) {
> static int aptr;
>
> if (&aptr != *ptr) {
> *ptr = &aptr;
> return MHD_YES;
> }
> *ptr = NULL;
>
> std::this_thread::sleep_for(std::chrono::milliseconds(40));
>
> size_t sz = sizeof(size_t);
> uint64_t epoch = 1;
> mallctl("thread.tcache.flush", NULL, NULL, NULL, 0);
> mallctl("epoch", &epoch, &sz, &epoch, sz);
>
> std::size_t allocated, active, metadata, resident, mapped;
> mallctl("stats.allocated", &allocated, &sz, NULL, 0);
> mallctl("stats.active", &active, &sz, NULL, 0);
> mallctl("stats.resident", &resident, &sz, NULL, 0);
>
> std::stringstream s;
> s << "allocated: " << allocated << ", active: " << active
> << ", resident: " << resident << "\n";
> auto msg = s.str();
>
> std::cout << msg;
>
> struct MHD_Response *response = MHD_create_response_from_buffer(
> msg.size(), msg.data(), MHD_RESPMEM_MUST_COPY);
> MHD_Result ret = MHD_queue_response(connection, MHD_HTTP_OK, response);
> MHD_destroy_response(response);
> return ret;
> }
>
> int main(int argc, char *argv[]) {
> struct MHD_Daemon *d;
>
> int port = argc > 1 ? atoi(argv[1]) : 10000;
>
> // epoll mode with thread pool
> unsigned int concurrency = std::thread::hardware_concurrency();
> std::cout << "concurrency: " << concurrency << "\n";
>
> d = MHD_start_daemon(MHD_USE_EPOLL_INTERNAL_THREAD |
> MHD_USE_ERROR_LOG, port,
> NULL, NULL, handler, NULL,
> MHD_OPTION_CONNECTION_TIMEOUT,
> (unsigned int)120, MHD_OPTION_STRICT_FOR_CLIENT,
> (int)1,
> MHD_OPTION_THREAD_POOL_SIZE, concurrency, NULL,
> MHD_OPTION_END);
>
> if (d == NULL)
> return 1;
> std::cout << "listening on port: " << port << "\n";
> std::cout << "hit key to stop"
> << "\n";
>
> // type a key to end
> (void)getc(stdin);
> MHD_stop_daemon(d);
> return 0;
> }
>
>
> Hi Erik,
> Which MHD version are you using?
> Some problems with externally added connections with epoll mode were
> fixed in v0.9.72.
> If you have any blocking calls, make sure that you use connection
> suspend/resume. Alternatively, you can you use thread-per-connection
> mode, this is less efficient, but simpler to implement.
> epoll mode does not have special memory allocation, connections are
> processed in the same way, like in other modes. MHD typically does
> not allocate memory during connection processing, except when new
> connection is started.
> Do you use postprosessor or authentication functions? MHD has some
> memory allocs in these functions.
> The issue is not connected with quoted comment definitely. It is
> just a bad wording. Actually nothing is leaked, but may be locked
> until end of sending of response. Moreover, MHD does not use memory
> pool in the way where such lock is possible. Memory pool is reset
> after each request-reply cycle. Memory pool size for each connection
> is fixed and cannot grow.
> A few suggestions:
> * make sure that you are using the latest MHD version (0.9.73 at the
> moment), * check whether you destroy responses and free all
> resources connected to responses, * if you are testing your code
> with ASAN, make sure that leak detector is enabled. You can build
> static MHD lib with ASAN and link it with our application compiled
> with ASAN,
> * use Valgrind or simpler tools like memstat or memprof.
> --
> Wishes,
> Evgeny
> On 22.11.2021 22:56, Erik Smith wrote:
> /* Reallocate a block of memory obtained from the pool.
> * This is particularly efficient when growing or
> * shrinking the block that was last (re)allocated.
> * If the given block is not the most recently
> * (re)allocated block, the memory of the previous
> * allocation may be leaked until the pool is
> * destroyed or reset. */
> Can anyone confirm whether this might be related?
> ASAN does not seem to detect any issues in our code presently (not
> sure about MHD)
>
> We have started to experiment with running MHD with epoll + thread
> pool as we do the FD limit in certain situations. We understand
> that there are caveats to this given that we have some
> blocking database calls. This seems to get us past the FD limit
> errors and the performance is similar. However, we are running
> into growing memory consumption in our server over time
> running epoll+threads that require a restart frequently. This does
> not seem to occur with just epoll (without the thread pool). We
> are running jemalloc, but it does not seem to be related to the leak
> when it is disabled. There is the following comment in the MHD code
> for the MHD_pool_reallocate function that might be connected to this
> issue:
>
| [Prev in Thread] | Current Thread | [Next in Thread] |