Re: celery connection problems to RabbitMQ

[Fernando Gutierrez]

Hi Ben

Sorry I think I didn't explain clearly. I only fixed the connection reset exceptions in celery but the bug with media changed to failed state is not fixed.

I will continue debugging but it may take some time. I don't know why celery thinks a completed task needs to be run again.

In the meantime I will submit a patch for the systemd file, the BROKER_HEARTBEAK issue and also a fix for the audio spectrogram code as I mentioned in the IRC channel.

I have a couple of questions:

1) I'm not familiar with the development process, I already created an account in savannah.gnu.org but I don't see how to submit a patch for review.

2) For the spectrogram I used the PySoundFile package. What file do I need to modify so it gets pulled during setup?, in my setup I manually called ./bin/pip install PySoundFile

Thanks

Fernando

On Sun, May 10, 2020 at 6:39 AM Ben Sturmfels <address@hidden> wrote:

Hi Fernando,

On Sun, 10 May 2020, Fernando Gutierrez wrote:

> I recently asked in the IRC channel about RabbitMQ connection reset
> errors in celeryd logs.
>
> I think there are two issues:
>
> 1) The example systemd file (mediagoblin-celeryd.service) from
> https://mediagoblin.readthedocs.io/en/stable/siteadmin/deploying.html
> does not specify that celeryd must be started after RabbitMQ, so it is
> sometimes started before and fails because RabbitMQ is not running
> yet.
>
> 2) In mediagoblin/mediagoblin/init/celery/__init__.py, it sets
> celery_settings['BROKER_HEARTBEAT'] = 1. In slower systems or
> under heavy load if the worker is too slow to respond in < 1 second it
> will miss the heartbeat and after a few missed heartbeats the
> connection is considered dead and reset.
> I'm not sure what is the purpose of changing BROKER_HEARTBEAT to
> 1 but the celery docs recommend not using such a small value. In my
> install I changed it to 20 and I no longer see any connection
> problems.
>
> Are you willing to accept a patch for
> mediagoblin/docs/source/siteadmin/deployment.rst and
> mediagoblin/mediagoblin/init/celery/__init__.py to fix those two
> problems?

Thank you very much for diving in and investigating the issue. We'd be
happy to take a patch on this. If you can add a comment to explain the
new BROKER_HEARTBEAT value in the code, that would be great.

I wonder if there there might still be a problem lurking here though,
even if your system is now working properly Not being able to connect to
RabbitMQ or an unresponsive celery worker probably shouldn't change
existing processed media items to failed.

Thanks for your work on this!

Regards,
Ben