Re: Running multiple site from one binary install

[ayleph]

On 5/13/20 6:10 AM, Olivier Mehani wrote:
> Hi again,
> 
> I am currently running my MG instance in the user_dev mode, where all
> the media data is co-located with the binaries and the rest of the
> software install.
> 
> I've never been able to work out a way to separate those. Essentially,
> something like:
> * /opt/mediagoblin <- all the software, lib and code
> * /srv/site <- custom ini files, database, and media
> 
> I've been playing around a lot with system files and config, but I've
> never managed to start MG in a way that's completely independent from
> where the software is installed. I suspect there is something to do with
> the %(here) template variable in a few spots in the config, but have not
> manage to work out how to set it to point somewhere else.
> 
> With this setup, this would also allow me to run multiple sites from a
> single install, say, by dropping a different config and data dirs in
> /srv/site2.
> 
> Any suggestion as to how to do this? Is it doable at all?

That's an interesting idea. I've always just created multiple
virtualenvs to run multiples sites from the same machine.

I do separate my templates and media from the install directory, so I
can definitely help with that part. I'm running a slightly older version
of MG but I think the config options still apply.

All of my virtualenvs are stored under /srv/mediagoblin. I have
locations like /srv/mediagoblin/mediagoblin (main site),
/srv/mediagoblin/mediagoblin-dev (where I track master branch for
testing), /srv/mediagoblin/mediagoblin-py3 (where I was testing some
stuff with python3), etc.

Each of those sites has a mediagoblin.ini config file located within.
For my main site, the config is at
/srv/mediagoblin/mediagoblin/mediagoblin.ini.

The first directive in my mediagoblin.ini file under the [DEFAULT]
section is data_basedir. I set this to a unique location for each site.
For my main site, data_basedir = /var/lib/mediagoblin/mediagoblin. For
my development site, data_basedir = /var/lib/mediagoblin/mediagoblin-dev.

Most of the other directories used by mediagoblin to store files/data
seem to follow under data_basedir, as you can see by inspecting the
mediagoblin.ini file. workbench_path (for media in processing),
local_templates (for customizing pages), queuestore storage (for storing
queued files), and publicstore storage (for storing processed files) are
all relative to the data_basedir entry. So as long as you set a unique
data_basedir location for each site, everything gets put into the
correct place automatically in the default config, and it's all
decoupled from the installation directory. Of course, make sure the user
running the mediagoblin process has write access to these locations.

You also need to configure unique names for the database and messaging
queue. I use postgresql for my MG database. Inside mediagoblin.ini, I
set sql_engine = postgresql://mediagoblin-prod (main site), sql_engine =
postgresql://mediagoblin-dev (development site), etc. I had to manually
instantiate those databases per the documentation. I also set unique
celery brokers, like BROKER_URL =
"amqp://name:pass@localhost:port/mediagoblin-prod" (main site),
BROKER_URL = "amqp://name:pass@localhost:port/mediagoblin-dev"
(development site), etc. That requires setting up per the docs as well.

This is how I keep things separate to run multiple instances on the same
box, and it works fine for me. I would suggest you start here and make
sure you can get this far first. Then you want to go a step or two
beyond what I've done. This is how I imagine you could do it, but I
haven't tried and can't verify it works.

I don't there's any reason you have to put mediagoblin.ini in the
installation folder. It looks like you can specify the location of
mediagoblin.ini within paste.ini. Mine have the default entry config =
%(here)s/mediagoblin_local.ini %(here)s/mediagoblin.ini which tells MG
to look for mediagoblin.ini in the same location as paste.ini. You
should be able to change this to whatever you want (as long as the
process has read access to that location). For example, config =
/var/lib/mediagoblin/mediagoblin-dev/mediagoblin.ini. There are other
entries in there but I'm not sure if they're honored or overwritten by
mediagoblin.ini. My paste.ini has /mgoblin_media =
%(here)s/user_dev/media/public, but there's nothing actually in that
directory on my machine. You would have to experiment with these
directories and see if you needed to change them.

If you get that far, then I think you've removed everything from the
mediagoblin installation directory except for the paste.ini file. But I
think you can even move that! If you're running paster with the systemd
file, you should be able to change the location of the paste.ini file in
the ExecStart command. If you're running via wsgi, then you should be
able to change the location of the paste.ini file in the ini-paste
directive. If you have multiple sites, you'd need separate systemd or
wsgi services for each one to point to the separate paste.ini files.

Theoretically that could decouple everything from the install location.
I honestly don't know if it will work. But it sounds interesting!


-- 
ayleph