Discourse proposal status

[Andrew Bernard]

I'm still keen on Discourse for our community and have been giving it 
some attention. People have right;y said the full historical archive is 
important. David Kastrup usefully pointed out all the mbox archives are 
freely available, back to the beginning. Discourse can import mbox 
archives and there is a special import instance with scripts to load them.

So I downloaded the complete set of archives and attempted to load them 
into Discourse. One problem that arose is that the documentation for the 
import script explicitly mentions that you need a system with 8GB of RAM 
(only for the import, not for normal running) and my linux servers are 
only configured with 2GB. Consequently the import fails. I use Vultr and 
Digitalocean and unfortunately upping the specification to 8GB plus is 
too expensive for me in my situation. Servers with extra RAM get 
expensive very quickly at these companies. So this is a problem. I am 
working towards a solution to work around this, perhaps doing it on my 
home server and uploading to a 2GB server later (but there are 
complexities with this).

But, the next current issue is this, and I have come across this before 
when importing mbox archives into GNU Mailman 2 and 3. The mbox format 
is rather loosely defined, and there are lots of small variations. But 
worse, even when you look at our archive set the format seems to vary 
slightly over the years. Some messages don't get unpacked properly due 
to inconsistencies in headers and my import of a sizeable batch showed 
quite a few messages with the initial message fine, but then follow up 
replies just appended in raw mailbox format rather than separate 
messages. I'll have to study this in more depth and write some scripts 
to pre-process the mbox files. Tedious, but I've had to do this in the past.

If I can load the whole history, then people can have a play with it to 
see if they like it. I also think I can get the Discourse instance to be 
a subscriber to the present list so ti would keep up to date. Of course, 
if people replied on the Discourse interface the lists would diverge, 
but this is just for proopf of concept.

As to operational matters, there is cost involved by going outside the 
GNU infrastructure. An adequate server costs around USD$20 per month to 
run. The way I handle this in the communities that I support is to ask 
for donations. Generally if a moderate number of people chip in it can 
be the cost of a cup of coffee a month. It's not excessive.

I think there have been some comments here indicating some 
dissatisfaction with this aging and somewhat limited email 
infrastructure, so I am encouraged to setup my proof of concept to 
demonstrate a modern system. I am also compelled to say that in 
practical terms the email list functionality of Discourse from the end 
user point of view is identical to what we have now with GNU Mailman 2, 
and you'd be hard pressed to find any operational/functional difference. 
So for people who reject web interfaces, in effect very little would change.

Finally, I'd just mention that Discourse of course supports plain text, 
but also HTML, and has full support for Markdown, both from email 
initiated files and web initiated topics. This means you can do nice 
formatting if required, and in particular you can make clear tables. And 
as with most forums now, you can mark code blocks so that they stand out 
clearly.

I'll post regular updates as I make progress towards a full proof of 
concept for people. And of course, if this was Discourse these posts 
would be in a distinct and separate category so as not to be noise in 
the main flow of the stream. :-)

Andrew