Re: Livelihood statistics of the SKS keyserver network

[Gunnar Wolf]

Andrew Gallagher dijo [Thu, May 13, 2021 at 12:34:20PM +0100]:
> > The first thing to understand what I'm plotting are the many graphs
> > that are _not_ present in the summary page: Scroll down to the table
> > that has many dates. Clicking on any such date will show you a walk of
> > gossiping servers, starting from basically a random server in the
> > network (I pick up whatever answers to pool.sks-keyservers.net -
> > That's why you will see many entries with zero or very few nodes: it
> > means I was unlucky with the resolution for a given day). The (quite
> > ugly and ad-hoc) source for those graphs is at:
> > 
> >      http://sks-status.gwolf.org/walk_sks.rb
> 
> It might be worth exploring whether starting with more than one initial node
> would help your luck - usually ha.pool.sks-keyservers.net, na.pool, eu.pool
> resolve to different servers and at least one of those should be in good
> shape at any given time.

I guess it would! It is... an improvement I'll save for later,
though. I don't want to put too much (more) work into it, as I only
get bad connections' "noise" in about 1/10 of the tries. But yes, that
would surely help!

> > Do note that at the beginning I sampled the network much more often
> > (hourly). I decided this was too much, and since June 2019, am now
> > walking the network only every four hours. I do hope nobody sees this
> > as excessive!
> 
> Given that fastly and happy-eyeballs spider the network several times a
> minute, I can't imagine anyone would have a problem with your hourly
> extravagance. :-D

That's very good to know :-)

> > This shows the very large drop the SKS network had in mid-2019, as
> > well as its behavior since then. I am happy, even hopeful, to note
> > that it seems the network hit reliability minimums between October
> > 2020 and February 2021, but it seems there is a slight trend for
> > improvement, at least back to the late-2019 levels.
> 
> I think we can attribute some of that to Casey's work on Hockeypuck which
> has downgraded the poison key issue from a killer to an annoyance.

Great! Well, it's very good to have a data point for this.

> > Please do tell me if this data sounds interesting, and if you can
> > thing of anything to improve on what I'm doing. Of course, I cannot
> > apply any changes  to already-collected data, but there are surely
> > many other things that can be considered.
> 
> This is very interesting work, thank you. I find the connectivity graphs
> hard to make sense of though as the nodes tend to be drawn on top of each
> other. A quick google hasn't thrown up any answers but I'm sure there's a
> way to address this by imposing some minimum separation between nodes. The
> graph browser we include in our software in $WORK has the ability to do
> this, but I need to check the details before recommending it, as I'm not
> sure if it has a noninteractive mode.

Right. Well, I'm now also producing links to the source Graphviz
files, it might be easier for you to produce something easier to
view. 

> Some simple changes that I would otherwise suggest:
> 
> 1. produce a connectivity graph with only working nodes

Added, as "Success" graph.

> 2. ignore localhost and private IPs, they will never work :-)

Of course. But still, it documents a given server works in a clustered
way. Of course they will fail, but who cares? ;-

> 3. should it not default to port 11371 and fall back to 80?
> 4. don't plot *.pool.sks-keyservers.net in the graph

Right... I could add this bit later on, but then again, it does not
clutter the graph much. AFAICT, It is only a single, starting node,
and no nodes ever point back to it, right?

Erm, no, _not_ matching reality. There are several servers as a
destination...

> Some more complex issues:
> 
> It seems that your Hockeypuck status parsing is broken, as a significant
> number of Hockeypuck nodes are listed consistently as Nil. It's probably due
> to assumptions you're making about the DOM in Hpricot. Thankfully,
> Hockeypuck emits a JSON stats page at 'pks/lookup?op=stats&options=mr' . If
> you get Nil from Hpricot, maybe you could fall back to the JSON? Beware that
> if you request the JSON page from SKS it will give you an HTML stats page
> regardless.
>
> Also, the peers of some Hockeypuck nodes are throwing InvalidURIError - I
> suspect this is because of an inconsistency between SKS and Hockeypuck in
> how peers are reported - SKS says "host port", while Hockeypuck by default
> uses "host:port", but beware that many Hockeypuck operators have changed
> this back to SKS format so that the pool spider can parse the peer tree.
>
> Unfortunately, it's not as simple as splitting on /(\s+|:)/ as this will
> bork on ipv6 addresses. I'd suggest something like the following:
> 
>   host, port = server.split(/\s+/)
>   if port == ''
>     host, port = host.split(/:/)
>   end

Ooohhhh! Important and interesting. I have to look into how to deal
with it, specially if the number of Hockeypuck servers grows! I cannot
put more time into this today, but will look back into it.

signature.asc
Description: PGP signature