Re: [Mldonkey-users] upload queues, priorities, and .mli files

[Jason Baker]

I would have replied sooner, but I've been using mldonkey as a
bittorrent client, and only recently had a chance to look at the
effects my local changes.

address@hidden writes:

> Hi Jason,
> 
> On Wednesday, July 6, 2005, 5:22:44 PM you wrote:
> 
> > I noticed the following in docs/developers/mldonkey-design.tex:
> 
> >>  The long upload queue sorts the clients requesting to enter the upload
> >> queue in a fifo order. This means that the oldest clients are always in the
> >> top of the queue (ie they have a lot of upload slots in other clients, even
> >> if they cannot use them at full speed), whereas new clients (for example
> >> newbies) have to wait for days to see their downloads start. Clearly,
> >> to get as many users as possible, it is important for newbies that they
> >> immediatly download something from the network, even if it is not a lot.
> >> Long upload-queues are good for long term users, those who are the most 
> >> used
> >> to waiting long for files...
> 
> > Is this still the current feeling on upload queues?  It is in tension
> > with the third argument against credit systems in the same document.
> > Someone waiting for a rare and interesting file may end up competing
> > against an ever-changing group of people waiting for more popular and
> > easily downloaded files.
> 
> There is an interesting article about this in the wiki:
> 
> http://mldonkey.berlios.de/modules.php?name=Wiki&pagename=Queues
...
> I'd like to emphasize that neither of the three variants have an
> impact on the sharing of rare files. The priority system is something
> rather independent, which can be implemented in either "queue" or
> "pool".


You seem to be talking about far bigger changes than I actually
implemented.  I found that in 2.5.30.16, the existing upload "priority"
system was not being enforced, so I wanted to get it working.  Along
the way, I noticed a couple peculiarities in the way that we select a
downloader (to upload to) from the pool of connected clients
   1. Enforcing priorities means iterating over the pool of connected
      clients until we find one that can be admitted.  Given the way
      that Intmap works, this means that we choose the admissable
      client with the highest ID (which is the client that we
      encountered for the first time most recently).
   2. We use Intmaps (which are functional binary trees) when either a
      simple list or a tree with destructive update would be more
      appropriate.
So, I got priorities working, and started maintaining a queue of all
pending requests.  With my changes, we still select a downloader from
the pool of connected clients, but we choose the one that has been
waiting the longest, rather than the one that we first saw most
recently.  It doesn't change mldonkey's behavior in a major way, but
avoids some potential unfairness in the priority mechanism.

I know that things like queues vs. pools, and priority enforcement do
not effect the sharing of rare files.  I wanted to use priorities to
group files by their popularity, and thus achieve similar upload rates
between all files.

That did not quite work for me.  All the files I share are somewhat
obscure (I could not find them on large public bittorrent trackers),
but the relative frequency of requests can change drastically from one
day to the next.  So, it doesn't seem possible to statically assign
priorities that give the behavior I want.

My actual goal is to schedule uploads so that each file, rather than
each client, is treated fairly.  Maybe the best way to do that is to
do just that:
   1. Each file has its own request queue, and we actually deal with
      requests for a file in a FIFO basis.  To do so, we would have to
      establish a connection with the first requester in line.  Surely
      the ed2k protocol supports this, but how would it work with
      lowID clients?
   2. We service requests for files of the same priority on a
      round-robin basis.
   3. I'm not sure about the current priority scheme as a whole.
      Right now, priorities are used for admission control, but not
      for link scheduling.  It might be nice to use priorities for
      both.  (For instance, one or two low priority connections can
      get a guaranteed 10% of your upload bandwidth, but soak up extra
      bandwidth when high priority connection are slow.)

Jason