beta9 downloads arts out of order

[spotter]

Until this beta (haven't had a chance to try b10 yet), parts downloaded in a "sane" fashion. if you had a main server and a fill server(s), it would queue up X parts (X being threads for main, say 8 for me) and download pieces in order.

The behavior I see in b9 is that it seems to queue up a huge number of parts 30-40-50? (perhaps a sum of all the backup server threads?), and it download the parts from them in a haphazard order (i.e. file 3 has parts going before file 2 is anywhere near done). i.e. of course it tries to download the first part from a file first, but then file 2 or 3 might finish that race first then it gets entirely queued up, so even once file 1's part 1 has finished, it doesn't start downloading for a bit.

guessing this has to do with the new priority system? see some things in b10 log that might indicate it's been seen/fixed, but figured no harm in asking.

[DThor]

Not seen that at all, but it's worthwhile for you to describe your server setup and what the priorities are set to. For example, I have 4 servers, astra us, euro, and two similar block servers, which are set to lowest priority. I flip the astra servers between 1 and 2 depending on time of day (evenings tends to be faster in euro, daytime the reverse), and everything behaves as expected - whichever astra is the higher priority does the lion's share, the other almost never gets touched, the fills only get used when a block is missing from astra.

Atm things are a little confusing since the latest has reversed priority so lower means higher (don't get me started), but basically by high priority I mean 'use me first'. I actually don't see any fundamental difference in behaviour apart from the flexibility to shuffle priorities around.

DT

[spotter]

example

http://imgur.com/41Qo7gx

I have 2 servers. main is "10" (8 threads) (highest currently, I know it switches to 1 in b10) and 9 is the block backup (12 threads). It only downloads from the highest, but as one can see in the image, more than 8 are are "active" (not really, but it impacts the ordering of how they download and it makes reordering of the queue forever to take effect)

[Quade]

There's some look-ahead in the download. Not all of those blocks represent actually downloading activity. Some of it is "getting ready to download".

At this point I'd say "that's the way it is". I have tightened up the lookahead for the next beta.

[spotter]

the problem I have is with obfuscated sets. What I used to do is queue up the set, let it download the first file, move to bottom when done, and rinse and repeat and do a partial extraction and preview in vlc to see if I wanted it. If I did, kept it at the bottom (As was probably rinse and repeating with a few things).

under current system, that's not really practical.

what's the point of look ahead?

[Quade]

Performance. It's like a cache in the processor. Keeps the downloaders busy.

Lets see what happens in the next beta. I cut the lookahead down to 10 chunks.

[spotter]

I guess, I don't quite understand why it helps that much for the majority of uses cases (i.e. I guess I can imagine if you have hundreds of threads going, but for the majority case?) but that could just be me not understanding newsbin's arch.

[spotter]

works better in b11, but still doesn't seem (at least to me) as good as it was before. could be just impression.

[Quade]

That's good to hear.