not-in-memory download list broke

[RayMark]

6.60 beta 3 build 3755

Some weird things happened: even though I added to the download list only 50 items or so, it began showing lines "Display ... pending" - I don't remember now the exact text.
Even if such a line is moved to the top of the download list it is not replaced by the real one, it remains "... pending" and paused. It is not possible to un-pause it.

Also, for some reason some items in the download list were shown multiple times, and the download progress was shown on them all simultaneously and identically, so the items were shown duplicated rather than somehow expanded into separate download threads. Some of those "pending" lines were multiple as well - with the same numbers.

Anyway, I finally removed all the "pending" lines from the download list, and after that everything seemed to work normally.
But now, when I tried to add to the download list just 2 items, I got "Download List Lookahead Reached" errors.
So I removed those two items, the download list is completely empty.
Still, the "Download List Lookahead Reached" errors are continuing to appear at a fast rate. After clearing the log - it is immediately again full of those messages. Even though the download list is completely empty.

Maybe all those strange things are somehow related: many pending lines - multiple (duplicated) lines per item - download list lookahead reached. A chain corruption...

Anyway, I closed and restarted NewsBin and now everything seems to be back to normal.
But the first impression of this new approach not to load the whole download list into memory is not the greatest

UPDATE:

I got now that message again: "Display Update Pending" - but that was only for a short time while retrieving stuff from the search server. I guess, it is normal for this new version.
So, perhaps there was an issue with the search server: if those "pending" lines are added to the download list but nothing is received from the server, they might be left in the download list forever.

Does that also mean that I am allowed to add as many posts from the search results to the download list as I like, and the required information is retrieved from the server gradually, not overloading it?
I am not sure how it was working before: If I add several items to the download list, they used to appear usually with some delay one at a time: perhaps the requests to the server were also sent one-at-a-time from some internal queue, not all at once? Only now those pending items are indicated?
But I had way too many of them when something went corrupt, unless perhaps some irregular requested post expanded into multiple pending lines.

[Quade]

1 - "Display Pending" after download. It's likely a real bug. Just delete them for now.

2 - Doubled up entries. I've not managed to reproduce that in B4. I'm thinking it's related to retries.

3 - Nothing has changed with downloading from the search server and "Display Update Pending" has nothing to do with search at all. By the time the files show up in the download list. It's already finished talking to the search server for these files. The in DB download list needs to read the details of the download out of the downloads.db3. That's what's "Pending".

[RayMark]

2. I haven't seen it again yet.

It was like this:

item 1 #####
item 2
item 3
item 4
item 1 #####
item 2
item 3
item 4
item 1 #####
item 2
....

So when item 1 is at the top and is being downloaded, the progress indicator (#####) is moving in sync on item 1 lower as well.
And if I run search in the download list for "item 1" and move everything selected to the top, I get:

item 1 #####
item 1 #####
....
item 1 #####
item 2
item 3
item 4
item 2
item 3
item 4

Also "pending" lines were multiple:

pending 130
pending 131
pending 132
pending 133
pending 130
pending 131
pending 132
pending 133
pending 130
.....

After moving all copies of item 1 to the top, I counted that the same line was 11 times.
With item 2 - the same, also 11 times.
Possibly - more, if, for example, pending 131 represents also item 2.
Of course, none of the pending lines had progress indicator actually showing progress, they all were shown as paused.

Actually, I am beginning to think that the whole problem was with one particular post rather than with different ones.
If only I could remember which one. Because I now remember thinking it looked like an expanded list but without the main line into which to collapse:

item 1 = some post.part01.rar
item 2 = some post.part02.rar
etc.

I now remember that some other posts were still not multiple. Therefore it probably was related to an irregular post or to some glitch in adding it.
Possibly those multiple pending lines were for the same post too.
If it happens again, I will try to reproduce it with the same post, or at least I will record what post it is.

[RayMark]

IMPORTANT UPDATE:

I think I reproduced it. It can be reproduced with any larger post, I did it with the first one I tried: I ran search for "web-dl" and took the latest larger post (4.31 GB, ends with -BOB)
But to do so, I had to behave in a rather weird way:

1. Expand the post in the search results (or in a group window, probably)
2. Let's say the post contains 39 rar parts + pars, and it does
3. Select, let's say 10 first rar parts and add them to the download list.
4. Then collapse the post, or just select the main line of the same post (so, the whole post) and add it to the download list too.
So now we have 10 first parts added separately and the whole post added as well.
5. Observe what is happening in the download list

So I got multiple copies of part1.rar being downloaded in sync, etc., but afterwards I am stuck with 9 pending lines, all of them are the same:
Display Update Pending 2608
Display Update Pending 2608
.....
I had to remove those pending lines eventually.

Notice that part01.rar was added twice: once separately, once as a part of the whole post.
But it showed up in the download list multiple times, way more than twice.

[RayMark]

IMPORTANT UPDATE 2

Turns out, I probably did not reproduce the previous problem but found a bug for sure:
You don't have to add the whole post after a partial post.
Adding, let's say, only the first 3 rar parts causes the whole post to be downloaded instead of the 3 intended rar parts.
And when downloading it, the post is shown in the download list three times, the progress indicator moves in sync.

Switching off the automatic unrar does not help. Still, the whole post is downloaded instead of the selected parts.
I also tried to download just one rar part, and not the first one, but part07.rar (let's say I need only this part) - still, the whole post was downloaded instead, even when auto unrar switched off.
Furthermore, one pending line was left after the post was downloaded, I had to delete it manually.
That is a serious bug, I sometimes need separate rar parts only. Previously it was possible to download selected parts only.

However, it is perhaps a related but a different issue, my initial problem clearly had multiple rar parts in the download list, with different numbers, not the main post duplicated multiple times as it happens here.

[Quade]

Adding, let's say, only the first 3 rar parts causes the whole post to be downloaded instead of the 3 intended rar parts.

You can't just add 3 rars. It's in the release notes. it's currently all or nothing. You are adding the same set of files multiple times though from search. I'll check that out.

Thanks.

[RayMark]

Oh, so it is a big step backwards!

Previously it was possible to download separately any rar parts you wanted.
I was able to expand the post and then to add whatever part I needed from it.
Especially if autopar is switched off, you should be able to do so - any grouping under the umbrella of a single set of files and handling the set as a whole only is artificial when auto-par is switched off: when the files are posted, they are posted in a sequence. The interpretation of that sequence should be left to me if autopar is off.

Another scenario: I may want to download only the first rar part or a few first rar parts to see what is inside and perhaps to unrar with -kb option if sample is not present.
As you know, currently more and more posts tend to have meaningless or distorted names to survive longer. And who knows in what closed forum you can find real names.
It is important to be able to download only the beginning of a post more than ever.
Or just the last rar in the post - as the last rar may show the whole directory structure of the post.
It was always possible to do so with previous versions.

So now, I cannot download only part07.rar at all, and in order to download only part01.rar or the range part01.rar-part05.rar I need to observe the progress of the post, to pause at the required time and later to delete the post manually, If I understand correctly.

[Quade]

Oh, so it is a big step backwards!

Dude, it's in the release notes as a "not working yet". You're jumping to some wild conclusions.

[RayMark]

sorry, I thought it was done by design or by cutting corners but permanently...
I admit I never read the release notes