6.30 beta 3 - duplications and omissions in download list

[RayMark]

I noticed two strange things that seem to be new to 6.30 beta 3, I did not notice them in 6.30 beta 2 or in previous versions:

1. When adding multi-part hierarchical posts to the download list (without expanding them) - they are added twice - but not together. The second time they are further down the download list in two (or perhaps more) groups, not all together.
I guess, it is possible to expand the posts and carefully add only inside or only outside, but it is not convenient. I like to add after "select all" or some marked range of posts. With 6.30 beta 3 it is no longer a good idea.

2. When adding multi-part hierarchical pots to the download list without expanding them - par2 blocks are not added to the list. In order to download them, I have to find the post, to expand it and to add them again.
It is very inconvenient because again, as in problem 1, I now have to add posts to the download list one-by-one after expanding them instead of adding all at once as was possible before.

BTW, I keep AutoPAR disabled and I want to download all the par2 files together with the posts, not to have to search for them later and perhaps when it is too late.
With today's crazy download speeds downloading par2 blocks is a very small overhead. I can delete them later if they are not needed.
Again, if AutoPAR is disabled, then all par and par2 files must be treated as regular files having no special meaning and should be added to the download list.


The inconvenience caused by the above problems 1 and 2 is so severe that I have no choice but to switch back to beta 2.
It is especially inconvenient when I have already added many posts to the download list the old way - all at once.
Now it is a total mess.

[itimpi]

Regarding 2) - have you checked the AutoPar options to see what you have said should happen to PAR2 files. There is now a sub-option under Disable AutoPar as to whether the PAR2 files should be sent to the Wish List. As I understand it if the option for them to go to the Wish List is not set then they should go to the Download list. If this is not happening then I think this is a bug.

[DThor]

What you're doing in 1 is unclear. How exactly are you adding? From what you've written, I'm guessing you load up spool, select a post, and add. If you're getting double entries from that, and haven't left anything relevant out, then sounds like a corrupt downloads.db3 file...

Trying to reparse what you wrote : are you describing a post that didn't properly 'compact' i.e. it's supposed to be one entry but because it was posted in a non-standard fashion it's in two pieces?

DT

[RayMark]

Regarding 2)

The option to disable AutoPAR is checked. Further sub-options are disabled. However, there is no such a sub-option about the Wish list there.
It is 6.30 beta 3 build 1718. This build has no sub-option about the Wishlist.

I did not even know about the existence of the Wishlist, but now I found it and those par2 files are in the Wishlist.
So it is not so bad, I assume I can move them to the download list from the Wishlist.

However, because AutoPAR is disabled, par2 files should be added to the download list, not to the Wishlist.

Furthermore, I see some rar files also added to the Wishlist. It seems that when par2 files are together with rar part files inside a hierarchical post, the first two rar parts are also added to the Wishlist for some reason.
It might be another bug.


Regarding 1)

I am just adding the whole pot without expanding it. A post contains rar parts and sometimes also par2 block files inside as represented by NewsBin (+P)
I just add the whole +P line, possibly with "Add to Download List - Bypass Filters".
So if I add such +P lines for several posts, let's say posts a,b,c,d,
they are added to the download list also as hierarchical (with +, expandable) a,b,c,d
But further down the download list there is duplicated a range from rar parts from post a, then from post, b, etc, - but not all the rar parts, only a sequence of several, further down the list there is the rest of the rar parts from a, b, etc.
So it seems that together all the rar parts are duplicated in the list eventually.
I cannot be absolutely sure that all the posts added to the download list are duplicated in such a way, some seem not to be. But many are.

But it still works fine with older versions of newsbin 6, the problem with 6.30 beta 3 only, I think.
So if downloads.db3 file is corrupt (and I think it is, because downloads.db3 = the download list, isn't it?) then the corruption is caused by newsbin 6.30 beta 3

[Quade]

I know what's happening and I have to think about it. Disabling autopar puts it into 5.59 mode. In 6.3B4, you'll be able to control whether the pars go to the wish list or the download list. You can't in B3.


I suspect what the problem is, is retries. A retry re-reads the data out of the storage DB and tries to update the files in the download list or add additional files. When the files are compacted together and that works fine. When the files aren't compacted together, they get added again to the download list.

So, I think I know what you're saying. I'm not really clear what to do about it just yet. When a file set is updated, you want the new files but, you don't want to re-download the ones you've already tried (though the dup checker should prevent re-download).

It's an experiment currently. If it doesn't work out, I might have to back out the change.

[RayMark]

I was not using duplicate detection. I know what I want to download (narrowing down with custom filters and search), and in some cases it is faster to download again rather than to get something from one of many off-line hard drives.
Besides, I get visual indication if something is already downloaded, so I can take the decision how to proceed. Quite often I want to download something that is only partially duplicated, so to have some parts missing is not what I want.
Using duplicate detection is a possible workaround but not a real solution and not a workaround I want to use.

I think I can confirm that this duplication problem was not present in beta 2, something was changed in the implementation of beta 3

[Quade]

I think I can confirm that this duplication problem was not present in beta 2, something was changed in the implementation of beta 3

Well, sure. There was no 5.59 mode in B2 so, it makes sense that it's different.

Did you understand my explanation of why it's happening? It's not clear to me that you understand it.

[DaveXVI]

With today's crazy download speeds downloading par2 blocks is a very small overhead. I can delete them later if they are not needed.
Again, if AutoPAR is disabled, then all par and par2 files must be treated as regular files having no special meaning and should be added to the download list.

Could not disagree more. If you want the pars to be downloaded, then just turn Autopar on, and check the box to disable automatic unraring if you don't want that done.

Some posters add nearly a gigabyte of pars to their posts. It is not trivial overhead, especially for those of us who have no choice but to use an ISP that limits bandwidth to a few gig a day. IMO the pars are not part of the post, and should not be added to the download list with Autopar turned off, so I think the change to put them into the Wish List is a huge improvement.

The optimal solution, for both of us, is to fix the switch that says "Disable Automatic Par Unpausing." That way, the pars go into the download list paused, and they stay paused for me if I check that box, but they get downloaded as needed for you. That switch has been there for months, but has never worked, unless it recently got fixed.

[DaveXVI]

I suspect what the problem is, is retries. A retry re-reads the data out of the storage DB and tries to update the files in the download list or add additional files. When the files are compacted together and that works fine. When the files aren't compacted together, they get added again to the download list.

I made a separate post about this a few minutes ago, and if I understand what you are saying, I don't think that's the problem. When I saw some parts that had already been downloaded reappear at the bottom of the download list, I ran Quickpar on the same parts that had been downloaded, and they were perfect. So there was no need to retry them.

It's an experiment currently. If it doesn't work out, I might have to back out the change.

Please don't.

[Quade]

Could not disagree more. If you want the pars to be downloaded, then just turn Autopar on, and check the box to disable automatic unraring if you don't want that done.

If you turn on Autopar, it won't download all the PARS. It'll download only the minimum needed or none if no repair is needed. I'm guessing he wants to download all the pars for some reason. I don't ask why any more.

What's going to go away is the adding of individual files to the download list. They'll be added compacted, download and be removed from the download list.

[RayMark]

The first time they are added to the download list compacted, the 2nd time - sometime later as separate files but not even all together, in separate groups.
And yes, when downloaded they are identical, just duplicated.

BTW, when adding compacted posts, how to move some files inside a compacted post up to the top of the download list? Not out of the compacted post, just up to the top inside the compacted post.
Sometimes I want to but there is no way to do it.

And no, I don't know what a 5.59 mode is, but it seems to duplicate posts in the download list.
They were not duplicated in 5.59, so it is not an exact 5.59 mode

Are you saying, that if when downloading a compacted post the download stalls, it is added again to the download list as a separate file, but then on retry is successfully downloaded after all, and the 2nd copy is downloaded later as well?
If so, some files would be downloaded 2 times, others - 3 times, 4 times and so on. Depending on the number off retries.
But they are always downloaded exactly 2 times, not 1 time, not 3 times.

[RayMark]

UPDATE:

Here is what is happening with beta 4:

The posts in download list are not compacted, but still duplications occur.
It goes like this:
When a file ( a rar part) is downloaded, it is added at the end of the download list again. Sometimes the duplicated files appear several at a time, like 5 downloaded, and after a while the same 5 added to the end of the download list.
So, instead of getting smaller, the list maintains its size and even gets larger until the original files are downloaded and the 2nd copies begin.
That also explains while the 2nd copy is somewhere further down the list and because there are delays and several at a time added, sometimes portions of different posts are mixed together.

So the absence of compacted posts in the download list does not help at all.

So then I closed beta 4 and switched to beta 2.

In beta 2, the same posts (added previously with beta 4) in the download list also appeared twice - first, as compacted, and then immediately afterwards, as separate files.
So, also duplicated, but immediately, not gradually appearing at the end of the download list.
Of course, it is not an error in beta 2 - I added posts to the download list with beta 4.

[Quade]

The posts in download list are not compacted, but still duplications occur.

Yeah, I did nothing about this for B4.

[RayMark]

I just thought this information might be useful to you:

when compacted posts are added by beta 4, then:

- if downloading with beta 4, the duplications appear in groups of several AFTER the corresponding items in the compact post are already downloaded.

- if downloading with beta 2, the duplications appear in the download list as separate files (a complete but not compacted post) immediately after the corresponding compacted post in the download list.
(with beta 2 somehow there are partially 3rd copy - some downloaded 2nd copy files are sometimes incomplete and the 3rd copy is present for those files)

[Quade]

I understand.

The whole issue is that when you add individual items to the download list (5.59 mode), they download and are removed, there's no state so, a re-try on a straggler re-adds them all. I mean the reason it's happening is very clear to me. The solution is to maintain state.

[RayMark]

Beta 2 is quite fine in that respect, no problems with duplications, but the problem with beta 2 is it is sometimes rejecting usernames / passwords.
Perhaps this approach can be used: add files to the download list with beta 2 but download them with beta 4.
It just occurred to me, I did not try it yet.