newsbin.com

I am sorry, but I cannot say when these problems started, but I am currently running 6.30b6 Build: 1824.

If I queue up any number of files (always >500 megs) and leave the system to do the downloading, I'll come back to many of the downloads being duplicated in the download list. They will not only be duplicated once, but sometimes 100s of times. For example: I queue files 1-10 totaling about 15 gig of data and listed in order of when added (in other words, no sorting at all). I go away for 20 minutes. When I come back, anywhere from 1-3 of the files are done (totally finished and proper), and then the rest show as having 4 duplicated 50 times, 5 duplicated about 40 times, etc. If I check the download directory it shows that I downloaded the data files and created many duplicate par files named (0001), (0002), etc. I went to bed the other night with 2 files queued. Came back to a queue of 400 items and about 800 extra pars. All of the pars are duplicate files, just named with the incrementing number.

The second problem I have is that when the system starts doing the phantom duplication, it starts failing to finish all of the downloads and I get a lot of .nb2 files in the download directory. The files appear to be correct and I can use quickpar to fix the files by manually adding the .nb2 files.

The last problem I'm having is that if I close the program (after getting 100s of duplicates in the list) and reopen it, the downloads don't pick up where they left off. The list is now showing what it should show (instead of 100s of duplicates in the list, it shows the remaining files that would have been downloading had it not gone off into the par2 duplication problem). But the files start from scratch. I don't end up with (0001) files or anything like that. It's as though the system overwrites the ones in the download directory.

Any ideas?

I've deleted all the download .db3 files and such. I even try to do a single download and it started duplicating after a few minutes.

I have my header download set to 15 minutes and have tried turning it off with no change in behavior.

The system that it's running on is a 64 bit machine with 32 gigs of ram and a raid of SSDs.

When you have bizarre behaviour like that, it's time to simplify. First and foremost, lose the ssd RAID - just do it to the local disk. If you say this happens with as little as 500M, then don't leave it, watch it. Try to track down what is happening. I look at a file in the queue, it downloads, I see it show up in download. Unless you're talking about very specific, unusually posted files, I don't get this duplication thing at all. It can't materialize out of nowhere. Anything odd about the posts? Anything odd about your setup, like doing this over the network?

And just double checking - when you delete the downloads.db3 - you're doing this when Newsbin isn't running, right?

DT

I have duplicated the problem on a second machine (a laptop running a single hard drive) and also in a virtual machine while trying to figure out the issue.

None of the posts are strange. They're what I would call standard fare for the a.b.hdtv group. I've tried many different ones with the same end result.

I've had it download a single file just fine. I've had it duplicate a single file (the same one as a test on another machine).

I've watched it for hours with it duplicating in front of me. I only mentioned walking away because that was on my mind at the time of writing the post.

I can't for the life of me figure out the pattern of behavior.

I do close the application and make sure it's not in the task manager when I delete the db3 files.

Someone else mentioned in another post that they had posts duplicate, but I wasn't sure if it was enough to justify putting my problem as a reply to his message.

Also, I wanted to make sure that Quade and anyone else reading understands that I'm not expecting a solution, but hoping for one.

I've used newsbin for over 10 years and this is the second time that I've run across a problem that I couldn't solve alone.

I'm just hoping that someone else has the same behavior and may have tried something else.

I don't know of anyone else reporting something like this. Retries are the only thing that might cause this but, retries won't insert duplicates. It's like duplicate detection in broken for you somehow.

Weird. I guess when I said watch it, I meant I'm curious precisely how the mechanism works. When there's an entry, it downloads, you see the progress, it clears out and there's the file. That's everyone else. For you... Does the entry re-download the file in the queue? It finishes, then shows up again? It only shows up once, then when it completes suddenly two files pop into existence? Are you seeing multiple nb2 files there when it's downloading? Quade is inferring you might be having retry issues, do you see a retry showing up in the status column when this happens? Also, this happens with only specific posts, or everything? Is it repeatable with the same post? You know, all the fun debugging stuff...

DT

well, I see the same thing or did. I had so many crashes and downloading of duplicate file filling up the server that I rolled back to 6.2 (which does not exhibit this behavior).

Do you have autopar disabled?

For myself (Not sure who you are asking) Yes. That was one of the first things I did since most (the vast majority) of the issues with file being re-added to the download list was around PAR files. When I last used 6.3 ( couple of days ago) the PARS were the ones it downloaded 12 copies of (and was still going until I deleted them from the list, which caused a crash). So, yes, AutoPAR was disabled.

I can re-install 6.3 and capture any debug information you want (and get screen-shots). It always reproduces within a given session if downloading more than a couple of dozen files/segments.

To be clear there are a number of symptoms (which may or may not have a common cause).
1) many sections of a download are re-added to the end of the queue (due to errors of some sort--unclear exactly what). They are re-downloaded as a duplicated file (the renumbering scheme kicks in).
2) deleting these files from the list once they are re-added to the end will cause a crash.
3) Many parts of the download sets are incomplete or corrupted. PARS (if they manage to download) can generally fix them. This is NOT a server issue since the exact same files/sets downloaded with 6.2 will succeed with no errors.
4) Once restarted after a crash everything (downloaded or not) from the previous session is now back in the wish list. Putting them back into the download queue will eventually recover as it figures out what parts were already downloaded.

The files are being re-added BECAUSE you have autopar disabled. It's a bug but, enabling autopar will make the re-adding thing go away.

I might have to do a 6.31 for people who disable autopar. The issue is that the retry is re-adding instead of updating because I put the smallest par file at the top of the list in the compacted set when autopar is disabled instead of below, like it is when Autopar is enabled. This confuses the retry.

Good enough, I will reinstall 6.3 and try this after this set is done.
Anything left (in terms of odd behaviors) after autopar is restored I'll document separately.

OK, I can verify that turning on AutoPAR appears to have removed almost all of the weirdness.
It is a little strange that things persist in the list as completed now when before they would remove themselves. That I can live with.
Thanks.