newsbin.com

6.61R

Having repeatable issues where a file in a Watch_list begins to download (Auto Mode), but only half the file parts are initially available. NB times out on retries and does not wait for the rest of the files and puts the post in the Failed List.

Is there a setting I am missing or is this potentially a bug?

Thanks

Had another report of this over email. I'm going to have to make a test setup and see what happens.

How often are you updating groups/letting search poll (depending on your mode)?

A partial file is getting into the download list because it's either not complete in search yet or not complete in headers (whatever mode you're using). It can't become complete until the next time you download headers/poll the search server. If the timing is such that it runs out of retries before the next poll, it'll fail out. As an experiment, you might set the retries to 100 and see what happens. That means basically it'll never give up.

To me the question is whether it becomes complete with the next update.

I posted a question yesterday and was directed to this thread as we may be having the same issue but I'm not sure. Files downloaded from my watch list show fully downloaded, no incomplete files. Even the par files show downloaded but the file won't unrar because it says multiple blocks are missing. It's like the par file is corrupt or a glitch is showing the files are downloaded when they're not.

Quade,

Currently NB is set to download Headers updates every 30 minutes, so Watch_Lists are updated accordingly.

I do not use Sickbeard and I download file posts from Headers 99% of the time. Sometimes a .NZB from Posts or one I have created from a service.

My Watch_Lists are set up to list Posts based on RE Strings and/or Posters. Some W/Ls are set to Auto D/L.

Will adjust Retries to 100 and advise.

Thanks

It's like the par file is corrupt or a glitch is showing the files are downloaded when they're not.

If you use QuickPAR on the PAR files how do the block counts look?

Will adjust Retries and advise.

I'm going to setup a repeatable test jig so I can run multiple sets of incomplete files through autodownload. See what happens.

Quickpar will report missing Blocks.

Many times, I will do a Ctrl-Y on the Post entry in the Failed List which will kick it back to the Download List. This does not work many times and the Post entry returns to the Failed List because apparently the needed part just aren't available.

Sometimes, I will delete the incomplete Post from the D/L List while leaving the RAR parts in my D/L Folder, and then do a Ctrl-Y on the Post. This does not resolve the issue more often than not, and Blocks are still missing.

This may also have to do with certain Posters and their locale and/or news server. If I wait until the next morning, I can usually get the same title d/l from another poster which is complete. ?????????????????? NB may be doing exactly what it needs to do.

Thanks

Hey Thanks for the help. This is what I get when one of the files gets stuck. It doesn't do this on every file by the way.

This is what Newsbin shows. Full file size. all files show completed


Right click on the file and go to properties and it shows this


Run Quickpar and it shows this


oops I was trying to show pics of what I was getting. How do I get the links to show or can I send them to you?

Edit: I deleted the image links. They were too revealing for this forum.

This is what Newsbin shows. Full file size. all files show completed

Just because Newsbin says it's complete, that doesn't mean your server can deliver all the parts of the file. If you see 430 errors in the logging tab then not all the file chunks are available on the news server. If you hit the plus icon on the download list items, you can watch the download. red means "not on server" black means "data is missing from the download".

On Usenet, the older the file the less likely you'll be able to download it.

The PAR counts listed there tell you how much of the files downloaded properly. You got 1136 blocks out of 1599. That means many parts of the file were missing. Probably 430's from the server. Many people run more than one server so, they can get missing parts from their backup servers.

The properties are telling you pretty much the same thing. You need to try to download this and keep an eye on the download colors and for 430 errors. If you're getting RED and 430 errors then the problem is really the server not delivering parts of the file.

You symptoms aren't the same as TBlacks after all. You got part of every file. Just every file was missing a block or two.

I can pretty much predict how this download went down.

1 - Every file had 430 errors.

2 - Newsbin started to download the PAR files.

3 - When the last PAR file downloaded, Newsbin assembled what it had and tried to repair.

4 - You just don't have enough repair blocks to fix the files.

I'd watch a download that has these problems and look for the problems I mentioned.

I see what you are saying but the file in the example is less than a day old so I wouldn't think it would be missing that many blocks. When the files have been doing this, I can usually remove them from the download list, download again by bypassing filters and they usually download again and have enough blocks for repair. By the way I'm not getting the 430 error.... the only error I'm getting in the log is "HIGH DecodeYenc - Bad YEnc Format" which I don't know if that has anything to do with it. I guess what is mostly confusing me is that normally when a file doesn't fully download, it shows downloading under progress (instead of downloaded) and usually will have little missing slivers from the colored bar showing that some data is missing from that particular file. A retry will try to download that missing sliver again. But in this particular case, because all the files show downloaded instead of downloading, a manual retry doesn't do anything and no matter how long the file sits, it stays at the same progress until I remove it and try again. Everything was working fine until a couple of days ago which has me wondering what could be the problem.

By the way I'm not getting the 430 error.... the only error I'm getting in the log is "HIGH DecodeYenc - Bad YEnc Format" which I don't know if that has anything to do with it.

It's a sign that something is wrong with the data. Wonder if it could be your security software stepping on the download? That's a relatively common problem. What news server are you using?

I guess what is mostly confusing me is that normally when a file doesn't fully download, it shows downloading under progress (instead of downloaded) and usually will have little missing slivers from the colored bar showing that some data is missing from that particular file. A retry will try to download that missing sliver again. But in this particular case, because all the files show downloaded instead of downloading, a manual retry doesn't do anything and no matter how long the file sits, it stays at the same progress until I remove it and try again. Everything was working fine until a couple of days ago which has me wondering what could be the problem.

In light of your bad data comment, I think the answer is Newsbin thinks the file is in better shape than it really is and it's assembling the files and attempting repair but not enough good data is there. Normally I think a "Bad Yenc" format is just a status message that can be ignored but in your case it seems like it really means bad data. In your shoes, I'd probably completely uninstall any non-MS virus scanners and firewalls, reboot. Try a couple test downloads then re-install or switch security software if the problem goes away.

That's kinda a big step. You could also try to make the scanners ignore Newsbin's connections to the news server and see if that improves things.

Quade,

Had another strange event last night.

Watch_list post showed complete with 13 Rars.

The d/l started normally due to Auto Mode (I actually saw the D/L List populate during a Header update). Only 5 parts downloaded and then kicked into Failed List. Could not get the post out of the Failed List back into the Download List with several Ctrl-Ys....it kept bouncing back to Failed List.

Clicked on the + sign to see if parts were missing. All seemed to be there (Green Dots) so manually downloaded the remaining parts and they were there. NB auto repaired and assembled the RAR.

This all happened with me in front of me in the space of about 5 minutes or less.

Is this a result of changing some code in 6.61 which identifies dups or file identifications?????

Never seem to have these issue before

Thanks

Is this a result of changing some code in 6.61 which identifies dups or file identifications?????

No clue. Did it say why it was rejected?

No....Error Column is Blank

https://onedrive.live.com/redir?resid=5 ... lder%2cjpg

TBlack wrote:This may also have to do with certain Posters and their locale and/or news server. If I wait until the next morning, I can usually get the same title d/l from another poster which is complete. ?????????????????? NB may be doing exactly what it needs to do.

The NZB I sent you this morning that would not complete from last night was OK mid morning from another Poster with absolutely no intervention from me. Maybe we need to trust NB more

I am still scratching my head about NB's Test Download functions saying the Post is there 100% before downloading and the Post List shows all present and available, but in fact there are many Blocks missing when downloaded. Might offer some insights with NB's logic on what's a good/bad Post.

Thanks

I am still scratching my head about NB's Test Download functions saying the Post is there 100% before downloading and the Post List shows all present and available, but in fact there are many Blocks missing when downloaded. Might offer some insights with NB's logic on what's a good/bad Post.

You have to remember that this is all dynamic. What's true this second might not have been true 30 minutes ago. You can't look at a set right now and know what it looked like 3 hours ago if it was just downloaded in the past few hours.

It might not have been complete when it was added to download hours ago but it's complete now. That's what I was explaining above.

Yes, appreciate Dynamic. I get used to NB acting a certain way and when things change, I take note. Seems the issues are external to NB.

As always, thanks for the feedback and dialog.

Thanks

Just ran the above test on a Post that won't d/l and ends up in the Failed List. NB said it completed at 100%, yet when the RARs are d/l'd, parts are missing. Might want to check this test out.

Thanks

Quade wrote:

What news server are you using?

That's kinda a big step. You could also try to make the scanners ignore Newsbin's connections to the news server and see if that improves things.

I'm using usenetserver and how do you make the scanners ignore Newsbin's connections to the news server? Is that some kind of setting in options?

Went to check today's partially downloaded files and this is my latest error. [05:47:11] ERROR AutoPARPlugin: Autopar Block Counts Indicate good files but, UnRAR FAILED: No UnRAR:

*update*
I just did some tests and this isn't just my watchlist. The same files my watchlist is downloading that don't complete, I try downloading manually and they aren't finishing (I mistakenly thought they were before). However, not all files are giving me problems, other files will complete and unrar just fine. I've tried different servers and have had the same results so I've pretty much eliminated my server as the problem. I turned off my firewall/antivirus and tested again only to get the same issue. I have a second computer and will now test on that computer and report back.

I have a second computer and will now test on that computer and report back.

Dex and I were working with a guy over email with similar issues. Dex and I both got perfect files. Guy over email had 100% bad files. He claimed he was only using MS Security essentials.

[23:27:22]  DEBUG AutoPAR: ScanFile 31 BadFiles : <bla>
[23:27:22]  DEBUG AutoPAR: Scan 31 Blocks File: <bla>

Every one of his larger files was bad. Newsbin and Quickpar both confirmed it. Dex and I had no issues with the files. Dex used the same server he did and tried both SSL and no SSL. The log will either say "GoodFiles" meaning the file is 100% correct or it'll say "BadFiles" meaning there are missing PAR blocks.

Ok just tested the same files on my second computer. Had to install Newsbin pro to the latest version so there was no previous configuration. The three files that I tested failed to complete on both computers. One difference is that on the computer with the fresh install, 2 of the 3 files Newsbin detected something wrong and they went almost immediately to the failed list. So some setting with a fresh install is at least catching the fact that it can't download all the way but the failed list does not give an error reason. The third file downloaded and failed to unrar like before so its not catching them all. I am running windows 7 on both computers and don't have ms security essentials. I normally use panda but have a different antivirus on my second computer and I haven't changed any software recently to cause this problem. I did update my primary computer to the newest version of Newsbin pro but that was only after I noticed the issue.

New development: After trying to download 4 files from 2 different servers on two different computers for the last couple of hours and getting the same results, all the sudden 3 of the 4 files download and complete with no problems. These are all files that were posted about 2 to 3 hours ago. Could it be that the files were taking that long to actually upload to the servers or for some reason the servers weren't getting all blocks until now? Maybe if I wait to download the files for a few hours that will solve my issues?

Quade/Dex & All,

Ok, Posts that would not d/l and go to the Failed List last night d/l'd this morning. There were Blocks missing last night and was repeatable. Ctrl-y worked just fine this AM for those entries NB showed already d/l'd.

Still have two concerns:

1. Retries are set to 100 but the incomplete files last night are going to the Failed List as opposed to waiting for them to complete as you suggested a couple of days ago.

2. "Test Download Files" may be broke because it says the files are complete 100%, but Blocks are actually missing.

Thanks

Been investigating this. You and a number of people are having this issue. I was able to detect the "bad yenc" errors. It's a case where the news server is occasionally delivering incomplete data. This will cause the need for a repair. I don't think this is necessarily the only thing going on but it's one thing I noticed while testing some bulk downloads.

I haven't looked at the watch list thing yet. It's not the same as "needing to repair more files than normal" which is what I'm investigating.

When you "test" download. Newsbin justs asks the server if it can deliver the chunk. That doesn't means the chunk will be correct. At least for these "bad yEnc" errors, the data is bad which triggers a repair. The bad Yencs aren't happening that often. You wouldn't even notice unless you were running under the debugger. That's why I don't think this is the only thing. More people with Newshosting have reported this than any other server. I'm not blaming them. It's just a data point I'm noting. I'm still getting a decent number of 430 errors from Astra too.

It does seem like it's a Server/Poster Issue.

Any way to enhance the Test Download Function to make it a better tool/indicator/tester?

Thanks

Any way to enhance the Test Download Function to make it a better tool/indicator/tester?

It's binary - 1 or 0. Either you download all the chunks and test the data, which is just downloading and buys you nothing OR you check the availability of the data from the server and assume the server has correct data. Usenet doesn't support any kind of middle ground. There is no such thing as a partial download.

I'm not at the point where I'm blaming anyone yet. I'm using AW and giga and while I see more repairs than normal. It's not enough that it prevents the files from landing. Numerous people know are reporting that files are too damaged to repair.

You need to distinguish between watch list problems, and problems landing files. If you can manually land the files then you have a watch list problem. If on the other hand every file fails, which is what some people are reporting then it's not watch list related. I'm willing to believe there's a watch list issue but I'm focused on the "nothing downloads correctly" people right now.

I'm thinking Dex and I are going to try some rapid beta's in the next couple days to try to see if tweaking the code changes this. 6.61 is more aggressive about assembling incompletes than 6.56 or 6.60 but I don't know if that related to this.

The more info I see here, the more I understand NB is spot on and the issues are external to NB....see below....Logfile.txt entry just a few minutes ago.

[09:16:15] DEBUG AutoPARPlugin: All files downloaded, not enough blocks to repair

Thanks

The more info I see here, the more I understand NB is spot on and the issues are external to NB....see below....Logfile.txt entry just a few minutes ago.

[09:16:15] DEBUG AutoPARPlugin: All files downloaded, not enough blocks to repair

Thanks

I'd love this to be true but you can't make that assumption. At least I don't make this assumption. Everything that touches the data is suspect at this point. The poster, the servers, the net connection, security software, Newsbin. There could be problems at any level. The only thing I can fix though is the newsbin side of it.

How many servers are you using? I'd try disabling servers and attempt clean downloads with one server active at a time and then see if you can finger a server for this. If you do the same files over and over, make sure you delete the old damaged ones first.

One server only....Usenetserver Secure.

Thanks

You're not seeing any 430 or yEnc errors on the log messages?

No, neither. This morning, am only seeing not enough files or blocks messages in the Log. Did not keep prior logs unfortunately.

[09:16:15] DEBUG AutoPARPlugin: All files downloaded, not enough blocks to repair

Am also seeing Badfiles in the NB Logging Tab though.

[10:19:00] DEBUG AutoPAR: ScanFile 7 BadFiles : xxxxxxx.2015.10.14.xxxxxxxxxxxxx.hdtv.x264-crooks.r03

Will watch and advise.

Thanks

So I've pretty much figured out what is going on for me but I don't know why. Well my problem (not sure how similar to everyone else) seems to be that the more recently the file has been posted, the more likely that it will be missing blocks and therefore won't download. This is what is causing my watchlist problems because sickbeard will download an Nzb as soon as it is available. This means that within at least an hour, the file is attempting download and will probably fail. Now after that file has failed, if I leave it sit for a few hours and then download it again, all the pars are there and it unrars without any issues. Not sure why this is happening but it seems to be the case for me. It seems like the full files are not available on the server for a few hours. I tried 2 different servers: usenetserver and blocknews and this has been my experience with both.

I have tried waiting for hours and then manually downloading the NZB and I am still having newsbin almost immediately send the download to the failed files tab. No amount of CTRL-Y has gotten past this point. I have had to resort to torrents!

Update:

Several Watch_List Posts failed into the Failed List late last night. Opened the same Watch_Lists this AM and successfully used Ctrl-Y on several failed Posts. Also, the same subject from other Posters d/l'd normally this AM.

Maybe, NB should wait longer for incomplete Posts with yEnc/Block Problems before going to the Failed List. NB successfully waits for Posts with good yEnc/Blocks for the rest of the parts and then performs as usual (which has happened countless times for me)

Or, knowing what we know now and better understanding NB's logic, NB is doing it's job advising us that there is a yEnc/Blocks issues and we just leave NB as it is now for the most part, and we use the facilities (Ctrl-Y) to recover at a later time.

Thanks

TomDavisSr wrote:I have tried waiting for hours and then manually downloading the NZB and I am still having newsbin almost immediately send the download to the failed files tab. No amount of CTRL-Y has gotten past this point. I have had to resort to torrents! Image link not allowed for unregistered users

So I found that waiting for hours doesn't always help if some of the rar files had previously downloaded to the hard drive before the nzb file moved to the failed list. Those rar files will be sitting on the hard drive with incomplete blocks. After a few hours, even if the full files are available on the server, newsbin won't automatically redownload the incomplete files it had previously downloaded. Make sure you delete those rar files before trying again and it should work.

I had two files that downloaded about 5 rars each before failing. When I retried hours later, all the remaining files were complete and the first 5 rars of each file were missing too many blocks to repair. Deleting those 5 rars and trying again allowed the files to be fully downloaded and then unrared just fine.

This seems to be related to another problem on this board:

viewtopic.php?f=43&t=39027

I've had these same issues in addition to the issues on this other topic. One thing of note that I did notice is that if I'm downloading through an XFinity connection directly using SSL, I'm having trouble a lot more than if I use a VPN to connect, so perhaps Comcast is doing something to try and interfere with usenet communications in some fashion. Not sure if there is any relation to these errors and certain ISPs.

I've also noticed that if I download the same post multiple times within minutes, I am often getting a different amount of data each time, often ending up with a file larger than the stated post size. This is something that I've never seen before.

One thing that I think is going on is that if a post's blocks are not present yet on the first server where multiple servers are in use, newsbin seems to fail the post and push it to the Failed Files tab without an error description rather than trying on the subsequent servers. For example, using the same NZB, when I had 3 servers enabled, the nzb failed to download in this fashion, however when I only enabled one server, in this case usenetserver, the nzb downloaded complete.

6.62B1 Retries set to 100

Had no issues last evening. Got all Watch_List items and nothing went to the Failed List, all files UnRared properly and NB cleaned up perfectly. Turned in around 10pm, left NB running until 1am. Last file d/l'd at 12:19am. All in all, a perfect evening

I have the LogFile.txt if you would like to see it.

Thanks

Wonder how many retries it's taking on average? You know with retries set to 100, it'll keep trying for weeks or months. The retry algorithm backs off longer after each try.

Although there still does seem to be a completion issue with some of the news servers for some reason, I agree that 6.62B1 does seem to have fixed the majority of the issues with regards to the request not getting pushed to the "Failed" column with no error, allowing the posts to complete download/retries if possible. I put the retries at 12 and that seemed to work pretty well. It is also now not just ignoring the additional new servers that do have completions when the first news server is having issues. Thanks for the quick fix.

OK.. Using the new beta and last night it worked on the one item in my watchlist. Optimistic. Tonight will be the big test!

Thanks Newsbin crew...

All 4 of my watch list downloads worked last night without a hitch! The beta is working wonders...

Just wondering if everyone else is having the same problem with data usage. Because of the way the beta is working for me, Newsbin keeps downloading the data over and over until the files are completed. I've downloaded 175 gigs of data since Sunday on files totaling no more than 30 gigs. The file I'm currently downloading has been attempting to complete for about the last 30 minutes and has downloaded approximately 20 gigs. The file is only 1.2 gigs in size and its still not finished downloading yet. I'm guessing it will be another 10 gigs or more before it finishes. Has anyone else noticed the same issue? Any way to fix this?

How are you measuring it? I wonder if Newsbin is counting the data it's reading back off disk.

My ISP has a data usage meter and it says 175 gigs since the 18th, not counting today. I don't do much online besides usenet so most of that data is my downloads. I've tallied up all the files I've downloaded since Sunday and its 30 gigs but I'm rounding up. When I first start downloading one of these problem files, the newly uploaded ones with the problems everyone's been having, the downloads are completely red. All the data that doesn't complete I'm pretty sure is still being downloaded (and possibly overwritten over and over). The file that I just downloaded a few minutes ago completed in about 45 minutes. My observation is that the blocks that show as red (incomplete) downloading over and over is my problem.

My observation is that the blocks that show as red (incomplete) downloading over and over is my problem.

If they're red, they're not downloading at all much less over and over.

Edit: I did think of one case where they'd turn red and re-download. If you have strict yEnc checking turned on and the chunks from the news server are reporting "bad yenc format".

If you have this, you might as well delete the download.


I did a test. I made an NZB file so all the source files had bogus message-id's so they'd all 430 (turn red) but all the PAR2 files exist. Then I added it to download.

1 - I cleared the server counters back to zero.

2 - I Watched task manager.

As far as I can tell, almost no data transfers during a retry (which is what I told you before).

Then I modified the NZB so about 1/2 of the chunks for the files downloaded but none of the files assembled.

1 - I cleared the server counters back to zero.

2 - I watched task manager.

With this setup. Newsbin will download about 60 megs of the file. If your theory was correct, then a retry should keep downloading that 60 megs over and over again. It doesn't. The counters in Newsbin don't go up with each retry. Watching the logs, I see nothing re-download. Download speed remains close to zero the whole time it's retrying.

So, I don't know what your ISP is counting but it doesn't seem to be Newsbin.

Do you download headers?

Ok I was using the older beta and wondering if that had something to do with it. I just downloaded the newest one to see if that changes things. Also, I don't normally download headers as I use an index site for nzb files. I'll keep a close eye on my internet data usage and report back. Thanks for all the help.

I'll keep a close eye on my internet data usage and report back. Thanks for all the help.

I'd suggest you use some local traffic counters too. There's nothing that says your ISP is actually counting the traffic properly.

Quade wrote:

I'll keep a close eye on my internet data usage and report back. Thanks for all the help.

I'd suggest you use some local traffic counters too. There's nothing that says your ISP is actually counting the traffic properly.

Ok just realized the problem didn't fix with the newest beta. I took video of the issue and pulled up the resource monitor to monitor the network settings. Can I upload that video or send you a link to download it so you can see whats happening.

Watched your video. You problem is completely out of left field. It's nothing like I expected.

I PM'd you. I really want to get to the bottom of this before we release 6.62. I'm going to hold the release up till we figure this out.