newsbin.com

Hi,

I'm wondering how to turn off the new retry feature in 6.30. The default setting of 2 apparently means "forever", as it never stopped retrying for me. The options say "0=infinite", so I guess "0" isn't it either. So I tried setting to 1. That doesn't seem to work either, downloads with missing parts still keep sitting in the download list with one retry after another and ever increasing delay time. I wouldn't even mind 1 or 2 retries, but currently, regardless of the setting, it apparently retries until eternity...

Thanks.

I noticed this behavior yesterday also. Had a single rar file constantly being retried and assemble incompletes did nothing. Finally had to remove it from the download list and grab par's to 'repair' the missing file.

A temporary fix is to exit newsbin and restart it. Yes, I have seen the problem too.

Got to say ditto for me too. Over time I get 100's of downloads stuck on retries. Eventually they block other downloads starting so it stops everything until I move them down. In some cases if I manually kick off the par files they repair. In some cases they have been on repair and I could the unrar file anyway which is wierd. Ideally I would see a few things happening:

1. Ability to switch off retries
2. Creation of another list (like the wish list?) that these go into? Maybe this can be referenced according to user preferences, or maybe moved back into the main queue if the user wants the download trying again
3. The ability to download all the PARS (optional of course) which may fix the download anyway.

Easy for me to say of course

I'll be looking into this. Is it mainly an issue with picture groups?

Hi

Thanks - much appreciated as ever For me its binary groups but that's just my experience.

Thanks

I mean, all groups newsbin uses are binary groups.

Picture groups, you might have a block of pictures and some PARS. Non-picture groups, you have blocks of RAR files. You add one block, it appears in one line and has PARS embedded in the block of rars. This sounds like an issue where the RARS and PARS aren't combined into the same unit.

It happened to me in a non-picture group (2.3 gig rar set). I noticed it after all the rar's had been downloaded but a lot of them were incomplete (thanks to the problems Astraweb is having atm). Some of the rar's would assemble with "assemble incompletes", but there a couple that would not and seemed to be stuck in some of sort of permanent retry loop. With another rar set (same group, same size), I baby sat it and did 'assemble incomplete' as soon as all the connections had moved on to the next rar and didn't have any problem. 99% of the reasons for the incompletes were 430 not found errors and 1% timeout errors. FWIW, the download source was from headers dl'd from Astra.

I use Astraweb with 4 servers defined , US Slave/Fill 1 connection, US Slave/Normal 3 connections, EU Header/Fill 1 connection and EU Slave/Normal with 3 connections.

I hope that information helps. I signed up for a free acct at xsusenet.com and using it as a fill server which has worked very well so far so I'm not having the retry problems since xsusenet is filling the missing pieces.

but there a couple that would not and seemed to be stuck in some of sort of permanent retry loop.

It won't re-assemble if some chunks are still in the process of downloading. What I've seen is that the connections can get stuck for minutes at a time and eventually timeout. Basically Newsbin sends a command and the data never comes back so, eventually the whole connection times out. I'm a little leary of tightening this time up but maybe I should. Maybe an option to have aggressive timeouts, 30 seconds say with the option to back off if there are weird networking issues.

FWIW, for me, this happens on RAR sets. No timeouts involved, purely "430 no such article". So perhaps the issue would be solved if the retry mechanism were changed to simply not kick in when the failure reason for a post is a 430? It should of course still kick in for timeouts or connection drops.

430 isn't always a permanent error. I've done that before, made it not retry for 430's. It didn't turn out well. The problem is some servers have the data but, return a 430 but, then when you hit them again, they return the data. I'm assuming it's server farms and not all the servers have the same data. I'd love to be able to assume "430" means "don't have it and won't ever have it" in practice, that's only true on a couple servers.

You also have cases where you have an NZB for some files but, they aren't on your server yet. In that case the 430 means "I don't have the chunks yet". In that case you want to keep retrying the 430's till all the data comes down.

After the retries Newsbin looks at what data it has and makes a decisions "Do I seem to have enough PARS waiting to download to compensate for the missing chunks". If the answer is yes, it assembles and downloads pars. If no, it waits. I think it's this decision making that I need to improve to make this more reliable.

Understood, so 430 shouldn't be a showstopper for retries.
However, in either case, it should ultimately honor the number of retries I allow in the settings. If I allow for 1 retry, then it should retry exactly once.

This is the behaviour I have observed:

1. The download stating Retry 44 Next Retry 13228 0% completed - but I've set retries to 1?
2. The retry count kicking in right at the start of the download, for example on the PAR2 file itself - and never completing - hence the exanple status of 379kb of 7.8GB Retry XX etc (so I have to expand the download and then kick off all the other paused items separately and so far they have all completed and unrar'd fine). If there are enough of these newsbin gets no-where as it retrying the same thing time and time again and so downloading stops until I perform some remedial action.
3. There is a retry conditiion as per 1 - but if I expand the download and kick off the PAR file downloads then the overall download is able to be unrar'd (I would say that is was repaired first but in all truth I am not absolutely sure that is the case - I'll need to check this to be sure). So in my circumstances, I would prefer the PAR files to be downloaded automatically rather than keep hammering on at the retry thing.

A question - why do I need to expand the download to then select the child items in order to start them? selecting the d/l and hitting the icon doesn't work, nor does resume / restart (AFAIK) - par files or rar parts stay paused? This is a bit of a pain in the arse - I had to expand 20-30 downloads the other day to manually start the PAR files downloads so they would complete. I was hoping that hitting the > icon would just start everything?

A question - why do I need to expand the download to then select the child items in order to start them? selecting the d/l and hitting the icon doesn't work, nor does resume / restart (AFAIK) - par files or rar parts stay paused? This is a bit of a pain in the arse - I had to expand 20-30 downloads the other day to manually start the PAR files downloads so they would complete. I was hoping that hitting the > icon would just start everything?

The odd thing is I don't have to. When I download, Newsbin unpauses the PARS it needs, does the repair and the files clear out of the download list. That's the goal anyway. I'm unclear why you have to mess with it when it just automatically repairs for me. Assembles incompletes, downloads PARS and repairs.

Quade wrote:

A question - why do I need to expand the download to then select the child items in order to start them? selecting the d/l and hitting the icon doesn't work, nor does resume / restart (AFAIK) - par files or rar parts stay paused? This is a bit of a pain in the arse - I had to expand 20-30 downloads the other day to manually start the PAR files downloads so they would complete. I was hoping that hitting the > icon would just start everything?

The odd thing is I don't have to. When I download, Newsbin unpauses the PARS it needs, does the repair and the files clear out of the download list. That's the goal anyway. I'm unclear why you have to mess with it when it just automatically repairs for me. Assembles incompletes, downloads PARS and repairs.

Unfortunately for me that seems to be the behaviour. In most cases newsbin doesn't try to download the PAR files - they all sit at paused. I have seen some examples where one PAR has been downloaded, but that hasn't been enough, I have had to download the others before the download completes).

Working on getting an RC3 up today. It gets rid of most of the "retry" lines and is more aggressive about downloading pars.

Hello,

I have download 6.30 bulid 1832 and I have the same problem on huge file (> 4Gb).
in the list I have the main entry and all sub files and at the end the PAR files...
When I have an error on a file, newsbin loops retries forever... (retry 33, 5455%...)

however, when I force manually to download PAR files, it change to reparing and then unrar correctly.

For configuration :
- I have checked "deactivate automatic PAR pause" (not sure about english translation, as I'm using french GUI).
- I put "max retry" = 2

do I missed something ?
if not, is a patch under developpement ?

thanks for your help
Stephane

viewtopic.php?f=44&t=30181

I'd give RC7 a try. There have been specific changes made to address some of this.

Hello,

thanks a lot for you quick answer.
I will give it a try this week end and I will let you know.

best regards
Stephane

I hate to say this - but I've just upgraded to 6.31 Final and it still retries forever and ever, although I've set max retries to "1" in the options.

What you need to figure out is why it's not finishing.

- Select one of these sets right click properties and check the PAR counts.

- Hit the plus button and look at the individual files.

There will always be cases where Newsbin's going to just leave a download alone and occasionally try it again.

- Not enough par blocks to repair

- UnRAR disabled in the options.

So, the question is why in your specific case a specific set of files isn't being assembled and repaired/unrared. What retries set is when Newsbin decides to look into assembling incomplete files. Even then, it'll only assemble them if certain specific things are true and it thinks it has most of the files.

Quade wrote:What you need to figure out is why it's not finishing.

- Select one of these sets right click properties and check the PAR counts.

- Hit the plus button and look at the individual files.

Will do next time and post here.

There will always be cases where Newsbin's going to just leave a download alone and occasionally try it again.

Wouldn't "Max Retries" mean that it will NOT, NEVER, retry once the maximum is reached? Otherwise I probably have misunderstood the meaning of "maximum"...

- Not enough par blocks to repair

- UnRAR disabled in the options.

Not using Auto-repair, not using Auto-unrar, so these shouldn't cause any problems.

Bismark wrote:A temporary fix is to exit newsbin and restart it. Yes, I have seen the problem too.

It's odd how we can all have different results with the same problem as that doesn't work for me and it just gets to the same point and sticks on the endless loop but bizarrely if i shut down the computer and restart it and then restart newsbin these files unstick and compete.

viewtopic.php?f=43&t=30262

one odd glitch i had yesterday that maybe totally unrelated is that i paused a download but it still carried on downloading even though the speed counter said 0 and the file had the pause icon?

Not using Auto-repair, not using Auto-unrar, so these shouldn't cause any problems.

I'll say this is likely the source of the problem. All the fixes are for people who actually use autopar to repair and unrar. Let me check the code. In cases where autopar is disabled, it should just be downloading and removing from the download list with no paused items. If the files are incomplete though, the retry logic kicks in.

one odd glitch i had yesterday that maybe totally unrelated is that i paused a download but it still carried on downloading even though the speed counter said 0 and the file had the pause icon

If your computer gets behind. The download might stop but, there might be up to 100 blocks of data in memory still waiting to get written out. If you've increased the cache size, it might be well over 100. Pausing just stops downloading, it doesn't prevent other processing from finishing up. The "Cache" line down on the status bar gives you visibility on how many chunks are queued up. It's a sign that the download is faster than the disk drives.

Quade wrote:

Not using Auto-repair, not using Auto-unrar, so these shouldn't cause any problems.

I'll say this is likely the source of the problem. All the fixes are for people who actually use autopar to repair and unrar. Let me check the code. In cases where autopar is disabled, it should just be downloading and removing from the download list with no paused items. If the files are incomplete though, the retry logic kicks in.

This is correct - completed downloads without missing parts are properly assembled and removed from the download list, no problem here. Downloads with missing parts, however, are retried in an endless loop instead of stopping after "max retries".

Because they don't have PARids, the regular assemble logic might not kick in.

Quade wrote:What you need to figure out is why it's not finishing.

- Select one of these sets right click properties and check the PAR counts.

So here we go:

Retry: 9 Next Retry 2295 0%: Completed

Number of Source Files: 107 Source block count: 8744
Complete Files: 0 Available block count: 0
Damaged Files: 0 Source block size: 640000
Missing Files: 107 Repair Block Count: 437

Which is strange, since 106 files are actually complete and one incomplete. All PARs are available and have been (manually) unpaused.

Quade wrote:- Hit the plus button and look at the individual files.

All "Downloaded", except for one "Incomplete".

Max. Retries set to 1, using 1 server plus 1 fills server. So in my book, the maximum retries I should ever get is "2" (1 retry from the main server, one from the fills server). However, this retry count will go up and up until eternity. For me, it looks like the "max retries" setting is - still - completely ignored.

Yeah, because you have autopar turned off, Newsbin will never assemble the incomplete file. Newsbin won't assemble till 90% of the blocks are available and with no autopar, it never counts blocks. Maybe I should make it count blocks even with autopar turned off but, not attempt repair or unrar. Either that or have some other failsafe mechanism to assemble incompletes when people disable repair.

I'll have to think about it. I imagine if you let Newsbin repair and unrar but, not delete the source RAR files it would work too.

Newsbin's designed to run with autopar and unrar enabled so, cases like this need special handling.

I'm not sure and i may be wrong but maybe i found out why people have endless retries.
When i set different number of retries like 2, 3 and 8 in the newbin options itself
the .nbi still shows "MaxRetries=0" all the time after changing.
(in the newsbin program options it shows 2 or 3 or 8 correct also after a shutdown/restart
of newsbin)

I think the proper value is "RetryBeforeAssemble". Nice try, though.

DT

Well spotted alexander038

The Max Retries Options in newsbin doesn't change the NBI and it is stuck on MaxRetries=0

That's because it's unused now

"RetryBeforeAssemble" is what's used now.

I'm clearly not making clear what's going on. When it's in retry mode it's telling you. "There's nothing else I can do this second" so, either posts are incomplete, there aren't enough par blocks to repair or all the files in the set haven't downloaded yet.

You need to check the autopar properties to see what it's thinking.

I need to figure some way to make this clearer.

I was indeed not sure if that value in the .nbi was still in use.
ps: i do not have problems with endless retries myself but i have a backup server.
I was just curious

My thought is that mostly everything posted is downloadable these days because of PAR files so, we don't really want the files to just stop trying. Maybe that's a bad idea. It's exceptionally rare for me to download something that can't be repaired these days.

I don't want to turn the retries off personally, i like the option! sometimes after a few retries
i'm able to complete and unrar a file without using the pars/repair simply because of the extra retries.

For the sake of a few pars i'd rather a classic download option where by newsbin acts as a dumb downloader and just downloads everything in the set including all the pars regardless if they are needed or not and then goes into par repair and unrar once all the parts have been downloaded, instead of retrying it just repairs what it can get.

The option above "Aggressive Mode" in the autopar options will make it download everything and then do the repair from all the files. I'm not mentioning the option by name because I changed the wording. It essentially disables any PAR pausing.

Come to think of it i remember the unmentionable thing above "Aggressive Mode" being in one of the release notes but why it never stuck in my mind i don't know.

maybe all that hairy 70s porn has taken it's toll. :p

Haha, It's unmentionable because I was too lazy to run 6.31. I only have 6.4 handy and know I changed the name because the old name wasn't particularly clear.

Plankton wrote:maybe all that hairy 70s porn has taken it's toll. :p

Hairy 70s porn doesn't do you any harm just look at Dexter. :¬)

Quade wrote:Yeah, because you have autopar turned off, Newsbin will never assemble the incomplete file. Newsbin won't assemble till 90% of the blocks are available and with no autopar, it never counts blocks. Maybe I should make it count blocks even with autopar turned off but, not attempt repair or unrar. Either that or have some other failsafe mechanism to assemble incompletes when people disable repair.

How about having it assemble the incompletes when you click "Assemble Incompletes"?

DaveXVI wrote:How about having it assemble the incompletes when you click "Assemble Incompletes"?

it does (at least for me). Are you saying this does not work for you?

itimpi wrote:

DaveXVI wrote:How about having it assemble the incompletes when you click "Assemble Incompletes"?

it does (at least for me). Are you saying this does not work for you?

It works, but only when it's good and ready. If it's in the middle of a retry countdown, and I right-click the file and say "Assemble Incompletes," it's a coin flip whether it will do it or not. About half the time it does it, and the other half it just continues the countdown, even if I keep clicking on "Assemble Incompletes." I can't find any pattern to tell me whether or not it will ignore me on any particular try. And this goes both for files that were put in the download list as units, and for segments that were put into the download list individually, after expanding the file in the headers list.

As far as I know, this has happened with every version above 6.0, when autopar is disabled. I didn't bother reporting it before because I've been made to feel like I'm some kind of weirdo if I don't want to use autopar, and I'm evidently incapable of making a convincing case for why I don't want to use it.

You're not a weirdo, but accept the fact you're setting yourself up to do more manual intervention. There may well be an issue where assemble incompletes isn't responding, but I sure haven't seen it, it's always been my goto intervention button that kick starts the autopar process, so perhaps it's tied in with that. I'm suspecting, though, it's more about what's going on with the connection to the server. I assume this only happens when there's some sort of trouble, otherwise you wouldn't be doing that action. Anything you can ascertain about precisely what's happening (chunk missing on server and nb is retrying, connection appears to be hung, etc.) would be useful.

DT

works, but only when it's good and ready. If it's in the middle of a retry countdown, and I right-click the file and say "Assemble Incompletes," it's a coin flip whether it will do it or not. About half the time it does it,

If there are still chunks in "Downloading" State, which you can see as chunks with 2 colors, it won't assemble. It assembles when the file is essentially idle. I'll bet you some of the chunks are still "Downloading" when it won't assemble. I've seen cases when Astraweb gets busy for instance, where Newsbin will ask for the block of Data and it just waits for the server to respond till it times out. I don't know what server you're using but, I do wonder if you're seeing this.

Quade wrote:

works, but only when it's good and ready. If it's in the middle of a retry countdown, and I right-click the file and say "Assemble Incompletes," it's a coin flip whether it will do it or not. About half the time it does it,

If there are still chunks in "Downloading" State, which you can see as chunks with 2 colors, it won't assemble. It assembles when the file is essentially idle. I'll bet you some of the chunks are still "Downloading" when it won't assemble. I've seen cases when Astraweb gets busy for instance, where Newsbin will ask for the block of Data and it just waits for the server to respond till it times out. I don't know what server you're using but, I do wonder if you're seeing this.

Yes, I use Astraweb, and lately (last several months) it has been very common to have Newsbin pop up a message that it's having problems with the server. I guess that's what is happening, but I thought "Assemble Incompletes" should mean that I want NB to stop trying to download the file, regardless of the reason it's not succeeding.

I mean, I could make it assemble come hell or high water. It's dangerous though in some ways. It's trying to be smart about assembly to get you the maximum number of chunks. You could throw the Euro astra server into the mix too.

Wonder if a shorter timeout would look better? Default is something like 4 minutes. It's possible to reduce it. Normally this problem just sorts itself out if you leave it alone.