Server "Use Typical Retention"

[Ribble]

Servers, Advanced Options, "Use Typical Retention" still retries from fill server, regardless of setting. Not sure what it should do, was trying to stop Newsbin from retrying 17 month old file on a one month retention server.


Help file says "earlier" than servers retention. Not sure if that means older or newer files than servers retention in days.

"Use Typical Retention"
"When enabled, this setting will only use this server for downloading posts earlier than the specified number of days. This is useful when multiple servers or service providers are enabled and have different retention lengths."

[itimpi]

Newsbin should not be trying a server for any file that is older than the Typical Retention value. If this is not being honored when set as a fill server then I would think that it is a bug (I could certainly see that case as not one that does not normally get tested as the fill server is normally the longest retention one).

[Ribble]

Master server is unlimited downloads on an annual contract.

Fill server is limited downloads, one month retention, monthly contract.

So my fill server is normally not used except for fills.

Sorry if I have this backwards. "Use Typical Retention" looked like it should have stopped retrying files older than server 30 day retention.

[itimpi]

Sorry if I have this backwards. "Use Typical Retention" looked like it should have stopped retrying files older than server 30 day retention.

It should! However my point was that it was not very common to have a fill server with shorter retention than the none-fill one, so that case could well have slipped through the net when testing.

I must admit it is not clear why you are paying for a server with one month retention which seems to offer less than your main one?

[Ribble]

"paying for a server with one month retention "

I am getting what I pay for. If I want more, I'd have to pay more.

It occurs to me that my date format may be causing the older/newer problem.

My date format: Today is "11:08:01" ... and no, I am not going to change that - it has to do with sorting files by date.

[itimpi]

The master server with unlimited downloads - if it has longer retention than 30 days I do not see what benefit you are getting from the fill server?

[Ribble]

In an ideal world, none of us would need more than one server.

Newsbin and my master server are telling me about a 99% completion rate with 900 day retention. The story changes greatly when the actual download is started and pars are exhausted.

The majority of my important downloads are within a week of the posting date. I have to decide whether or not the second server is of any help. I think it is, or at least for now, worth paying a few bucks per month to have a second server available.

I am convinced that Newsbin (now 6.01B1 Build 1127) is partially at fault for some of the missed downloads. There seems to be a general problem when the last file of a set is downloading and Newsbin has already started a new set of files. If I pause a file in the middle of a set and Newsbin downloads everything else in that set, there is a high likelyhood that the paused file will not complete. A manual retry usually works.

OK, I could blame the server for some of the problems. Or perhaps my 4 core AMD computer with 16 GB RAM. And I am also testing Newsbin on a somewhat faster redundant computer system. It all takes time. I am a user and abuser and believe it or not, reluctant to report problems unless I can narrow down the cause.

Please excuse my diatribe on master and fill servers. It does concern Newsbin 6.0 as seen from the eyes of a Newsbin user. Just trying to stay on topic in this thread. I like Newsbin else I wouldn't care.

[mho]

My date format: Today is "11:08:01" ... and no, I am not going to change that - it has to do with sorting files by date.

Not that it should matter, but that is a time format, not a date:-)
2011-08-01 will sort just as well (or 20110801 for brevity) and be much more iso-8601:-)

- mho

20110802T033930+0200

[Ribble]

Mho - Opposite of Ohm - I like conductivity in a world of resistance.

Would rather use "110801" which worked in earlier Windows (pre-XP Pro).

Newsbin seems to work fine with the odd date or time format. I guess the timekeeper is in Windows ticks anyway.

[Quade]

Looks like typical retention is coded to work whether it's a fill or not. If it's the ONLY active server though, it's ignored.

It's in unix timestamp. Seconds since 1970. It doesn't matter how you display it, comparison and filtering is all done in unix timestamp mode.

[Ribble]

I am convinced that Newsbin (now 6.01B1 Build 1127) is partially at fault for some of the missed downloads

To narrow this down, I halted a download operation and noted:

This was a file posted Nov 18, 2010. It includes 49 RAR's, an NFO, an SFV, an SRR, and 10 PAR2 correction parts.

What are the odds that the last part downloaded is the only one to fail? If it were the last part in the file, I would blame the server. In this case I paused RAR.r44 and Newsbin skipped RAR.r44, completed the remaining five RAR's, and paused. Then I resumed the RAR.r44 download which completed and autocompleted. After a minute delay (Newsbin downloading next file set), Newsbin started downloading "Vol03+04.par2", indicating an error somewhere, and I paused this.

Running QuickPar showed all files complete except the .SFV (not needed) and RAR.r44 which showed 19/20 blocks available. I then renamed the downloaded RAR.r44 file and manually downloaded RAR.r44 again - it showed segment 37 and segment 38 (of 40 segments) already complete before the download even started. Unpausing the RAR.r44 manual retry allowed completion of RAR.r44, then Newsbin started downloading "Vol01+02.par2" which I paused.

Running QuickPar again showed all files complete except the .SFV file. I then did a manual "UnRAR/JoinFiles" and NewsBin decoded and made a video file which played fine.

What I was trying to show here:

- NewsBin tends to fail the last downloaded file.
- NewsBin tends to fail the last downloaded file, even when the failed file is not the last file in the set.
- Newsbin started downloading PAR2 correction file before reporting the completed RAR.r44 download.
- NewsBin started downloading a second PAR2 correction file even after having a complete file set (sans .SFV file).

Prior trials showed that NewsBin will autocomplete without downloading the .SFV file and in fact did not need this .SFV file to succesfully decode.

My apologies if I am the only one who sees a high likelyhood for the last downloaded file part to fail.

[Ribble]

This could be a Schrödinger's dead cat in a box problem - we don't know if the cat is dead or alive until we open the box to check. Could be that opening the box kills the cat.

Or in NewsBin terms, the mere fact that I am watching causes the failure to happen.

So, set NewsBin to delete completed downloads and place the deleted files in the recycle bin. Then before emptying the recycle bin, examine the slime trail for failed files and PAR2 correction files to see where future failures are most likely to occur.

[Quade]

My file downloads are almost universally perfect. So, I'm looking at this with a good deal of skepticism. I have this 80 GB test run I do. It combines larger sets (10+ Gigs) and smaller sets. Some are purposely damaged to test repair and some are perfect. I see no kind of "last file is damaged" problem. The damaged files repair, the good files download perfectly. In my normal downloading, I have 2-3 servers active so, it's exercising the multi-server code.

So, the question is why do you seem to see problems in the last file that I don't see. The IRC guys aren't reporting it either. I'm not expressing any opinion about whether it's your machine, servers or Newsbin. I don't know. All I know is that it doesn't seem to happen here which means it'll be problematic to find and solve. There are other people who have reported having to repair high percentages of their downloads. The same files will download perfectly for me so, again. The mystery is what's different between them and me even though we run the same software. I do use SSL on all of my connections and no virus scanners to mess with the data.

During download are there errors logged? 430 errors for instance?

[Ribble]

Not exactly sure how to look for errors.

The NewBin logging list gets a lot of entries. There are 3,240 entries and I didn't see any errors. Get about 2,000 log entries per hour.

I think I was getting 430 Errors from the one month retention server when retrying segments older than one month.

And there usually is some kind of log that opens in a new window when I first start NewsBin, and I tell it to go away.

Will look for logged errors when odd things happen.

I was downloading a lot of old movies when I started tracing failures and I can't say one way or another if the age of the downloads is relevant.

What was really odd was that when I did a manual retry of the failed RAR.r44, 2 of the 40 segments were already there. I saw this several times before - it tells me that NewsBin already downloaded those segments and put them somewhere, then failed the file anyway. So it looks like a race condition as NewsBin starts a new file set before/while completing the current file set.

[Quade]

Turned off my other servers and just downloaded 10 gigs from Astra. Variety of different file types and sets. Not one single 430 error, not one single repair was required. The files ranged in age from 3 hours to 70 days old. Going to the US server, using SSL with 10 connections.

So, this is why this won't be straightforward. It's working so well, I have no need for a fill server.

[Ribble]

Haven't seen "last downloaded file error" since I went to Build 1179.

Looks like you fixed it. Thankyou.

[Quade]

That's good to know. Thanks.