Started having problem with incomplete downloads

[TaintedShirt]

I am not sure if this is related to updating to v6 or not, but it seems to have happened since I updated.

Lots of posts (but not all) seem to be missing the very first block. So if there are 30 rar files I am missing a block from each and am missing the .nfo, srr and sfv files.

It's really been driving me nuts. I doesn't seem to happen with every post just some posts. grrrrr.

I am using Win 7, the latest beta of Newsbin Pro and have KIS 2012 installed. Newsbin has already been added to KIS exclusions. I use Newshosting on the XL package. I have had to download files via it's easynews deal just to finish off downloads.


The error message I get is :
[18:53:10] ERROR NNTPSocket - Socket HandleError - Server:nhxl.newshosting.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[18:53:11] ERROR NNTPSocket - Socket HandleError - Server:nhxl.newshosting.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[18:53:11] ERROR Download Engine NHXL - INT_OpenItem -DL_DI - Failed to retrieve Article from Server::NHXL


Anyone got any ideas on what I should try?


edit forgot to say I am using Win 7 64bit.

[Quade]

Get a better news server? Download newer files?

430 errors are from the server saying "I don't have part of the file". It you're using NZB's, and the files are real new, it's possible they're not on the server yet but, eventually will be. If they're older files, nothing can fix this except for many another, better server you use to fill in the blanks.

Lots of posts (but not all) seem to be missing the very first block.

On Giganews, missing the first block is normally "Dmca takedown notice" meaning they were legally required to remove those chunks from the server. I don't know if Newshosting actually does DMCA takedowns though.

[TaintedShirt]

Quade, I think you've hit the nail on the head. I think newshosting must be doing the same. Checking through the problem files, they all originate from the same company.

I also noticed in the logging "[19:33:12] HIGH NNTPSocket - ARTICLE <XXX.XXX.X@eu.news.astraweb.com>"

Why would astraweb be mentioned as I don't have an account with them or a server for them set up?


Thanks Quade for your reply.

[Quade]

The files were posted through astraweb so, they have an Astraweb message-id. It doesn't really mean anything.

[TaintedShirt]

thanks for clearing that up Quade.

ps am loving v6

[thebigtexan]

Hmmmm... I've been having this problem as well (also on newshosting), but it is inconsistent. I can download one post, say an IMMERSE upload of episode 4 of "some show" with no trouble, but episode 5, from the same poster and on the same NG, is mangled as described by the OP. I'm wondering why newshosting would enact a dmca takedown on one post, but not another. I assume the answer is they do so only by request, but that leads us to ask why the owner of the copyrighted material would only request that one be taken down and not the other, especially when they were both posted some time ago, so are not new. I would think that the copyright owner, if they were concerned at all, would be on top of each post so they could make their request ASAP.

Any thoughts?

[Quade]

On Giganews, a DMCA takedown is easy to ID. The first chunk of every file is missing. I don't know on other servers. I'd expect it to be similar. If you're just missing random chunks of the file, the chunks may have simply never been on this server in the first place. Server quality comes and goes, sometimes there are issues that damage the files that never get resolved as time passes but, the server recovers and newer files are just fine.

It's a reason many people use a backup server. Damage that affects one server might not affect another so, the missing chunks can be downloaded from the backup server. Then you have to factor in PAR files. In many cases there are enough PAR files to repair the damaged file so, it's not an issue at all.

In the latest 6.01B6, you can "test download" the files. It checks the quality and tells you how much of the file is donloadable. It might be worth "test downloading" and then adding a backup server into the mix and seeing how much of the file you can download. If you're missing more than 10% you'll want to consider not downloading it. Depending on the number of pars you have.

[haileris]

The trouble I have with incomplete downloads is that the logging tab fills up with 403 messages (gets to around 3.8k of them) and then Newsbin tends to just give up completely, going to zero download speed. If you place a download above this file or alternatively try a new download like a group list update things will start up again. However, for those of us with "timed" internet connections it can "waste" a nights download plans. I could be doing something wrong of course, just not sure what.

[Quade]

Sounds like you might want to test the downloads before you add them? Testing doesn't require much bandwidth so, you can do it during peak hours.

The latest 6 is much more aggressive about assembling partial files so, the problem might already be solved. I've also extended how far down the list Newsbin will try to download from.

[haileris]

Just updated to the latest release so we will see what happens - thanks

[-pk-]

The only problem I've noticed with the files missing the first chunk, is that auto assemble is not zero-padding the missing chunks. To recover most of the sets I've been downloading they only need about 9 more blocks. For me that is 40MB through p2p if they were padded properly for file hashing, or 139MB (9 * 15mb files) to completely redownload these files with misaligned chunks. Eitherway it doesn't affect par hashing because quickpar is way more intelligent than p2p hashing.

I made a batch process for padding the missing chunks, but that's only going to work for the first chunk..

[DThor]

That...doesn't sound right. Newsbin downloads and saves a file with the first chunk missing, then if there's enough pars, it will download enough pars to repair it. I get what you're saying, but it's not in line with what I've experienced. If I follow you, and 10 files had missing first chunks, you're suggesting Newsbin redownloads all those files. Doesn't happen. What am I missing?

DT

[Quade]

He's suggesting Newsbin leave empty space for the first missing blocks instead of packing all the blocks together. I'm not sure I'd want to build in the padding for normal downloads, I could use the par files to generate padded files but, I don't see a reason to do this automatically. It'll just slow down downloads for everyone else. I can consider a "pad the files" command. Basically re-write the files on par block boundaries.

Why not just use PAR files to fix them?

Thinking about this. If in the NBI file, you add this option.

[SETTINGS]
MemCacheLimit=0

I believe it'll pad the files.

[-pk-]

I added MemCacheLimit=0 and restarted newsbin, but the files came out the same size.

The problem with these episodes is that they don't include enough par2 recovery blocks to repair 1 block missing from each file. I end up with 33/42 blocks available for repair.

I have to pad the missing chunks to get the data to line up with p2p hashes, so that I can download the missing pieces over p2p. I only download enough pieces to cover enough blocks to start the PAR repair (N+1 blocks). So if I'm missing 9 blocks to repair the set, I can download any 10 pieces if properly aligned.

The obvious downside is that p2p pieces are usually 2MB - 4MB. This ends up being 10 times larger to download the missing pieces than if they would have included more par2 recovery blocks to begin with. I'm sure most people just download the next higher quality episode, but that too might have missing chunks. And I'd feel bad for the lost bandwidth if I were to download a 2GB, 720p episode just because I needed another 3.5MB for par repair.

[Badcar]

It's a reason many people use a backup server. Damage that affects one server might not affect another so, the missing chunks can be downloaded from the backup server.

I just wanted to quote that.

[fossi]

would be great if we had an option to pad the missing parts.
in case of incompletes where the provided par set does not provide enough blocks this would allow an easy/economic way to grab the missing parts via a torrent.
without padding, a 80% complete (par2 wise) download can result in a meager 4% complete torrent

[Quade]

The whole mechanism has changed since the last post on this topic. Newsbin doesn't write to NB2 files during download anymore. Instead the chunks are cached in ram and/or flushed to disk depending on whether there's space in RAM or not. Then the chunks are assembled into the file. What this means though is that padding is relatively easy now.

I'll add a note. The padding will work as long as the prior chunk size = pad. That's true 99% of the time but, isn't always.