No Such Article in Newsbin, yet SABnzb can download it

[Saskija]

I've been trying to download a show that was posted 60 days or so ago, but Newsbin doesn't download anything. Lots of red files, none of the rars will download. When I check the status I read:

[17:39:33] HIGH - Failed to retrieve Article from Server:Supernews ARTICLE <1348453050.77037.1@eu.news.astraweb.com>
| 430 no such article
[17:39:33] HIGH - Failed to retrieve Article from Server:Supernews ARTICLE <1348453027.06100.1@eu.news.astraweb.com>
| 430 no such article
[17:39:33] HIGH - Failed to retrieve Article from Server:Supernews ARTICLE <1348453427.04971.1@eu.news.astraweb.com>
| 430 no such article
[17:39:34] HIGH - Failed to retrieve Article from Server:Supernews ARTICLE <1348453427.04800.1@eu.news.astraweb.com>
| 430 no such article

Yet when I load the same NZB file in SABnzb the files will download and I get an extracted MKV media file. How is this possible? I'm using the same news server config (Supernews) for both programs.

[Quade]

Hmm don't know. Want to email me the nzb? ts@newsbin.com. Don't zip it but, RAR is ok. We have a zip filter in the mail server.

Wonder if it's luck of the draw? Like one time you get on one server that can't deliver it and then sab, connects to a different server that can (within the usenetserver farm).

[Saskija]

I sent you the file

[Quade]

Got it thanks. I'll check it out.

[Quade]

Sab reported the same missing articles Newsbin did.

2012-11-24 21:52:51,516::INFO::[downloader:591] Thread 5@news.giganews.com:563: Article 1348453028.15812.1@eu.news.astraweb.com missing (error=430)
2012-11-24 21:52:51,516::INFO::[decoder:218] <Article: article=1348453028.15812.1@eu.news.astraweb.com, bytes=787037, partnum=1, art_id=None> => missing from all servers, discarding

But, I did have to manually repair the files in Newsbin so, it's something I need to look at. The file was old school DMCA'd. The first chunk of each file was missing. Interestingly AW had the missing parts.

Thanks, I'll check it out.

[Saskija]

Will you implement a change so Newsbin will find and download DMCA'ed files where only the first chunk is gone and that can still be downloaded? I've found a few more NZBs that Newsbin won't download but can be completed using SABnzbd.

[Quade]

Yeah, it's fixed. Will be in the next beta.

[Saskija]

Great service!

[Quade]

I'm glad you reported this because I missed that it was broken.

Basically what happens with DMCA posts is that the smallest par file is purged. What's supposed to happen is that after 2 retries, it'll just fail that file and go into DMCA mode which is "Download everything" mode. What wasn't happening was the failing of the smallest par so, it'd get stuck.

[Plankton]

rezzer reported this in the Astraweb section ages ago.

[ppan]

Was this ever fixed? If so, in what version? I am using 6.41B4 and it keeps trying with error message 430 w/o advancing down the Download List.

[Quade]

Yeah, fixed in B4 and now RC1. It depends on how many retries you have set though. After the second retry (if you have retries set to 2), it'll assemble and try to repair the file. This assumes you have PARS and enough of the files to repair. If you don't have pars or 90% of the files in the set already downloaded, it'll just sit there waiting for you to do something. If you have autopar disabled, I think it becomes purely a manual process (right click, assemble incompletes).

I was testing this yesterday with Giganews delivering a bunch of incomplete headers. It assembled and repaired all the files.

[ppan]

I upgraded to 6.41RC1, but the problem remains (Retry is set to 2; posts are old).
Nothing is being downloaded in my download list anymore (speed < 1 KBps), the log now just shows several thousand entries like:

[17:40:31] HIGH - Open Post - news.supernews.com ARTICLE <**********@**********> | 430 no such article
[17:40:31] HIGH - Failed to retrieve Article from Server:news.supernews.com ARTICLE <**********@**********> | 430 no such article

If I look at particular post, the rar files show several missing blocks and the PAR2 files are paused.
Status shows: [PAR: 12 of 309] D:2 DL:16 Inc:0 P:0 R:13 N:1482,None,5/1/2010 11:17
where N keeps changing (currently counting down)
[what do the the different parameters in the Status mean?]

Furthermore, 1,610 of the 2,115 downloaded files are nb2 files (not properly renamed)!

[Quade]

Furthermore, 1,610 of the 2,115 downloaded files are nb2 files (not properly renamed)!

Sounds like nothing is complete. Maybe there's nothing else to download at this point. Sounds to me like you have a more serious problem than just 430 errors. Why aren't your files finishing? Are they all incomplete? These numbers mean you have a majority of partial files. You probably need to examine the files more closely and see what's up. The first file set you're looking for, little has downloaded complete.

[PAR: 12 of 309] D:2 DL:16 Inc:0 P:0 R:13 N:1482,None,5/1/2010 11:17

You got 2 downloaded files out of 18. The rest aren't downloaded (DL: Downloading). You've had 13 retries, the next one will be in 1482 seconds. That means everything else below it is getting attention by the downloaders. If they're all as messed up as this first set, I'd say you've got very little that can successfully be downloaded. You could try assembling the RARS in this set by right clicking and "Assemble Incompletes" on the files but, Newsbin doesn't think there is enough data available to finish the download. Newsbin is pretty conservative. If the set seems to have more then typical number of PARS it might still repair. As you assemble the files, you should see the par blocks going up.

If you don't have 90% of the files available, they're probably not going to get repaired. If this is headers, I'd download headers and see if more records show up in the download. If this is NZB's, you probably want to delete them and find something else to download.

If the blocks indicate the files are completed or have turned gray (meaning assembled) and you only have 12 out of 309 par blocks, I'd say this download is unfixable.

[ppan]

I have several thousand files in my download list but nothing else is downloading as all threads are stuck on retrying. Isn't there some way to tell Newsbin to stop retrying the problematic posts and move down the list?
I resorted (by name instead of date) and then the download continued again.

Should I E-mail you the group name and dates?
I guess that I could contact Supernews and ask why so many (old) files are incomplete? Their retention is supposed to easily cover the current file dates.

[Quade]

The fact they're all retrying means Newsbin has ALREADY moved down the list. All of your NB2 files are telling you the same thing. It's gone through, downloaded what it could, assembled what complete files it found and now doesn't know what to do. It looks like you're trying to download files the server can't hand you complete copies of.

If the files are really old then that's likely your problem. The older the file on Usenet, the less likely it is to download. I never download near the edge of retention, it's a crap shoot.

What happens when you add something much more recent to the top of the download list?

[ppan]

The fact they're all retrying means Newsbin has ALREADY moved down the list. All of your NB2 files are telling you the same thing. It's gone through, downloaded what it could, assembled what complete files it found and now doesn't know what to do. It looks like you're trying to download files the server can't hand you complete copies of.

If the files are really old then that's likely your problem. The older the file on Usenet, the less likely it is to download. I never download near the edge of retention, it's a crap shoot.

What happens when you add something much more recent to the top of the download list?

As I mentioned above, if I resort the list (e.g. according to name rather than date), it started downloading again. Thus, Newsbin hasn't downloaded all the other posts yet.

[Quade]

You've got 1200 failing/partial downloads at the top of the list. That's your main problem. The rest is just a sideshow. Until you solve that, I'm not really surprised at weird results. It takes work to work your way through all the failed downloads you've queued to the remaining sets, which might also be screwed up.

I think solving the main issue will make this a moot point.

[ppan]

The Usenet provider wanted me to send them the message IDs of the failing/incomplete posts. I guess that I can just copy those from the log tab?
Is it safe to send this to them? I am not sure if they are incomplete because of DMCA takedowns?

[Quade]

Might be better to pick the top one and generate an NZB from it. Then send them the NZB

If you want to email me the NZB too, I could check it out too. ts@newsbin.com.

Right click menu in the download list lets you gen the NZB.

[Quade]

Ok, got an NZB from PPan. It's not a group that see's DMCA takedowns.

Giganews was missing a bunch of the file chunks too (Giganews = Supernews as you pointed out). Astraweb has the missing chunks so I was able to get a clean download by combining AW and Giga together. I'd say this is just "Usenet". The server simply didn't have the posts.

[ppan]

Ok, got an NZB from PPan. It's not a group that see's DMCA takedowns.

Giganews was missing a bunch of the file chunks too (Giganews = Supernews as you pointed out). Astraweb has the missing chunks so I was able to get a clean download by combining AW and Giga together. I'd say this is just "Usenet". The server simply didn't have the posts.

The strange thing is that file chunks are missing from several thousand other posts as well, probably originally posted to many different news server.

BTW, how can I find out which news server got the original postings?

[Quade]

You can't always. If you look at the NZB and see message-id's with a server name in them, that's typically the server they were posted to.

Rule of thumb is, the older the post, the harder it is to download.

[ppan]

Ok, got an NZB from PPan. It's not a group that see's DMCA takedowns.

Giganews was missing a bunch of the file chunks too (Giganews = Supernews as you pointed out). Astraweb has the missing chunks so I was able to get a clean download by combining AW and Giga together. I'd say this is just "Usenet". The server simply didn't have the posts.

I sent SuperNews the NZB and part of Newsbin log file showing a couple of thousand failed message IDs.
Here is their response:

Thanks, I've checked this against our servers and found that a number of articles have indeed been removed. They show as being removed due to DMCA complaints we've received, so there is no way to recover these articles. If you manage to find a newer copy of this content, it would be more likely to have all of the articles intact.

Not sure if this is an excuse for them not having the posts or if it is correct.....? It also seems unlikely that almost all post were subject to DMCA complaints? Furthermore, I thought that Astraweb had even more DMCA complaints?

[Quade]

Well, they can tell you the posts were missing and they don't know why or that the posts are missing and it was because of the evil content owners. I know which excuse I'd choose. It looked nothing like a DMCA takedown and the group contents aren't typically something that gets taken down.