B10: Downloading many small files

[g51]

Adding many (>1000) files from a group to the download list. File sizes vary from <1KB to >1MB. After adding them, click on the "size" column of the download list so that they are sorted by size (ascending). NB starts downloading but many of the small files (less than a certain size, i'm not sure which is it though) end up to the "Failed" tab for some reason. If at some point later though, the files in the "Failed" tab are re-added to the download list and the list is re-sorted by size, all of them are downloaded normally, without any problems! It seems that downloading many small files directly from a group causes an issue.

[Quade]

They're probably duplicates. Check the logging tab for errors.

[g51]

They're probably duplicates. Check the logging tab for errors.

The logging tab wasn't much help. It would be a great help if the "failed" tab had a "reason" column. I'll try again later today to see if i can dig-out anything.

I doubt they were duplicates though. The download folder was empty and i selected all headers from a single large post. I had noticed this behavior with B9 too. It seems that some tasks fail for a reason which is not clear to me and if re-inserted in the download list, they are downloaded normally.

[g51]

Today i faced a similar situation. Here's part of the log:

Code: Select all

[11:41:31] HIGH   news.<my.usenet.provider>.com  - ARTICLE <4d9ac66a$0$58248$a82e5ee0@news.premium-news.net>
[11:41:31] HIGH   news.<my.usenet.provider>.com  - ARTICLE <4d9ac66c$0$58248$a82e5ee0@news.premium-news.net>
[11:41:31] HIGH   news.<my.usenet.provider>.com  - 220 0 <4d9ac666$0$58248$a82e5ee0@news.premium-news.net>
[11:41:31] HIGH   news.<my.usenet.provider>.com  - 503 Backend Error 0xc0edbee1 -- Please retry later
[11:41:31] ERROR  news.<my.usenet.provider>.com  - Socket HandleError -  Server: news.<my.usenet.provider>.com SendCommand - Permenant Server Failure:  503 Backend Error 0xc0edbee1 -- Please retry later
[11:41:31] HIGH   news.<my.usenet.provider>.com  - GROUP <my.usenet.group>
[11:41:31] ERROR  Socket  - Error: WSASend Socket Error Code: %u:%u 10038 812
[11:41:31] ERROR  news.<my.usenet.provider>.com  - SendCommand Worker -  Server: news.<my.usenet.provider>.com SendCommand - failed to send command to server:  Unspecified Network Error
[11:41:31] ERROR  news.<my.usenet.provider>.com  - Socket HandleError -  Server: news.<my.usenet.provider>.com SendCommand - Server Disabled Service on connection: 
[11:41:31] ERROR  Download Engine <my.usenet.provider>  - DL_OpenItem - DL_DI - Failed to retrieve Article from Server:: <my.usenet.provider>
[11:41:31] HIGH   news.<my.usenet.provider>.com  - 503 Backend Error 0xc0edbee1 -- Please retry later
[11:41:31] ERROR  news.<my.usenet.provider>.com  - Socket HandleError -  Server: news.<my.usenet.provider>.com SendCommand - Permenant Server Failure:  503 Backend Error 0xc0edbee1 -- Please retry later
[11:41:31] HIGH   news.<my.usenet.provider>.com  - GROUP <my.usenet.group>
[11:41:31] ERROR  Socket  - Error: WSASend Socket Error Code: %u:%u 10038 812
[11:41:31] ERROR  news.<my.usenet.provider>.com  - SendCommand Worker -  Server: news.<my.usenet.provider>.com SendCommand - failed to send command to server:  Unspecified Network Error
[11:41:31] ERROR  news.<my.usenet.provider>.com  - Socket HandleError -  Server: news.<my.usenet.provider>.com SendCommand - Server Disabled Service on connection: 
[11:41:31] ERROR  Download Engine <my.usenet.provider>  - DL_OpenItem - DL_DI - Failed to retrieve Article from Server:: <my.usenet.provider>
[11:41:31] HIGH   news.<my.usenet.provider>.com  - ARTICLE <4d9ac66e$0$58248$a82e5ee0@news.premium-news.net>
[11:41:31] HIGH   news.<my.usenet.provider>.com  - 503 Backend Error 0xc0edbee1 -- Please retry later
[11:41:31] ERROR  news.<my.usenet.provider>.com  - Socket HandleError -  Server: news.<my.usenet.provider>.com SendCommand - Permenant Server Failure:  503 Backend Error 0xc0edbee1 -- Please retry later
[11:41:31] HIGH   news.<my.usenet.provider>.com  - ARTICLE <4d9ac66c$0$58248$a82e5ee0@news.premium-news.net>
[11:41:31] ERROR  Socket  - Error: WSASend Socket Error Code: %u:%u 10038 812
[11:41:31] ERROR  news.<my.usenet.provider>.com  - SendCommand Worker -  Server: news.<my.usenet.provider>.com SendCommand - failed to send command to server:  Unspecified Network Error
[11:41:31] ERROR  news.<my.usenet.provider>.com  - Socket HandleError -  Server: news.<my.usenet.provider>.com SendCommand - Server Disabled Service on connection: 
[11:41:32] HIGH   news.<my.usenet.provider>.com  - ARTICLE <4d9ac670$0$58248$a82e5ee0@news.premium-news.net>
[11:41:32] HIGH   news.<my.usenet.provider>.com  - ARTICLE <4d9ac680$0$58248$a82e5ee0@news.premium-news.net>
[11:41:32] HIGH   news.<my.usenet.provider>.com  - 220 0 <4d9ac66e$0$58248$a82e5ee0@news.premium-news.net>
[11:41:32] HIGH   news.<my.usenet.provider>.com  - ARTICLE <4d9ac685$0$58248$a82e5ee0@news.premium-news.net>
[11:41:32] HIGH   news.<my.usenet.provider>.com  - 220 0 <4d9ac670$0$58248$a82e5ee0@news.premium-news.net>
[11:41:32] HIGH   news.<my.usenet.provider>.com  - 503 Backend Error 0xc0edbee1 -- Please retry later
[11:41:32] ERROR  news.<my.usenet.provider>.com  - Socket HandleError -  Server: news.<my.usenet.provider>.com SendCommand - Permenant Server Failure:  503 Backend Error 0xc0edbee1 -- Please retry later
[11:41:32] HIGH   news.<my.usenet.provider>.com  - GROUP <my.usenet.group>
[11:41:32] ERROR  Socket  - Error: WSASend Socket Error Code: %u:%u 10038 812
[11:41:32] ERROR  news.<my.usenet.provider>.com  - SendCommand Worker -  Server: news.<my.usenet.provider>.com SendCommand - failed to send command to server:  Unspecified Network Error
[11:41:32] ERROR  news.<my.usenet.provider>.com  - Socket HandleError -  Server: news.<my.usenet.provider>.com SendCommand - Server Disabled Service on connection: 
[11:41:32] ERROR  Download Engine <my.usenet.provider>  - DL_OpenItem - DL_DI - Failed to retrieve Article from Server:: <my.usenet.provider>
[11:41:32] HIGH   news.<my.usenet.provider>.com  - 220 0 <4d9ac685$0$58248$a82e5ee0@news.premium-news.net>
[11:41:32] HIGH   news.<my.usenet.provider>.com  - MODE READER
[11:41:32] HIGH   news.<my.usenet.provider>.com  - MODE READER
[11:41:33] HIGH   news.<my.usenet.provider>.com  - MODE READER

It seems its getting an error 503 and then the connection is somehow lost. I also keep getting in the log the following lines every 12":

Code: Select all

[11:50:41] HIGH    - Bad XML Line: 1 Column:  0
[11:50:41] HIGH    - Bad XML Line: 1 Column:  0
[11:50:41] HIGH    - Bad XML Line: 3 Column:  0

[g51]

OK, it seems there's a pattern. If i send 10 posts (not very big, about 200K each) from a group to the download list, only 2-3 are downloaded directly, all others fail. If i place them to the wishlist and then send them to the download list, the same thing happens. However, if after they fail i re-send them to the download list, none fails. The recurring "Bad XML Line" log message makes me suspect it's an NB issue and not a usenet provider one but there's no way to be sure...

[itimpi]

The 503 errors are definitely problems at the server end. It is the server that is returning that error code.

Not sure about the XML errors - I do not see this behavior so cannot really comment.

[richy99]

would indicate its your server giving the errors, who do you use?

[g51]

Easynews. I contacted them but i'm not sure this is the problem.

I enabled "debug logging" and i now see the following message several times in the log:

LOW - Handle Chunk - Duplicate Detector Match

However, the files i'm trying to download are not duplicates. If this was the cause they are put in the "failed" tab, then when i re-inserted them from the "failed" tab in the download list, they should fail again, shouldn't they?

[Quade]

1 - 503 errors are from the server end and have nothing to do with Newsbin (other than causing failed downloads).

2 - XML errors are probably Newsbin scanning the NZB autoload folder.

3 - With the duplicates, it's either a real dup (or common prefix) or a false positive. False positives are rare but, possible. The dup checker is checking the first 23 K of the file. Files with identical prefixes will get duped out.

[g51]

2 - XML errors are probably Newsbin scanning the NZB autoload folder.

I see. Are those log messages really needed? Please remove them if possible.

3 - With the duplicates, it's either a real dup (or common prefix) or a false positive. False positives are rare but, possible. The dup checker is checking the first 23 K of the file. Files with identical prefixes will get duped out.

Yup, that was it. No more "failed" items after disabling the duplicate detector! Thanks. I still believe it would be nice though if a "reason" column existed in the log with this information (even in normal log mode).