Loaded NZB files are no longer stored in loaded-folder

[BadYear]

I noticed, beginning with RC2 i think, that loaded NZB files are no longer stored in the "loaded"-folder within the watch folder. It's probably a feature since until then that loaded-folder had to be cleaned by hand frequently. Or is my installation defective?
I updated to the latest RC4 with no effect, the NZB files still disappear after they are loaded.

[Quade]

They're moved to the NZBs folder in the data folder and then eventually purged. When Sonarr sends NZBs to Newsbin they're also stored in the NZBs folder.

[BadYear]

Thank you, there i found them.

[neofit]

Hi,

I just noticed this today. I figured I had lost all processed nzbs since the 2017/12/27, and this happens to be the day I installed 680B13 over 6.73 release.

The old way:
1. Processed NZBs where I want them, outside the data\ directory, together with the watch folder, on another drive
2. the user purges them manually whenever he sees fit

The new way:
1. processed NZBs pollute the data\ folder and the c: drive
2. NB decides when I don't need these files, and currently I only have on day of those

What happened is that today I noticed that an indexing site is generating me NZbs that NB considers as malformed. I don't know who is wrong and who is right, this is another issue, but for the record NB is choking on
<!doctype nzb public "-//newzbin//dtd nzb 1.1//en" "http://www.newzbin.com/dtd/nzb/nzb-1.1.dtd">
while from NZBPlanet
<!DOCTYPE nzb PUBLIC "-//newzBin//DTD NZB 1.1//EN" "http://www.newzbin.com/DTD/nzb/nzb-1.1.dtd">
is working fine (difference in case), but this is another issue.

When NB sees a file like this, I have no message anywhere (only one line saying "Bad XML" if I turn on debug logging), and it silently moves said NZB into its new data\nzbs\ folder as if all was normal and the file has been gotten. Apparently to be purged within 24 hours. This could be quite confusing if people are building download queues from many indexers and have low download speeds (my land line only gives me 500 KB/s). Only later one may find that darnit, I am sure I asked this and that to download and there was nothing in the failed DL tab, wtf happened?

At the very least NB should keep the file it deemed bad in the "watched" directory, and display an error message even in "info" logging mode. But, closer to this thread's topic, I don't know when this started happening with the indexing site, I have probably missed a few as I will discover later, I could have figured it out with the old way of managing processed NZBs, but with the new way everything was purged automatically so I have no idea.

Is there any way this could be changed, with or without an option? At least let us decide when to purge them please.

[Quade]

When NB sees a file like this, I have no message anywhere (only one line saying "Bad XML" if I turn on debug logging),

1 - Newsbin doesn't look at the doctype . The issue is whether the NZB is valid XML or not. It sure sounds like bad NZB's to me.

The most common cause of problems loading NZB's these days is people have poster lockouts that block the posters in the NZB files. You could argue that poster lockouts shouldn't apply to NZB's I guess. You can email some of the NZB's to me if you want me to look at them. ts@newsbin.com. The way I see it, it's either good XML or it isn't.

2 - As for the location of the NZB files. The "Loaded" folder was never a feature. It was just a dumping place for NZB's Newsbin had already processed. I made the change to make NZB loading more reliable because I know I can read/write to the data folder. Where people stick the watch folders is out of my control and Newsbin may simply have no ability to do anything in those folders. All Nzb's get moved into the data folder so that Newsbin can process them.

Nothing prevents you from maintaining your own copy of the NZB's and then feeding them into the watch list. There's a hidden option in the NBI file which control the data folder purge time. If you want to change the purge time I could tell you. You could also copy the NZB folder from time to time. You might check the recycle bin for the deleted NZB's too.

I'll consider making a copy of the NZB go to the loaded folder too. I simply won't even look in the loaded folder and copying the file there won't impact loading any.

I'd upgrade to RC5 too. B13 has been long superseded.

[neofit]

I upgraded to RC5, no change.

1. When I replace the doctype line where everything is in lower case with the one with the proper capitalization, then NB processes it without a problem. When this line is in all lower case, "Bad XML". Which can make sense since the correct one refers to an URL with a Unix path having a directory name in caps.

And in any case, when there is a "Bad XML", please don't move the file silently into the data\nzbs\ directory as if its contents were processed successfully. I don't know how many people constantly monitor their logging window, I have too much stuff there for me to notice a failed file in there on a day to day basis. IMHO we still should have one line there for every failed file even in "Normal" logging mode, as well as something in the "Failed Files" tab which is much easier to monitor.

I'll send the bad file to the address provided as soon as I finish typing this.

2. I can't maintain a copy of every file manually since many come from api calls right into the watch folder, others are manually saved there from an indexer site and are processed by NB before I can do anything with them.

I'd appreciate it if you could share the name of the nzb retention delay parameter. Maybe there is one for an "archive" folder as well? I am currently seeing data\nzbs\ as well as data\nzbs\nzbs\ a copy of the same files in each.

[Quade]

I upgraded to RC5, no change.

I wasn't expecting there to be any. I just make no efforts to fix issues in old versions. If we're not on the same page, there's no point trying to trouble-shoot.

Code: Select all

The document type declaration must be placed between the XML declaration and the first element (root element) in the documentwell-formedness constraint.
The keyword DOCTYPE must be followed by the name of the root element in the XML documentvalidity constraint.
The keyword DOCTYPE must be in upper casewell-formedness constraint.


So it is bad XML. The question is whether I can work around it or not. Ideally you could ask whoever generates these NZB's to fix it.

I can't maintain a copy of every file manually since many come from api calls right into the watch folder, others are manually saved there from an indexer site and are processed by NB before I can do anything with them

You could copy them from the NZB's folder. You have 2 days before they're purged.

Code: Select all

[SETTINGS]
PurgeDays=?


In Days. Keep in mind this controls ALL purging of the data folder.

[Quade]

I have a work-around for the bad NZB's. I basically just trim the DOCTYPE line out of the NZB since it's not needed for anything. It turns out both DOCTYPE and PUBLIC are illegal XML when lower case.

It'll be in the final release. We're hoping RC5 is the last RC.

[neofit]

Maybe you could also take a look at the exception handling in this part? Because when NB is processing a file like that, it processes and files it as if were a proper file, with no error message. What should happen imho:

1. A line in the logging window, not only in debug mode but any logging mode, including "Normal"

2. A line in the "Failed Files" tab. I is probably reserved for the files that are included in the nzbs and not the nzbs themselves, but we are still facing a "file" that "failed" its processing if not download.

3. Instead of being moved into the data\nzbs\ directory, it should remain in the "watch" directory so we can find it and fix more easily. Maybe renamed to "FAILED_<orig_filename>" or something.

----
Also, for some reason, I have data\nzbs\ and data\nzbs\nzbs\ directories, with the same files in them.

[Quade]

#1 - Maybe. I'll think about it. Probably not a bad XML error but a "Nothing in NZB" error at a different level.

#2 - bad XML = not loaded. So, there's nothing to put into the failed list. You can't fail something that never loaded in the first place.

#3 - It's not loaded till after it's moved to the NZB's folder so, this probably isn't practical.

I wrote a note to look into more logging for bad NZB's.

[neofit]

#1 - Maybe. I'll think about it. Probably not a bad XML error but a "Nothing in NZB" error at a different level.

#2 - bad XML = not loaded. So, there's nothing to put into the failed list. You can't fail something that never loaded in the first place.

#3 - It's not loaded till after it's moved to the NZB's folder so, this probably isn't practical.

I wrote a note to look into more logging for bad NZB's.

Logging will not be enough. It will disappear from view too quickly.

What happens now:
1. User gets one actual bad .nzb or one that NB cannot process properly(*), in a batch of 20 for instance
2. This one out of 20 is not processed. Result:
a. Nothing is downloaded
b. At best, one line appears in the logging window in debug mode, that will be scrolled out of view by the rest of the debugging info within seconds
c. Nothing appears in the "Failed Files" tab, even though, technically, the .nzb is also a file and has failed.
d. The .nzb is moved out of the "watch" folder, and everything looks as if it had been properly processed
3. With absolutely zero feedback as to the failure of some .nzb, or, worse, the same feedback as we have for files that have been processed properly, one may not and probably will not notice that something is missing for days, weeks or months, until is is too late.

Please add something to properly report actual "bad" or presumed "bad" nzbs. Either copy them back into the watch folder, or better yet add them to the Failed Files tab, or add a "Failed nzbs" tab.

(*) NB 6.80 RC5 is unable to process any .nzb from the "a.b.newznzb" newsgroup. I downloaded Sabnzbd to check these out and it has NO issues. Again, no feedback in NB, the xml looks OK so not even a line in debug mode, nothing downloaded yet the .nzb is hidden from the watch folder. If there are .txt or .nfo files with the "main" download then they appear without the latter, otherwise nothing is shown like with the "bad XML" .nzbs. I have a bunch of those if you want to check things out.