Version 6.40 Beta 1 Available...

[dexter]

We are ready to start the public beta test of 6.40! It has been available on our IRC channel for our internal testers for a couple of weeks now so most the glaring issues have already been identified and resolved.

The biggest new feature in this release are the NZB Watch Lists. You can create an NZB Autoload Folder in the Groups tab now, and apply a distinct download and unrar path for NZB files processed out of each folder. For example, you might have different folders for Images, Music, and Multimedia, and you want the processed files to show up in different folders to keep them better organized.

Summary:

• NZB Watch Lists
• AutoPAR improvements
• PAR state now shown in Download List (Status column, says "x or y PAR blocks"
• Submitting an empty Internet Search returns the most recent 2000 items added to the search database
• Improvements to the Spam filtering
• MOTD now doubles as the splash screen for unregistered users.

More details and the download link are on the Newsbin Beta Page.

[Quade]

This removes the "Retry X of X" that people all seem to hate too.

[Semel]

The biggest new feature in this release are the NZB Watch Lists.

thanx a lot. this will come in handy for sure.

[Quade]

thanx a lot. this will come in handy for sure.

Person on IRC requested it so, I put it in.

[Semel]

Person on IRC requested it so, I put it in.

So all it takes for a feature to be implemented is to be on IRC?

i remember requesting something similar on the forums( improved watch folder a la sabnzbd)

anyways, my thanks goes to this anonymous person as well =)

[Quade]

You request stuff, then when I implement it, you ask me how to then disable the new features. It's made me a bit skeptical of your feature requests.

Multiple people are asking to be able to disable simultaneous unrar and download so, it's on my list.

[alanstarr]

I think I found a small bug. When clicking on the 'From' column, the posts are grouped together correctly, but its not sorted alphabetically.

(Windows 7, 64-bit version)

[Quade]

I'll check it out. Thanks.

[tl]

I think I found a small bug. When clicking on the 'From' column, the posts are grouped together correctly, but its not sorted alphabetically.
(Windows 7, 64-bit version)

FYI, confirmed on Windows XP, 32-bit, I see exactly the same thing there.

[re838uk]

I can't seem to get anything to happen when downloading to a new folder (via right click menu).

[Quade]

Thanks. "Download to Existing" seems to work.

[ppan]

Why are the par2 files grouped together separately from the rar files in this version?

[DThor]

There have always been posts like that. Depends on how they're posted. Unless I misunderstood your meaning.

DT

[ppan]

There have always been posts like that. Depends on how they're posted. Unless I misunderstood your meaning.

DT

I thought that they usually were grouped together for a particular set; rar files w. par2 files?
When I clicked on the '+', I saw the rar files followed by the par2 files.

Now they are two separate groups; clicking on the '+' in group 1 shows all par2 files and clicking on the '+' in group 2 shows all rar files

[DThor]

They have always been there, nothing has changed afaik. Apart from some oddball groups, you should find most compress properly. It's related to what's being posted, and how.

DT

[Quade]

Example:

- Posted by the same guy, same time, format of the subject is totally different, they won't compact together.

bla bla UK chan4-DVDrip S01-E01 [01/11] - "bla bla_S01E01.mp4.part01.rar" - 01/11 yEnc
(01/10) "bla bla_S01E02.mp4.par2" - 53,42 MB - yEnc

- Not using RARS

Something [2 of 9] "Something s01e05.Something.ws.dsr.x264-cldd.mp4.par2" yEnc
Something [1 of 9] "Something s01e05.Something.ws.dsr.x264-cldd.mp4" yEnc

There's lots of reasons it might not compact. Not following the "Standard" is the normal way.

[ppan]

I still have several issues with Downloading/AUTOPAR on some sets.
Example 1:

1. Post Window shows that all rar files have been downloaded. However, several rar files are missing on disk and some are incomplete (missing files verified with Autopar properties).
2. Only one par2 file was downloaded; 11 Par files paused ("Don't Pause PAR" files is unchecked).
3. Set is not repaired and unrared ("nothing is happening")


A. What could have happened to the missing rar files?
B. Why doesn't Newsbin start to download the rest of the Par files in attempts to repair?
C. How can I troubleshoot what is happening (or rather why nothing is happening)?

[DThor]

You are always going to find *something* that's is posted in an odd manner that can befuddle the compressor. Downloads should work properly, though, true. I've found that most grief seems to happen with users that feel the need to alter the autopar settings, but yes, they should work in a predictable manner of course. The moment you see odd behaviour like this, it's logging tab, logging tab, logging tab... That tells you important information so debugging doesn't involve tail chasing. Like, a post missing on the server, a problem saving to disk, etc.

Also, any other relevant information such as restarts during this download, remote filesystems, etc. Anything that if you take a step back and think about for a moment, you say to yourself 'hmm, that should work, but might be related'.

DT

[Quade]

3. Set is not repaired and unrared ("nothing is happening")

Hit the plus icon and look at the files. The ones that are missing from disk are probably currently incomplete. I'd watch for errors in the logging tab too.

[ppan]

3. Set is not repaired and unrared ("nothing is happening")

Hit the plus icon and look at the files. The ones that are missing from disk are probably currently incomplete. I'd watch for errors in the logging tab too.

There are both files missing and incomplete. Why does Newsbin not start to download the Par files to try to repair? Isn't that the whole point; the par files are paused until needed?

[DThor]

It won't do that if it determines there are insufficient pars to repair the post.

DT

[Quade]

Lots of data there to look at ppan. How many par blocks you have, how many you need. Unless you look at this data, I can't even guess why this specific download is causing you problems.

Incomplete files that aren't assembled contribute no blocks to the download total. By default, after the second retry and if it looks like 90% of the file has downloaded, it'll automatically assemble the files. If you have files that haven't assembled yet, Newsbin won't do anything till they're assembled, either manually or automatically by Newsbin timing out.

[ppan]

Lots of data there to look at ppan. How many par blocks you have, how many you need. Unless you look at this data, I can't even guess why this specific download is causing you problems.

Incomplete files that aren't assembled contribute no blocks to the download total. By default, after the second retry and if it looks like 90% of the file has downloaded, it'll automatically assemble the files. If you have files that haven't assembled yet, Newsbin won't do anything till they're assembled, either manually or automatically by Newsbin timing out.

Well for this particular example,I tried to re-download the incomplete files and at some point the par files were now downloaded. However, in the end, I am still missing 59 blocks so repair was not possible. Not sure what triggered the sudden par file download?

Example2; In this case, a few rar files in the set were never downloaded. They show as Idle/old in the post list. I had to manually download those files. I wonder why they were not downloaded when I selected the whole set to be downloaded?

[Quade]

I wonder why they were not downloaded when I selected the whole set to be downloaded?

Yeah me too. There's no way for me to tell. You on the other hand can look in the logging tab for anything that mentions these files.

[ppan]

BTW, I understand that the way Autopar works is constantly changing. However, it would be very helpful (e.g. when v6.40 final is released) if the complete logic (step-by-step) for how Autopar works could be explained, including the different Autopar options.

[ppan]

I wonder why they were not downloaded when I selected the whole set to be downloaded?

Yeah me too. There's no way for me to tell. You on the other hand can look in the logging tab for anything that mentions these files.

Those files were supposedly downloaded yesterday, but it appears that only the most recent log entries are shown? The logs show no date, but only times are shown, and they are all from within the last hour (I have not closed Newsbin for > 48 hrs). Is there some way of keeping more log entries and to also show the date?

p.s. Is there a way of searching the logs (I copy and paste to Notepad++ to search)?

[DThor]

I'll repeat what I mentioned before, you really should just leave things at their default. The reason the options exist are primarily for users that hate hands-off behaviour and can't do without micro-managing their downloads (which I don't think is you) , aggressive aside. Aggressive is for the impatient who have very reliable servers (which based on overnight downloads probably isn't you either) . What seems to be happening is you're coming across posts that are either incomplete or posted in an unusual manner and you're assuming it's time to start fiddling with the options. All it tends to do is cause tail chasing. Instead, run with the defaults, and when something happens that you feel is odd, look in the logging tab. When a post gets 'stuck' in the download queue I would say 9 times out of 10 it's very straightforward : there aren't enough chunks and pars to complete the post. That's it, and you can see this in the status column.

Of course, it's possible there's something else at play here, but without more data, it's hard to say. If you have a post that somehow fails, and it's not too huge, you can always just re download it alone and watch it, that gives you your logging tab data.

DT

[Moondawgie]

There's lots of reasons it might not compact. Not following the "Standard" is the normal way.

Is there a formal, written definition of the "Standard" best recognized by Newsbin?

[Quade]

There's no "Newsbin" standard. Newsbin tracks the standard way that other people post. For 95% of the downloads, Newsbin compacts them properly.

If you use yenc with properly quoted filenames
if you use RARS or QPMS (.001) format
if the pars use the same format as the rars

Newsbin will probably compact the posts. Probably because it's not 100% ever. Most of the posting robots post in consistent compact-able patterns. It's the individuals that post that seem to have issues setting up the posting software.

[rmason]

I'm downloading a large file, and like many before it, the file will not likely need Pars; it will be error-free. But there's a gotcha. Any time I have to restart Newsbin, and I have a partially downloaded RAR-Part (in the form of an NB2), the RAR-Part will contain errors when its download is complete. It's been this way for many moons, so I've made it a practice to delete any NB2s before restarting, thus pretty much guaranteeing a clean Part.

With 6.40 beta, this doesn't work. The "deleted" NB2 remains. I suspect it's because the nb2 is now in memory as well as on disk. I hate having to reboot my computer (a 20 minute process) every time I restart Newsbin. For now, it's back to 6.3. Is there a workaround for this change?

[Quade]

I've tried to make this failure happen and it simply doesn't for me. How big are the individual files? How many servers do you use? Do you see any errors on restart?

What kind of machine is this?

In Newsbin before you exit, in the options, click "Open Data Folder". Now exit Newsbin. Look in the temp folder. You should see some nb2 files listed. Delete those to make it start the partial download over from scratch. What changed is where I store the information about the partial downloads. You should be able to leave the NB2 files in the download folder and they should just get re-used.

Even if Newsbin kept some of it in memory, once Newsbin exits, any in memory data is gone. So, if you don't see Newsbin listed in task manager, you know that the persistent information HAS to be on disk.

[rmason]

<How big are the individual files?> Sometimes a little over 50GB. The "Parts" are typically 200-500MB each.
<How many servers do you use?> One. It's a fairly well known, decent pay server.
<Do you see any errors on restart?> Nope
<What kind of machine is this?> Winders XP (SP3)

<In Newsbin before you exit, in the options, click "Open Data Folder". Now exit Newsbin. Look in the temp folder. You should see some nb2 files listed. Delete those to make it start the partial download over from scratch.>

That's basically what I've been doing, except I have been opening the data folder using Windows Exploder after closing Newsbin.. I delete the nb2 files, then restart Newsbin. At that point the Parts corresponding to the nb2s I've deleted show as partially downloaded. Before 6.4, they would be virgins. When I went back to 6.31 this is what happened...fresh parts ready to start downloading.

That's weird that you're not seeing this, especially in light of what you're putting in memory (I had thought it was the whole nb2s). I'll go back to 6.4 and keep an eye on things and let you know if I see anything interesting.

Thanks for the quick get-back, Quaid.

[Quade]

That's basically what I've been doing, except I have been opening the data folder using Windows Exploder after closing Newsbin.. I delete the nb2 files, then restart Newsbin. At that point the Parts corresponding to the nb2s I've deleted show as partially downloaded. Before 6.4, they would be virgins. When I went back to 6.31 this is what happened...fresh parts ready to start downloading.

I'll bet you're deleting them from the DOWNLOAD folder. I'm suggesting you delete the ones in the DATA\temp\ folder.

By default, it only puts the entire NB2 in memory if you set some hidden options. If you haven't done that then, it's writing to the NB2 during download. It's easy enough to confirm this one way or the other. Do some downloads and see if the NB2 is being written to during download. Since it doesn't fail for me. I'd like to give you test versions to try different fixes for this. To try to get to the bottom of it. That fact you're running XP might be related to this and might explain why it's not happening to me. I've noticed that thread timing is different between XP and Win7.

I have a version here for example that writes every binary block out to disk during download. It's a bit slower but, it would be interesting to see if that works for you. The NB2's aren't used at all there.

So, if you're game, I'll contact you when I have another version to try so, we can get some testing in. What I suspect is happening is timing. On exit, my version is able to save off the updated data but, because the exit timing is different for XP or CPU speed or something like that, you don't get a complete partial save.

[rmason]

Indeed I was deleting from the DL folder. Next time I see the prob., I'll check the Data folder. If that is where the nb2s are hiding, my problem is solved. But I'd be quite happy to play test guinea pig for you guys. I've done that with others and have always enjoyed. If the "no nb2" method eliminates the errors I always see in half-finished parts after a restart, this would be a fairly big deal in my life. (I obviously don't really have one of those things <g>)>

[Quade]

I believe I have a fix for this. Took some doing to get it to fail for me but, once I knew how, it was reproducible. The fix is in 6.4 but, I'm going to put it into 6.33 too and post it up sometime in the next couple days.

[rmason]

I believe I have a fix for this. Took some doing to get it to fail for me but, once I knew how, it was reproducible. The fix is in 6.4 but, I'm going to put it into 6.33 too and post it up sometime in the next couple days.

That's great. Thanks Quade.

And for my next act...

While checking out my data folder per your guidance, I found an anomaly. I seem to have two Data folders, one under "<username>/Applications Data" (this is the one Options->Open Data Folder points to), and the other under <username>/Local Settings/Application Data". Both folders have files with today's date on them, several in the "Appl. Data" folder and a couple in the "Local Settings" folder. One I recognize in the "Local Settings" folder is my .nbi file.

Assuming this is indeed an anomaly, can you tell me how to combine the two folders? I could move the .nbi file and the other with today's date (guItems.db3) to the "Applications Data" folder but I'm assuming Newsbin would have to be told where to find them.

[ppan]

I am getting multiple temp directories in my UnRAR Folder (e.g. UnR11B3.tmp). Shouldn't they be deleted by Newsbin?

I have checked:
Override Unrar path
Add a path prefix
Save to unique Folder...

[tl]

Found another issue with 6.40B1, it's a lot slower on sorting/displaying large groups, it's hard to say how much slower but 6.40B1 takes at least twice as long, probably more like 3-5 times what 6.3x needs on the same data!

At first I wondered if it could be because I'd finally purged a lot of old headers from disk (SSD), even if I couldn't quite see how it would affect resorting with all data already loaded but then I saw 6.40B1 slower to process headers than 6.3x and wondered if it was the same issue. So I installed 6.33 and loaded up exactly the same group and time-span and it was much, much faster (and similar to what I remembered from 6.32-something I had before 6.40B1). I do think the last header download was pretty slow, so I suspect I can also confirm the header slowness issue he's reporting.

Given that sorting is also known to be broken in 6.40B1 I'm not going back to it, these are serious group of regressions that needs to be fixed (it could even be the same core bug causing them all?).

Platform: Windows XP SP3, fully up to date (32-bit).

[possamai]

I have a feeling unrarring is a lot slower with 6.4 then it was in 6.3.
I'll have to test some more, defrag my disk etc. etc but am I the only one seeing this?

Also, opening an NZB file in windows by double clicking it immediately adds it to the downloadlist if newsbin wasn't open yet.
If newsbin is allready running it just opens the NZB normally.