Newsbin Peformance Degregation

[jimerb]

Newsbin for some reason has gotten itself into a near standstill that I can't seem to get around. The chunks folder has over 450,000 files in it! After a little research here I've:

Turned on: Aggressive assembly mode
Turned off: Pause Pars and only download the ones needed for repair
Turned On ause download during UnRar/Repair

I then restarted newsbin. It doesn't seem to be helping.

What I appear to have is huge swaths of files that show a 100% complete bright blue bar in current progress. Most have a status similar to: (4 files, 0 pars) D:4 DL:0 Retry:0 N:10). If I expand a file set they status bar is all grey and it says downloaded. Why would newsbin be waiting to unrar? Even the ones at the top of this list are this way.

I'm not sure how I should proceed? Why would they

Will assembling incomplete help here? Should I delete everything in the chunks folder?

What would Quade do?

I'm running 6.72 Build 4776.
I have a pretty fast (200mbps) connection.)

thx.

[Quade]

As long as you're not running 32 bit windows, I'd try the current 6.80 RC1 version. If you're running 32 bit, it won't work for you.

[jimerb]

Thanks Quade. I'm on a 64bit machine. Do you see any issues moving everything to the wishlist and installing the upgrade? Will it maintain it's current state during the upgrade?

[Quade]

You should just be able to install and go. You could make a backup of the downloadsv2.db3 before you upgrade.

I wouldn't move anything.

[oldswede]

I have upgraded to the RC1 one release and I have the same problem. The files are downloaded but will not unRAR. They are complete and I can force them with WinRAR in the download folder, but Newsbin is paralyzed.

[Quade]

One thing that can cause this is if the download or unrar drive is missing I've verify that the drive letter is still there and hasn't changed. You need to make sure unrar is enabled in the main options and that you didn't disable it in the per group/gog options.

Are you seeing any errors in the logging tab? JimmerB's problem isn't that nothing unrars, it's that his download is much faster than the unrar so, the chunks stack up. This issue is addressed in the 6.80 series.

[jimerb]

OK I upgraded to 6.80RC1 Build 5005 and things started off great. The system started chewing through the backlog and I thought I was out of the woods.

Then all of a sudden after about 30 minutes it just stopped. I have 994 files waiting to download and the system is busy downloading something but there a tons of sets that have completely downloaded and it is not unraring them. Instead, I suspect the backlog is just continuing. (I don't know why the system is downloading anything at all if I have "Pause download during UnRar/Repair" turned on? I have lots of completed sets. I would think it would be working on those?)

They all have a status like this:
PAR: 0 of 0; D:2 DL:0: Retry:0 N:0. Blue status bar at 100% completion. I cannot find any green status bars anywhere showing an unraring is happening.

If right click on a file and say "assemble incompletes" it doesn't do anything.

I restarted Newsbin to see what would happen and again it starts off aggressively chewing on unrars suggesting that there is indeed lots of stuff it could have been unraring.

The logging tab shows a normal situation where there are some files that were not available, but that doesn't explain to me why i have about 200 file sets totally downloaded with no errors and no unraring going on?

Not sure what to do?

[Quade]

I have a theory. Exit Newsbin, go into the Newsbin install folder then rename "par2Repair64.dll to par2Repair64.dll.old.

Then start Newsbin and see what happens. This will force it to use the backup repair code. Dex and I might need to rush out an RC2 in the next day or so.

PAR: 0 of 0;

This is suspicious though. Newsbin doesn't use the DLL to count par blocks.

[oldswede]

I found my problem. There was an inconsistency in the download/unrar paths. Sorry for the unnecessary static.

[Quade]

I found my problem.

That's good.

[jimerb]

As suggested I renamed the "par2Repair64.dll to par2Repair64.dll.old.

Then I fired it back up and downloads look like the were pausing as unraring was being done. After an unrar downloads would resume. All looked good.

I left it running for about 90 minutes and came back to the same situation:

The Condition:
* 1128 files waiting to download. Active Downlin in progress.
* Over 100 full sets are downloaded and showing 100% complete. (Solid blue bar) and similar statuses that look like : (PAR 0 of 0) D:3 DL:0 Retry:0 N:0) A few are missing a couple of pieces but most are complete.)
* No green unraring anywhere.
* The count on "Downloading Files" tab is not going down.
* The only errors i see in the logging tab are "no such article" errors.
* Files List tab shows lots of files downloading and new ones popping in there but it is not making the "Downloading files" tab drop in size.


Restarting Newsbin cleared all of those blue bars and it once again went through the process of downloading a few files and repairing and unraring them.

I suspect the condition will likely resurface given some time.

Let me know if you want me to try anything else or provide more details.

[Quade]

Prepping an RC2 for you to try. It'll probably be up tomorrow.

[jimerb]

Ok,. Thanks for the update Quade.

[Quade]

If you email Dex at ts@newsbin.com. He might email it to you early. We're trying to get to the bottom of this.

[jimerb]

I've got RC2 running on a heavy load. Should have some real world feedback soon.

[jimerb]

Some initial feedback:

I think there is a problem with the display of unraring (green status bars.) I started a heavy download and could not find an unrar happening anywhere.

Upon closer inspection I found some unrars happening silently without any indication on the status that it was in progress.


Right now, I can see that an unrar is happening and I have a full blue bar on the parent of the item in the downloading files tab.
Display: Par: 2754(1) OF 2799 d19 DL:0 Retry:0 n:8

If i expand it, everything (including pars) shows downloaded. The status bars are rolling from blue to grey as time progresses and then the extracted file eventually appears fully extracted.

Nowhere does the status bar turn green.

Reminder of my autopar settings:

* Turned on: Aggressive assembly mode
* Turned off: Pause Pars and only download the ones needed for repair
* Turned On : Pause download during UnRar/Repair


Also I just had 1 crash where the app just disappeared (no error message.) I've restarted to see if that occurs. I realized after I restarted that there is a debug log. If it crashes again i will pass a long the last notable things I may see in the log.

Testing continues..

[jimerb]

More feedback:

After about 40 minutes a backlog has started and UNRaring seemed to stop. I cannot find any unraring happening.

Full sets including pars are available to unrar.

Let me know if you need any other details.

[Quade]

Are all of your unrars going to the same drive? This symptom is similar to what happens if the unrar drive is missing. So, if some went to one drive and some went to some other drive, the unrar would stall but the download could continue.

I downloaded about 30 gigs in one shot the other day and they all unrared so, I can only guess at this point.

You might disable "Pause on UnRAR" and see is anything changes.

I think there is a problem with the display of unraring (green status bars.) I started a heavy download and could not find an unrar happening anywhere.

Upon closer inspection I found some unrars happening silently without any indication on the status that it was in progress.

This can happen with obscured files that don't compact together. They've all left the download list by the time the unrar starts so, there's no line to display the unrar on anymore.

[jimerb]

Are all of your unrars going to the same drive? This symptom is similar to what happens if the unrar drive is missing. So, if some went to one drive and some went to some other drive, the unrar would stall but the download could continue.

Yes they are all going to a single network path. My work flow uses the "Download To Existing Folder" and the newsbin internet search. The main folder is not a mapped drive letter but rather a network path to a very capable qnap nas drive. Tons of free space and about a year old. Solid performer. It's been well broken in with newsbin. Performing solidly for over a year.

They would all go to a path similar to \\mainserver\data\cars for example. I figure i would mention this because it might mean something to trying to figure out what's going on. NOTE: My chunks folder is on a local drive to the pc (1tb free space).

You might disable "Pause on UnRAR" and see is anything changes.

I tried this tonight. The first time I launched after this setting it ran for about 30 minutes unraring. Then it stopped unraring and just kept downloading. After a restart it still just kept downloading with no unrars.. Then, in an effort to try and get a fresh start, I deleted all the files in my chunks folder and started newsbin. When I did that, unraring began again.

Some random things I noticed that may be a factor (not sure):
* Lots of files pile up in my chunks folder. Like over 100,000 just today in about 2 hours of use.. Should I be changing a chunk size somewhere?
* Even when newsbin is unraring, I can see it's skipping unrars at the top of the queue that look totally downloaded. I can't figure out why that's happening. They have a status right now of PAR: 3 of 164 D:3 DL:0 Retry: 0: N:0.

Not sure what to do at this point?

[Quade]

They have a status right now of PAR: 3 of 164 D:3 DL:0 Retry: 0: N:0.

This means the download isn't even 1% done so, they're getting skipped because the files haven't been parsed for PAR blocks yet.

Used to be, 6.73 and before, the chunks would backlog during a repair and unrar. Now, as long as you let it download and unrar at the same time, the chunks don't backlog instead the files get saved to disk and the files backlog.

On the other hand there's, a single thread that handles PAR processing and unrar, so, now if you download faster than the files can be processed the files backlog. Meaning you might have all the files downloaded for a set but if the repair thread is busy, the par blocks won't count up will the busy thread finishes processing. This sounds like what you're seeing.

Downloading to a network drive is probably not a great idea. On the other hand unraring to a network drive probably isn't that bad. The problem is that PAR processing requires a bunch of back and forth to the disk file so, downloading to a network drive will slow PAR processing and unRAR. I'd download local, unrar to network.

More than likely your chunks collection is files that are incomplete that haven't been "force assembled" yet.

If I was you at this point, I'd install and run "Procmon" and see what Newsbin is doing during this time period. I'd set a filter to only show Newsbin's file IO.

[jimerb]

Ok, I restarted newsbin and now it's nicely going though the files so far downloaing and unraring.

I have procmon ready with just reads and writes filtered on newsbin. As soon as it appears to not be unraring I will start capturing to see if it is writing to the NAS or just downloading to chunks.

For a long time I've been unraring to this nas drive. What i don't understand is that when i reboot the app things behave normally for awhile. Is it possible that only certain types of posts are causing the wheels to come off the wagon?

They have a status right now of PAR: 3 of 164 D:3 DL:0 Retry: 0: N:0.

This means the download isn't even 1% done so, they're getting skipped because the files haven't been parsed for PAR blocks yet.

...

On the other hand there's, a single thread that handles PAR processing and unrar, so, now if you download faster than the files can be processed the files backlog. Meaning you might have all the files downloaded for a set but if the repair thread is busy, the par blocks won't count up will the busy thread finishes processing. This sounds like what you're seeing.

If the file hasn't downloaded fully yet why are all the items showing downloaded when I expand the item? There are no par sets that I see but every single line is blue at 100% downloaded. I would think that the item at the top of the queue would be the very next thing to get the unrar threads attention?

Is there another way I can tell if the repair thread is busy or stuck? I don't see any yellow status bars when things look like they are stuck.

[Quade]

For a long time I've been unraring to this nas drive. What i don't understand is that when i reboot the app things behave normally for awhile. Is it possible that only certain types of posts are causing the wheels to come off the wagon?

The only thing I'm wondering is whether the unrar is still happening hidden behind the scenes.

It apparently wasn't working right for you in the old version and it's not working right for you now so, I'm wondering if there's some underlying issue that's independent of the version.

Newsbin lays the file on disk and as it's doing so, it's calculating the MD5 for the whole file so, IDing a new good file is essentially free. That works great until you get an incomplete or damaged file. Then it has to scan the file byte by byte looking for PAR blocks. That's when using a NAS becomes a real problem. You start paying for round trips to the NAS and honestly writing and reading a file over a network is inherently less reliable than doing it to a local drive. That's why I'm suggesting "Download local, UnRAR remote" because UnRAR is purely a write to the destination with no back and forth. Whether it impact your issue or not, I don't know. I'm just telling you what best practices are.

You mentioned the hidden unrar so, that's what I'm looking at right now.

[jimerb]

Ok I have some test results with procmon: (Good idea using this.) I turned on "Pause Download during UNRAR/Repair to minimize network traffic for this test.

I can see when it appears to fail that the system is creating the RAR files from the chunks.

I am monitoring a file set right now. The rar's are all 100mb. The first 6 took 30 seconds to 2 minutes to create from chunks. After that, it spiked to 10-12 minutes each. 2GB of rar creation has taken 2 hours.

Simultaneously I'm monitoring the network traffic and the machine is generating less than 1/2 of 1% of network traffic on my gigabit Ethernet network. I checked this from the PC; NAS and the router. Traffic is minimal. I also reset the router before doing the test.

I do not have downloads paused. No downloads are happening.

So i guess the question is what could cause a sudden slowdown in processing the chunks? Could it be that there so many of them?

[Quade]

Now I'd set the download path to the local drive and see if it does the same thing.

[jimerb]

Thanks Quade.

I'm going to spend some time trying to prove out the NAS. It's a little tricky for me because the PC does not have the same storage availability as the NAS so stress testing it will take some work.

I've also upgraded the NAS firmware and I'm going to experiment with that as well. I'm also going to try and map the network location to a drive letter and download to that to see if it makes a difference. I really have no idea why all of a sudden this started to develop. Nothing in the configuration changed.

So I will likely take some time (maybe a week) to dig through that.

Quick Question: I noticed that procmon was doing something with the main download folder even though all of my downloads were pointed to a specific target elsewhere. Is there anything notable about that? I'm wondering if it is irrelevant or perhaps the main drive should be local for performance reasons? Just want to make sure it's not a factor.

Thanks for hanging in there with me.

[Quade]

It looks at the main download folder when it's checking available disk space. Could be that.

One of the guys on IRC was commenting about after he replaced a failing drive, his NAS was running double the speed it had been. I'm just suggesting some testing.

The fact your chunks folder fills up suggests Newsbin is having a hard time assembling the files. With the new design, the chunks folder should collect up chunks if you have many incomplete files. Other wise you shouldn't have more than the currently downloading files worth of chunks in there. This is a sea change from earlier versions.

[jimerb]

Ok great. This is helpful.

One thing I'm trying to get my mind around:

What is the difference between:

* Assign a new path
* Assign a new unrar path
* Download to an existing folder

I want my end state to wind up on the nas if i can. I can certainly carve out a sandbox for newsbin to download and create rars locally. Just not sure which ones I should be using.

thx.

[Quade]

Should have said "Shouldn't" collect chunks in my last post.

* Assign a new path
* Assign a new unrar path
* Download to an existing folder

1 and 3 are essentially the same thing. #1 can generate a new folder. #3 should use an existing folder.

#2 really only applies during unrar.

[jimerb]

Hey Quade,
I'm still testing this. Nothing conclusive yet.

Quick Question:
Is there a way to setup a button on the toolbar (or right click) and specify an unrar directory while having the "download directory" stay local on the PC?

I'm looking for a "Single Button" that accomplishes this without having to manually set both the download path and unrar path each time.

I'm trying to establish a workflow that does all of the downloading and repair on the pc and then unrar's to a folder on the NAS. The intermediary work location is irrelevant to me --just the unrar location.

Possible?

Thanks.

[jimerb]

An update:

After some heavy duty stress testing, putting everything on the local drive, I'm still seeing situations where lots of stuff is downloading but nothing is unraring.

I'm also seeing everything has "display update pending 72721" for example in the list. Over 1.5 million items in the chunks folder. I cleared the chunks folder and started newsbin and everything started off fresh again when i launched newsbin.

Should the downloads just stop at some point if they get too far ahead of the unrars? I would think the most efficient situation would be for the downloads to get just far enough ahead (plus a little bit of a buffer) to keep the pc unraring.

There is no antivirus on this windows 7 machine.

[Quade]

Over 1.5 million items in the chunks folder

This indicates a problem. After each file downloads and is assembled, most of these files get delete per file. I keep one around but it gets purged when it's 2 days old. My chunks folder has 32 items in it and I did a fairish amount of heavy downloading yesterday.

I might clear it, then watch some downloads and the chunks folder and see if you can characterize when they don't get deleted.

You want to make any security software ignore the download and data folder too.

[Questar_99]

I came here looking for a fix to the same issue.

Queue up downloads, it appears that I can download faster than files are par checked. All downloads are s-l-o-w-l-y par checked, CPU utilization is near zero. Then after a time the files files are unrared, with a pause of about a minute between files when Newsbin is doing nothing.

Download path is to an SSD, unrar is to a different drive. None of my settings have changed in years.

Edit:Confirmed the par checking is running super slow, and not using any cpu.

https://imgur.com/a/AD65U

[jimerb]

I came here looking for a fix to the same issue.

Queue up downloads, it appears that I can download faster than files are par checked. All downloads are s-l-o-w-l-y par checked, CPU utilization is near zero. Then after a time the files files are unrared, with a pause of about a minute between files when Newsbin is doing nothing.

Download path is to an SSD, unrar is to a different drive. None of my settings have changed in years.

Edit:Confirmed the par checking is running super slow, and not using any cpu.

https://imgur.com/a/AD65U

When things seem to slow down try closing newsbin, deleting all the files in the chunks folder and then starting back up. That usually gets things back to normal for awhile before it occurs again.

I'm trying to figure out based on Quades recommendation if it would be the file at the very top of my queue that is sitting idle that is causing the problem or one further down in the queue.

The best guess right now is that the fact that there are so many files in the chunks folder that it is slowing down everything -- yet like Questar_99 mentioned my cpu is near 0% too when this happens.

[Quade]

I've noticed there seems to be a one par file delay between getting all the chunks and par blocks in and starting unrar. I'll look at that. That combined with a bunch of other downloads might explain why the unrars are delayed but no CPU is used. It's essentially counting down a timeout.

[jimerb]

I've noticed there seems to be a one par file delay between getting all the chunks and par blocks in and starting unrar. I'll look at that. That combined with a bunch of other downloads might explain why the unrars are delayed but no CPU is used. It's essentially counting down a timeout.

If it helps my fully downloaded unrars are sitting like this right now:

[par 0 of 0] D:10 DL:6 Retry:0 N:10

The DL:6 apperars to be the pars that are paused. (I've turned back on to "pause the pars" because it didn't seem to be making a difference.)

[Quade]

[par 0 of 0] D:10 DL:6 Retry:0 N:10

This indicates the small par has downloaded and none of the rest of the downloads have registered. Since there's only one thread that does unrar and repair, the downloaded files will stack up behind any unrars so, eventually you'll have a bunch of downloaded files which then unrar over time.

I had a guy complaining at me the other day. He downloads 3 TB a day on a 1 Gbps connection. It takes 9 hours to actually download and another 4-5 hours to get the files completely repaired and unrared. His complaint was that after download, shuffling the download list order didn't also shuffle the unrar order. That fact he gets decent downloads when banging on it that hard suggests it ought to be possible for you to get that too.

I'm investigating the one par delay thing. I think fixing that would keep the pipeline going with less stalling.

[jimerb]

Thanks Quade. I'm wondering if it has something to do with my recipe of settings?

Just some background:
I'm exclusively using the internet search. I don't download headers. (Not sure if that matters.)

Everything is done with "Download to folder"

[Questar_99]

[par 0 of 0] D:10 DL:6 Retry:0 N:10

I'm investigating the one par delay thing. I think fixing that would keep the pipeline going with less stalling.

That would be a help, but please look into the new par checker. It's seriously slow.

Edit:
I took a video of the par processing, where can I send it?

[Quade]

You should test again with RC4. I ran 80 gigs through it with no issue. It fixed the "waiting for one par" delay too.

[Questar_99]

RC4 is much better, getting rid of that delay was a big help.

The par checker still seems to not really be able to keep up with my connection though (500mb). Like I said earlier, practically zero CPU usage when it falls behind.

[Quade]

The download will always finish faster than the par check, repair and unrar. Particularly on a high speed connection.

There's one running thread that does PAR checks, repair and unrar. When it's unraring or repairing, the checking and counting stack up. When the unrar completes, the files start getting checked again. So you could say it's by design. With current hard drives, trying to run checking, unrar and repair in parallel is kinda pointless.

Banging on the drives with multiple heavy duty reads and writes slows down more than you might expect. Having two things banging on the disk isn't 1/2 as fast, it's probably 1/4 as fast. Until everyone has NVME or faster SSD drives for this, I'm not seeing it changing in Newsbin.

[Questar_99]

But it was so much better before...

I mean my machine is sitting here with 3% CPU utilization and I'm just watching the numbers slowly count up. I'm not I/O bound, I have Primo Cache sitting in front of my SSD's.

Is there a way to rollback to the last stable release?

[Quade]

Sure you can use any version you want.

[Questar_99]

I just rolled back to 6.73 (it's so nice you can do this!) and the par checking is an order of magnitude faster. It could even get caught up after a repair.

Is the par checking in 6.8 that different?

[pjcamp]

I've got exactly the same issue in 6.73. What I've noticed:

1. It sets in when the attachment size is around 1.5 GB. Below that behavior is normal.
2. Though it is reporting the file as completely downloaded, the .rar segments are not actually present in the downloads folder. They appear there slowly, over time.

Effectively, it is downloading everything first and then decompressing.

Didn't used to do this. It appeared to start about the time I upgraded my motherboard, CPU and memory but it is hard to see how that would have an affect.

Right now I've got 9 files that are reported as fully downloaded but that are not un-raring.

[Quade]

Your downloads are faster than assembly. You should upgrade to the latest. It works differently. I'd use 6.81RCX. I've already handed the 6.81 release off to Dex. It'll be out before too long too.

[jimerb]

Quade you helped me out tremendously with this awhile back.

I figured I owed you and other readers what I perceive to be the root cause of this problem.

We had spoken about unraring over a network drive.

Turns out I was unraring locally but had an intermittent hard drive issue on the pc i was using to do the work.

Over the years this hard drive took a real beating and it's time was coming to an end.

The problem was that I didn't get any drive failure messages. The drive acted like it was fine, but in reality it wasn't.

What alerted me to this is one day the drive just disappeared from the OS but would come back after a reboot.

After replacing it, things are rock solid once again for going on months now.

So other users if you are experiencing this issue, ask yourself how old is your drive and how much have you been using it?

hope this helps others that may search for this condition.

thanks again for all your help on this.

[Quade]

Turns out I was unraring locally but had an intermittent hard drive issue on the pc i was using to do the work.

This isn't that uncommon. Maybe less so now with SSD's though. I tend to cycle out old hard disks from time to time.