downloaded files slow to show up

[RayMark]

I am not sure when this began, likely in 6.70 beta 4, continued in beta 5.
Files are shown in the download list as downloaded but do not show up in the download folder at once.
Only the currently repaired/being extracted files are in the download folder and perhaps the files from the next post are showing up in the download folder. Many other posts are shown as downloaded but still no files.
Instead, the files show up one-by-one later - and very slowly - it feels like much slower than it takes to download them in the first place.
I don't like this new behavior. Also, I think I had problems with "Add to Top" and "Move to Top" working without delay when there were lots of such "already downloaded but in limbo" files.
The "pre-allocate files" is checked, although not sure if it is supposed to have anything to do with it.

[Quade]

You'll eventually be able to disable "Download while unrar/repair" which will stop this. It'll just stop downloading when repair and unrar is active then start up again when the unrar completes.

That'll keep everything in lockstep.

[RayMark]

The key word being "eventually".
Because I don't see such an option in the current version. Although it used to be in older ones, I think.
OK, thanks.

Actually, I used to disable autopar/unrar completely. But recently I began using it because I think it must be faster while the downloaded files are still in the memory cache.

[Quade]

Because I don't see such an option in the current version. Although it used to be in older ones, I think.
OK, thanks.

You can always revert to 6.62. That's how it works.

Actually, I used to disable autopar/unrar completely. But recently I began using it because I think it must be faster while the downloaded files are still in the memory cache.

The way Newsbin does repair is inherently faster than QuickPAR. It's multicore so, it's faster to actually repair and If you only have one bad file, The PAR lib only has to scan one file instead of all of them like Quickpar does.

[RayMark]

But you intend to fix 6.70?
Not just to add an option to stop downloads while repair/unrar is working?

Because I have no problem with unrarring and downloading simultaneously as long as downloaded files actually appear in the download folder - as they used to.
Currently they do not. And when eventually they do - they do so very slowly. So the whole process is very slow. As if the files are being downloaded twice.
In addition to that, "Move to Top" is partially ignored - even though the moved to top item is downloaded, but its files do not show up in the folder, instead files from items below keep showing up and are extracted. So the order of files showing up remains the same, moving to the top does not change it.

The third thing - in the case of some multi-part rars, not all rar parts show up. Some remain missing even though indicated as complete and downloaded (when expanded, those parts remain marked with arrows although also indicated as status: downloaded). And those items remain stuck and not extracted- obviously.

But if 6.70 will work correctly only in the "stop downloading while repairing/extracting" mode it will be in a limited performance mode.


And also, when I tried to use "Add to Top" for many rather large posts, I got Cache 0/200 (or 0/2000 in my case) pretty fast. So this issue is not fixed either. Help/About does not work as well as restarting Newsbin.

BTW, perhaps my large cache (2000 instead of 200) causes the downloaded files not to appear in the Download folder as quickly as they would with a smaller cache size?

[Quade]

Not just to add an option to stop downloads while repair/unrar is working?

There's nothing to fix.

Here's how it works.

1 - Download normally. After the last chunk of the file is downloaded, a notification is sent to the assembler and the assembler assembles the file.

2 - Two things happen now.

- If this last file is enough to complete the set or enough to repair the set, the set gets repaired or unrared then repaired.

- It's not the last file of the set so the assembler goes idle waiting for the next file to finish.

3 - If a repair/unrar starts. The assembler becomes busy so it can't assemble files.

4 - Download continues during the repair/unrar. When the cache fills, chunks are flushed out to disk.

5 - When the repair/unrar finishes, the assembler starts to assemble the files with downloaded chunks. As they assemble they show up.

The third thing - in the case of some multi-part rars, not all rar parts show up. Some remain missing even though indicated as complete and downloaded (when expanded, those parts remain marked with arrows although also indicated as status: downloaded). And those items remain stuck and not extracted- obviously.

I have yet to lose a RAR. You might want to look at your security software to see it it's doing something with some of these rars.


I'm thinking about making the assembler sleep for a long time so I can make my machine simulate slow disks. See if I can make it lose rars.

[RayMark]

So is it possible that I am experiencing a very exaggerated behavior because I have set the cache to 10x size (2000)?

That file that had two rar parts downloaded but not showing up - it finally was assembled, but before that I lost patience and also downloaded additionally those two parts separately (at the top) - they did not show up either.
Now, after some time, I see that the whole file is assembled, but not sure if my separately downloaded parts were needed or not.

[Quade]

That file that had two rar parts downloaded but not showing up - it finally was assembled, but before that I lost patience and also downloaded additionally those two parts separately (at the top) - they did not show up either.
Now, after some time, I see that the whole file is assembled, but not sure if my separately downloaded parts were needed or not.

Unless you're downloading sets > 10gigs maybe 5 even, you really shouldn't be backing up. I'd say your main problem is slow disk IO. For the 500-800 megs sets I download, there is no backup. If your download speed is faster than your PC can keep up with then things are going to backup someplace. That's just the nature of receiving something faster than you can process it.

I don't think your cache size has much to do with it. Whether it's cached on disk or in ram, it works pretty much the same. In RAM would assemble a bit faster.

[RayMark]

Today I put into the download queue many posts 1.7-2 GB each. Maybe 150-200 GB total. And I lost control completely. Moving to top does not force a post to be assembled faster.


I am downloading into a WD Blue 3TB 5400 rpm WD30EZRZ but it is connected to the internal sata III (6GB/s). My computer case has a hot-swap docking station on top which is connected directly to the internal sata III.
So the hard drive is not the fastest, sure. But it is convenient: cheap, cold, and hot-swap replaceable by a new one when full.
After it is full, it usually is connected to an external USB 3.0 when needed, but at that point I no longer download to it with NewsBin.

The download speed sometimes > 400 Mbps, but sometimes < 100 Mbps. I have an impression that usenet servers deliver different posts at different speeds. It seems that the older ones are often being downloaded slower. Perhaps they are kept in some archives on slower servers, not sure.


If you say that it is normal with slow drives and high download speeds, then I say: why earlier versions, before 6.70 were able to work just fine with the same speed hard drives and the same download speed?
Only 6.70 has trouble handling this combination.

[Quade]

Today I put into the download queue many posts 1.7-2 GB each. Maybe 150-200 GB total. And I lost control completely. Moving to top does not force a post to be assembled faster.

Because files have already finished download. You're just changing the download order. It doesn't changed the order of already downloaded files.

I am downloading into a WD Blue 3TB 5400 rpm WD30EZRZ but it is connected to the internal sata III (6GB/s).

Ouch. You'd be better off downloading locally and unraring to the external drive.

If you say that it is normal with slow drives and high download speeds, then I say: why earlier versions, before 6.70 were able to work just fine with the same speed hard drives and the same download speed?
Only 6.70 has trouble handling this combination.

Because early versions didn't download and unrar at the same time. 6.62 for sure doesn't. Older versions it was optional. I've already mentioned there will be a switch to make it work like 6.62. You don't seem to understand that that means not downloading while unraring. Just like 6.62. The switch is for people like you with slow drives and fast internet.

When I think about it. You might be better off changing the cache back down to 200-400. I'd start with 200. The extra cache might be exacerbating your slow disk issue because the number of cache blocks that need to purge to disk is larger.

[RayMark]

6.70 needs a fix: the assembling order must be always adjusted to reflect the current order in the download list. When something is moved in the download list, it should be moved accordingly in the assembly queue. Or on the assembly line
I hope it is possible to implement?


.......

I often use 6.55 final. It can handle simultaneous downloading and unrarring just fine. Plus, it does not have cache = 0 stalling issues (6.70 still has).

So I am inclined to reject the argument that my hard drive is too slow.
It is possible, though, that 6.55 just slows down the download and waits until the file is written to the drive before continuing downloading. So that might result in lower average download speeds. But not lower than with 6.70 anyway.
But the advantage is that Move to Top or Add to Top works instantly in 6.55.

Unfortunately 6.55 has some other issues, for example, it sometimes creates some incomplete/corrupt files that are not properly indicated (not in all capital letters). That happens sometimes with multiple small files such as ebooks. Then I have trouble sorting them out, which ones are good and which ones are bad, because I get multiple copies of the same files and usually one is OK, but others are not, and how to know that (-0002) is OK if all are in small letters? That is somehow related to the state of my headers in those groups or something.

But with large files 6.55 works mostly fine, perhaps because I am more careful what I put into the download list.


......

I guess I could put the cache (with the whole NBData, SPOOL_V6, etc. - probably there is no way to separate them?) on the internal drive. I still don't want to use SSD for that because SSD has a limited life in terms of the number of write operations.
But my other internal drives are all 720p rpm WD Black drives. So, I guess, faster than Blue ones.

As to putting the Download folder onto a faster internal drive and then extracting to an external one, hm. Not everything is extracted. Not all posts have par files, or even are rars. Having Downloads on an internal drive would require periodical probably manual attending, as opposed to just replacing the full drive with the new empty one - I always have a big box of new empty drives

............

Just started a new download: 280 GB in 180 files.
I just hope I will not need to download something urgently in the next 4 hours...
Download speed so far 200-250 Mbps, but ETA: 4 hours.

[ddaniel51]

You will NOT wear out a current gen SSD. They will outlast the life of your computer. I run 16gb ram with 4000 cache to a 1tb ssd for NB's data,spool, downloads, etc then unrar to a NAS. It's so silly fast it keeps up with my 300mb bandwidth.
All 5 computers in the house run only ssd's with the only mechanicals being in the NAS'es. Quiet, cool, fast...

[Quade]

You will NOT wear out a current gen SSD. They will outlast the life of your computer.

Yeah this is my thought too. I don't even consider longevity anymore. Modern SSD's are really over-built.

http://www.amazon.com/Samsung-512GB-2-5 ... rds=850pro

Prices have really come down too. I paid more than twice this price for a 512 840 pro.

[RayMark]

Doesn't it say (for the 512 GB model at least) "Samsung warranties the drive for a lifetime total of 150TB" or something similar?

As I understand it, you have 10-year warranty as long as you do not write more than 150 TB total to it.
And there is a reason for this limit. Wearing out. Probably it is not healthy to write to the drive much more than 150 TB.

With Newsbin, it would be easy to exceed the 150 TB limit very quickly. At 300 Mbps continuously - in 50 days. In one year if you download only 3.5 hours a day. And the SSD is supposed to last several years....

[Quade]

I cycle out drives every couple years. Drives become unreliable over time.

In 10 years, we won't even use drives like this anymore. The transition away from SATA has already begun. We probably won't be using flash either.


http://techreport.com/review/27909/the- ... e-all-dead

The samsung pro drive in this test did 600 TB before it started to use the backup flash. Every SSD is over-provisioned and will start using the backup flash as cells start to die.

With twice the endurance of the previous model*, the 850 PRO will keep working as long as you do. Samsung's V-NAND technology is built to handle 150 Terabytes Written (TBW) which equates to a 40GB daily read/write workload over a 10-year period. Plus, it comes with the industry's top-level ten-year limited warranty.
*840 PRO: 73 TBW > 850 PRO: 150 TBW

You're downloading more than 40 GB a day?

[RayMark]

At least 5x more, but most likely more than that. Enough to try to avoid SSD, I think.