beta 3 cannot load headers or handle local searches

[RayMark]

With 6.50 beta 2 I can load all the headers of a pretty large group, for example: 7.5 million headers (memory usage: 5.5 GB) - loads with ease in seconds.
Also, I can run on such a large group a local search, for example, for a single dot (".") - that returns all the posts in the group - exactly the same number as with "Show all posts".
Again, very quickly.
And I can do it both together! - memory usage (when together): 10.7 GB (with 7.5 + 7.5 million posts)

With 6.50 beta 3 I can do nothing: loading all the headers, or just 300 days of headers - out of the question - it shows a blank window for a long long time, the memory usage increases very slowly and then NewsBin becomes non-responsive.
The same with such a local search for a letter "t" - non-responding. Forget about a dot....
Memory usage usually about 0.5 GB, I've seen once 2.4 GB though - but it got non-responding.
It is almost as if it is a 32 bit version - but when installing I chose the default 64 bit and the exe file is: NewsbinPro64.exe

BTW, when beta 3 is non-responding it still is downloading and unrarring, just the GUI and header loading and search is non-responding and never ending.

But beta 2 is working very impressively, just great! Thanks.

[Quade]

Both are working here.

Try loading 10 days and see if that works.

[RayMark]

beta 3 is able to load 10 days - but not smoothly:

First, I tried loading a folder with 7 groups: It loaded something initially, clearly incomplete, then became unresponding - Windows 8 shows that turning circle and does not allow to access Newsbin - it becomes pale.
But that was temporarily - non-responsiveness went away, and the propper number of posts was loaded.
This took almost as long as loading 2000 days with beta 2. The difference is , that even when loading 2000 posts, beta 2 does it smoothly and quickly. No non-responding phase.

Then I tried to load 10 days of a single group with beta 3: this time there was no non-responding phase, bet after initially showing some posts the window went blank and stayed blank for a pretty long time. Then the correct number of posts appeared.
Again, beta 2 does a better job with all the posts.

And certainly, beta 2 is able to search for a dot in that folder with 7 groups. Again - smoothly and quickly. No non-responding phase. Beta 3 cannot do it.

I am using the same .nbi file with beta 2 and beta 3, pointing to the same data with spool, download lists etc, and to the same download folder.
Strictly speaking, a copy of the same .nbi file - when I decided to try beta 2 again, I copied .nbi file from beta 3 into it. So at that point they were identical.
It is just more convenient for me to have .nbi in the same folder with NewsbinPro64.exe, but I can always copy and overwrite the newer one over the older one - from beta 2 to beta 3 or from beta 3 to beta 2.
So I don't think .nbi is an issue.
Maybe beta 3 does not like something in that .nbi that is ok for beta 2?
Or maybe beta 3 does not like something in the data/spool folder - again, even if it is ok for beta 2?

[RayMark]

If it really is working fine for you, it is probably something with my headers.
I remember having a similar situation long ago with certain Newsbin versions too. One earlier version was loading headers just fine, and later versions somehow were stuck showing an empty window.
I think I posted complaints about it. Or was it also that downloading headers with the newer version was making gaps too.

Today, I am using a different computer, a different OS, a different version of NewsBin (64 bit). But I moved the same SPOOL with headers and everything else in newsbin Data folder.
So perhaps a similar situation. For some reason beta 2 likes my headers, but beta 3 - does not.
But I don't even remember, perhaps the previous problem was when still using SPOOL_V1.

[Quade]

If it really is working fine for you, it is probably something with my headers.

It's working fine for me because my machine is a beast. I'm pretty sure that I know what the issue is. The fix to always show all updated files when a header download completes AND "Hide Old" is set, requires more CPU than the older mechanism did and this CPU usage goes up with number of elements in the list.

One solution would be to unfix it.
Or find another solution
Or don't show partial loads while loading the group up. Only show when the load finishes.
or doing a post list display re-design

If you want to use B3, you probably have to limit hits for now. I do 60 days and don't notice the additional processing but, other groups might require more CPU.

[RayMark]

But if beta 3 requires more CPU and more time to load those headers, it needs probably hundreds if not thousands of times more than beta 2.
I was waiting maybe half an hour while Newsbin was non-responsive, and then I gave up.
And its memory usage was still very low and growing very slowly - maybe hours more were needed to reach those 5.5 GB, if at all.

Beta 2 loads those headers probably in less than 1 minute. Maybe in about 30 seconds.

My PC is just i7-3770K with 4-cores, but in virtual 8 core mode (HT, 2 threads per core), which supposedly uses those 4 cores more effectively.
Memory: 32 GB.
OS: Windows 8 - again, because it is supposedly faster and better optimized than Windows 7.
I got rid of Metro anyway.

[Vimes]

I use NB on a PC with a 3570k and 8GB of memory, running W8 64, minus the Metro.

In the short time I have used NB I found that beta 2 seemed to load all the headers fine when, for example, I downloaded 5,000,000 from each of the groups that I had combined within one folder, if that makes sense. .? (So if I had a folder named "Cars" then two newsgroups related to that would be within that folder).
So to load all headers from the disk and display them all worked a treat, and was pretty responsive.

With beta 3 it seems to be really really slow to the point of hanging the application when trying to do the same thing, with a large number of posts from several groups. Several times I have purged the groups of all headers, thinking that they have got corrupted, but the same applies.

If I download a period of 15 days or so it will often work fine.

I think that another user here is experiencing the same thing...

viewtopic.php?f=43&t=32111

EDIT:

I have just purged two groups and downloaded 5,000,000 header in each (from a "master " folder) then I right clicked on that folder to display all the posts and it took around 20 seconds to display all of them going back to the 16th of March (78,883 posts). So maybe it depends on the number of posts and groups you are trying to display in one go before it becomes an issue..?

[RayMark]

but with beta 2 I can load all the headers of a group with 7.5 million of them. All the way going back 2000 days (even more than actual retention).
I am pretty sure that of much larger groups as well.
And it is done maybe in 30 seconds.
One might ask why I need so many headers at once. Why not? - if they all are loaded so quickly and if they take only a fraction of memory I can give newsbin.
However, when there are problems with one month worth of headers, it is really something wrong with beta 3.

Another thing - the same problem is with local searches. Beta 2 allows to have many hits and performs the search very quickly.
Beta 3 hangs not responding on local searches too. Maybe it can handle very narrow searches only.

[Quade]

but with beta 2 I can load all the headers of a group with 7.5 million of them. All the way going back 2000 days (even more than actual retention).

The part you seemed to miss is "more results = more CPU" so, the more you load, the worse it gets.

It's the only change in B3 that might impact this do, I don't really doubt this is the problem.

[RayMark]

But it is not acceptable, as it is in beta 3. It cannot smoothly load even 10 days.
And even if I do not have a PowerMac with 12 cores or some multi-cpu server mobo, my PC is still one of the fastest of main-stream PCs:

i7-3770K (4 cores running in HT mode: 8 threads/virtual cores), 32 GB of memory, 1536 CUDA cores.

So, perhaps you could add to settings something to switch off the improvement you made in beta 3 - for those who want to load more than a few days at a time?
Or some more sophisticated switch: to switch off that beta 3 improvement if loading more than N days of headers - N to be set to some default but allowed to be changed by the user. So 0 would mean - the improvement is completely off.

But it affects local searches too, so a setting is needed to control that improvement in local searches too.

BTW, seeing those 1536 CUDA cores - do you use them for something? Maybe for par operations? There are some par tools that can use them.
Maybe for more things, like making beta 3 to load headers faster?

NewsBin could use OpenCL to offload CPU to both nvidia and AMD/ATI cards.

[Quade]

You're beta testing and I appreciate that you found this

But it is not acceptable, as it is in beta 3. It cannot smoothly load even 10 days.

You're not reading what I wrote are you?

http://www.cpubenchmark.net/cpu.php?cpu ... GHz&id=903

Your machine's no slouch for sure but, when I say beast, I mean beast.

[RayMark]

Yes, you have a crippled processor
They deliberately disabled two cores out of 8, I think.
And a step back from Ivy Bridge to Sandy Bride-E.
I think native USB 3.0 and Thunderbolt may be very useful when downloading with NewsBin to external drives.
In fact, I am downloading to a USB 3.0 drive.

[Quade]

I think native USB 3.0 and Thunderbolt may be very useful when downloading with NewsBin to external drives.

Yeah, love USB3. I think I have 4 ports. Tbolt, I'm not sure it's ever going to become mainstream.

[RayMark]

OK, so I made this test:

start newsbin, hit pause, load a folder with 7 groups (Show all posts)

with beta 3:

threads used: 88, memory usage gradually increasing, but very slowly, all the time > 500 MB but < 550 MB, CPU usage - 14%, almost never reaching 15%
the window with headers - blank.
After 9 minutes, headers showed up in the window and the memory usage jumped to 800 MB.
Then I made a click on those headers - that caused status "non-responding". I should have known.
I know that the expected memory usage is about 5.5 GB, so with 800 MB there was no reason to touch it.
I guess, it changed nothing, responding - but only until you touch it - the same as non-responding. I guess it continues those headers anyway. But GUI could run in a separate thread and never become non-responding.
I waited 11 more minutes - CPU about 14-14.5%, memory usage very slowly increasing, from maybe 815 to 822 MB.
So, after 20 minutes of waiting, I had enough. Remember, expected memory usage is 5.5 GB, and here - still 822 MB.

with beta 2:

threads used: 90, memory usage - 1st minute up to 800 MB afterwards fast increasing up to 5.5 GB - showing some headers.
after approximately 2 minutes 15 seconds - the screen went blank and stayed blank for maybe 15 or 20 seconds. Then everything showed up at once.

The headers were fully loaded in maybe 2.5 minutes.
The processor usage was about 8%, there were a couple of short peaks of 10-14%, and towards the end the CPU usage dropped to 1-2% then got back to 8% at the very end.


CONCLUSIONS:

the 20 minutes that I tried beta 3 - I probably still was in the 1st stage (where beta 2 also had about 800 MB of memory usage).
I did not wait long enough to get to the stage where the memory usage increases much faster.
But how long more would I had to wait? And the second stage takes at least the same time, maybe longer. So we are speaking about an hour total, best case scenario. Maybe several hours.
However, even with beta 3, CPU usage was only about 14-15%, so I don't think that my slower processor was limiting the speed of loading.
Newsbin could have used much more of CPU, something else was probably limiting the performance?

Another thing: why GUI is not responding in beta 3 while loading headers? GUI should run independently, on a separate thread.
In beta 2, I can start downloads without waiting for all the headers to load: I can start download of something that is accidentally on the screen during loading of the headers.
So, I guess, I cannot do the same with beta 3. Something went wrong with the independence of GUI.

[Quade]

Beta 4 is up...thought you noticed.

My GUI never locked.

[RayMark]

Thanks, I did not notice beta 3 for several days either - turns out, refresh of the page was needed in Opera.
With beta 4 - again the same.

[Quade]

I'm skeptical. Beta 4 reverts to the B2 way except during header downloads. Loading is as fast as B2.

[RayMark]

No, it isn't!

Beta 4 immediately started growing used memory and finished everything in less than a minute. Then that blank period started.
I guess I was wrong about its length, it is not 15 seconds but more like 25 seconds (probably with beta 2 as well).

So the whole thing was loaded in 1 minute 15 seconds, and beta 2 needed 2 minutes 30 seconds.

Unless there is some accidental dispersion, beta 4 is twice as fast as beta 2

[Quade]

With beta 4 - again the same.

Which I interpreted as "as slow as B3". Which it sounds like it isn't

[RayMark]

UPDATE

You were right, exactly the same.

Yes, something went wrong with beta 2, maybe antivirus was updating at that time or something.

I retried again:

with beta 2:

1 minute 19 seconds, the blank period at the end: 27 seconds.

with beta 4:

1 minute 20 seconds, the blank period at the end 28 seconds.

My time taking also +- 1-2 seconds

[RayMark]

With beta 4 - again the same.

Which I interpreted as "as slow as B3". Which it sounds like it isn't

Opera again did not see it

[RayMark]

Local search returning all the headers - I search for a dot - is just as fast, probably a few seconds faster.
So what are pros and cons using local search and not regular loaded headers?
Local search, when all the headers are included, can then be narrowed down instantly, at least reacts instantly. Maybe it is a pro.
Although typing a longer string may be a con - better to paste at once.

Filters are not available, including custom profiles. Probably the main difference.

BTW, I just recently found out that a local search is also possible in a single group, not just in a folder: right-click "Search in Groups" is misleading, it should be "Search in this group"
You can recognize when right-click is not on a single group but on a folder and then call it "Search in Groups" or "Search in groups in this folder"

[Quade]

- Each new local search results in a reload from disk.

- Loading all the records and searching within them doesn't.

- If you actually search for something, and not just load everything up, it reduces RAM usage.