Why is 6.34 so slow?

[rfrischer]

Upgraded from 5.59 to latest ( I think 6.34) and the program runs very slowely. Sorts take forever. Went back to 5.59 and everything is fast again. Win XP pro 32 byte, plenty of memory.

[DThor]

Please explain what you mean by 'runs slowly'. Do you mean just sorting, or other operations too? What hardware are you running? There are some new performance options for older hardware, but I doubt it would affect sorting, unless you're doubling up operations like sorting during a par repair.

I suspect some aspects of the program would require a bit more horsepower, just like I suspect I would find v4 faster than v5, but you need to offset that with new features that make using the program faster. I would use it for a while to become familiar with the new features before discounting it just based on a sort. For example, compressed posts make for much more spool information being available at one view without the clutter of the old display method.

DT

[rfrischer]

xp pro with service pack 3, Intel core i7 cpu 920 at 2.67 ghz, 2.99 gb ram
When downloading a large number of headers it takes forever to sort the headers on 6.34 and virtually instantaneously on 5.XX.
Even if one then sorts on poster it takes a long time. Close the program, do control-alt-delete and newsbin still has a large file running in memory. Kill that file and the computer works normally. Otherwise everything is molasses. Went back to 5 and all is well. Large memory leak?? Who knows/

[Quade]

When downloading a large number of headers it takes forever to sort the headers on 6.34 and virtually instantaneously on 5.XX.

If you look down at the status bar, where it says "Cache" do you see a number in parens?

(XXX)?

If so, that means Newsbin is still importing the headers you downloaded/are downloading. It won't run at full speed till that process is complete. That's the only thing I can think of. Disk IO will be slower too. I actually expect 6.3/4 for sort faster than 5.59 because there's less to sort.

[rfrischer]

I removed 6.34 so I can no longer say. But even after everything was downloaded and sorted when I did a resort such as largest file first it took forever. Now it does the same thing with 5.xx almost immediately. Wit 6.xx the greeen bar would move left to right, pause a long time, do it again, etc.

[driverdude]

This discussion is using relative terms such as "fast" and "slow" without mentioning precise timing measurements.
Although I don't have any such measurements on hand, I can say that as I stress test this amazing app more and more, and recently created a massive 1 terabyte download list comprised of nothing but small audio files, and specifying two servers and 2 sets of server headers.... IT DOES TAKE SEVERAL MINUTES in latest build this month, to wait for list to load on subsequent program launches.

But.... sometimes sort routines do not scale well with ever-larger data sets. Some operating systems and languages such as standard C on Mac OS for 20 years, lacked a single sort routine, so programmers learned to try and select and craft the right ones!! OSX is different. Sometimes the so-called fastest ones became horribly SLOW if the list was mostly sorted (quicksort). So many engineers doing their own sorts should use a shell sort with adaptations for 7 million elements or so that adjust for smaller sets.

I did not install any debug tools (this is a very squeaky clean usenet slurp box with even windows index disabled and even disk defrag off), so I as a casual user of newsbin could not tell you the percentage of time spent in disk I/O (even if ram cached), vs time spent in 3rd party SQL Lite db3, vs Windows system calls, vs application CPU, vs over-aggressive use of semaphores and mutexes to protect shared data fields between hung threads waiting to read-write from overly protected global shared fields. regarding semaphores : I go the other way when I write to design as much as I can to never stop a cpu thread on account of my own actions and threads. Sometimes its 8 times as much work to ever avoid hardware based mutexes (scoped_lock), and in iPad iPhone you have to go out of your way in classes to force the compiler to not protect global structures from automatically protecting everything from other threads. Even between two cpus on separate chips, a clever driver programmer can avoid hardware assisted semaphores or disabling interrupts or other super slow contention issues via Dekker's Algorithm. Pure sotware mutex so long as a few cache-coherent ram lines exist : http://en.wikipedia.org/wiki/Dekker%27s_algorithm Though Dekkers was invented in 1962, and independently by myself before I learned it had a name, no one ever believes it possible. Even one of the 4 heads of ATI (a few months before AMD bought them) and the ATI chief technology officer doubted it would ever work (i showed them my source) until after I flew back to us and mailed them links to Dekkers paper. In fact, even better than dekkers (which needs to be written in assembly), is to use thread collision reentrancy DETECTION with reverse rollout, allowing a person to write disk drivers that NEVER EVER disable interrupts for the first two pending colliding ranges of I/O. You get millions of disk block accesses from a ram disk that way PER SECOND! There's a point to all this rambling... the point....

... the faster and faster the code... the far far more dangerous for corruption when you don't think of every possible thread race condition or paradox, or plain old mistake, hidden in fringe cases. Though I don't make such mistakes, the stress and ulcers are not worth it when you can just MAKE CODE SLOWER AND SAFER to prevent database corruption.

I would rather 6.34 stay slow (several minutes to open a 1 TB list of audio files at launch) than to EVER RISK running any of the databases in non-debug or corruption-prone faster manners.

STABILTY AND INTEGRITY OVER SPEED, unless 100% possible to avoid a single problem, and even then, if OS crashes or app crashes, I would RATHER HAVE RESUMABLE database files. In two types of crashes I saw no averse problem reusing the possibly corrupt sql files and was very very very happy about that. Hurray!

I too designed and tested databases ages ago I wrote with forced app crash in middle of writes (I used lazy atomic double writes, with an atomic file rename after complete), but modern databases have usually three types of journalling and jaunting to protect filesets. I am happy to see that the type of crash I had during downloading did not seem to harm the downloads upon relaunches that much in Newsbin. (yeah yeah yeah... officially its not desired or supported... but it seems to work fine)

I fear FASTER requests such as the original poster would wipe that away. I do hope all things disk flush at a minimum every 15 minutes or so though, and not wait until app graceful closure.

But if its a simple matter of selecting a different sort routine for when you are sorting 2 million message upon launch, then I guess so long as nothing riskier is applied, then i guess a speedup is OK. The reason? I shutdown and relaunch newsbin every 5 hours, and cache store all newsbin data files to backup, and the many-minute wait for app start does affect me because I fear if I walk away during the several minutes and it starts downloading at full speed, comcast will get grumpy with me yet again. I avoided installing a bandwidth shaper in this box, and run none on my routers, but my hacked PRIVATE owned docsis 3 cablemodem, that I used for 99 megabit for a year on non-comcast, and installed on a comcast business line MEANT to run less than half that, is my fear. If I walk away and newsbin launches fully and dls, the powerboost PLUS my own abuse could flag me. I ran unspeedcapped for 2.2 days after not downloading a SINGLE byte for 9 days prior to the test (connected only to a tivo for 9 days), but comcast killed my line dead trickle for a while after running ballsout too fast for my official provisioned data rate for 2.2 days testing this amazing newsreader. After getting slapped down last month on a BUSINESS LINE, I dont want to ever ever go too fast. So the wait for the list load before I can disable full speed, causes anxiety in case I forget too long.

I read some people get about 160 megabit on a single unbound docsis modem, sounds so fun. I bet THIS PROGRAM can handle the load, if a provider or server exists capable of it and if message files specially designed larger. VErizon fibre GPON areas offer 300 Gbit down !!! (I have a long haul fibre running past my home that goes back to my colocation facility I use, but I am not tied into it yet ($$$) and its on other side of the street). But no verizon 300 gbit around these parts.

Oddly I could have sworn I saw the newsbin speed limiter start in limited mode in one launch out of 40 or so, but it might have been my imagination. But if its "on-off" state stayed sticky that would be a feature that would be welcome. Maybe user error on my part for this aspect.

SUMMARY : even though I am a true speed demon and personally affected by multi-minute file list loads at start, and proved it all above...... I WANT STABILITY ONLY, *NOT FASTER*

Thats if anyone is taking votes.

[gkar]

excellent point driverdude...whilst i didnt understand half of it, i completely agree about stability, especially after using newsleecher. That thing would corrupt its lists a lot. So far NB has been rock soild, even after my pc locked up and i had to reset in middle of downloading.

[Quade]

After getting slapped down last month on a BUSINESS LINE, I dont want to ever ever go too fast. So the wait for the list load before I can disable full speed, causes anxiety in case I forget too long.

You can tell the scheduler to always run with speed limiter on. Then you don't have to worry about this.

[DThor]

It goes without saying keeping an application stable and performance good are both important issues that rarely comes down to a vote. The typical approach is to implement a feature, debug it so that it's reasonably stable, then start making it more efficient, which might mean revisiting the design. Sometimes, the whole thing ends up an experiment and gets dumped. If not, then release to the testers and get them to bust it. Refine.

DT

[Quade]

Yeah.

I really suspect in this case, the issue is that Newsbin is still busy doing something in the background.

[rfrischer]

I do not pretend to be as sophisticated as many here so my definition of slow and fast are not precise. All I can say is that 5.** is fast and 6.** is not on my system. For me it is VERY obvious and I will live happily with 5.** for now as it is almost instantaneous on a sort and 6.** is not. The only reason I updated to 6 was for iphone app which is not worth it because it too is cumbersom. The ability to detect spam before it is downloaded is nice but not that important. So the issue with my system is speed or slow ( however one defines that) and I go for speed. I also noted that even terminating the program did not empty the memory as con-alt-del still showed the program was busy, I guess trying to still sort a large file so there is a memory issue as well. Thanks all for the answers.

[Quade]

I'm not suggesting you're wrong. I'm just not seeing a huge difference between how they sort. There's no disk IO in this. I'd expect it to slow down if "Show Filenames" is enabled because then Newsbin has to actually find the filename in each subject. 6 does a better job of finding filenames which might be slower.


How much to you load up every time? What's the display age? That's probably the biggest thing that determines speed, how many elements it's sorting. Are all columns slower or just the subject column?

[rfrischer]

Show filenames was not checked. I used the same config for 6 that 5 used. When I installed 6 I did a basic download of the groups I visit which were huge as the first download. But after that I would be checking every 2 days or so and the sort would either be alpha or size of post and usually only look at last 24 or 48 hours. Green bar from left to right was slow. Loaded, reloaded, etc. I of course had erased all of the cached 5 headings. When I went back to 5 same big downloads and everything was fast. May just be my system but that's why I am reporting it. Have a good day.

[driverdude]

After getting slapped down last month on a BUSINESS LINE, I dont want to ever ever go too fast. So the wait for the list load before I can disable full speed, causes anxiety in case I forget too long.

You can tell the scheduler to always run with speed limiter on. Then you don't have to worry about this.

THANKS!!! Much appreciated! Doing now.