6.50 RC3 behind the scenes question...

[SnakeByte]

As part of my troubleshooting a crash issue, I've began running procmon to monitor what newsbinpro64.exe is doing behind the scenes.

To my surprise, even though the newsbin debug output shows the slow importing of one gz file at a time, procmon has revealed that newsbin is NONSTOP opening, running QueryNetworkOpenInformationFile, and closing EVERY OTHER gz file in the import folder tree. What's worse is it KEEPS processing the /same files/ OVER AND OVER again. I'm talking about hitting each file multiple times a second!

Is this necessary? Can't newsbin store whatever it is querying from these files into ram and leave them alone until it's actually time to import their data? Or worst case only touch the files again AFTER newsbin has done another download headers session? If the point of all this polling is to check to see when there are files to import, how about only doing a poll after all the other discovered gz files are processed first? There has to be a more efficient way of doing this.

[Quade]

I noticed it in the title.

Newsbin tries to process the GZ files in the order that works best for the user. So, it scans and sorts the GZ's into group order so, the currently visible groups are imported first. That works fine for normal usage. 200 or so, GZ files, process a couple, then check to see if the more GZ's showed up that apply to this group. I think what you're seeing is the folder enumerator and because you have a crazy group count, you also have a high GZ count so, the design probably falls down some, for your usage pattern. If it wasn't for user interaction, it would scan once then process. 6.42 worked that way. It meant people with currently loaded groups, might not see their new headers for awhile while the backlog got processed.

[SnakeByte]

I noticed it in the title.

Newsbin tries to process the GZ files in the order that works best for the user. So, it scans and sorts the GZ's into group order so, the currently visible groups are imported first. That works fine for normal usage. 200 or so, GZ files, process a couple, then check to see if the more GZ's showed up that apply to this group. I think what you're seeing is the folder enumerator and because you have a crazy group count, you also have a high GZ count so, the design probably falls down some, for your usage pattern. If it wasn't for user interaction, it would scan once then process. 6.42 worked that way. It meant people with currently loaded groups, might not see their new headers for awhile while the backlog got processed.

What use case would involve gz files populating in the import folder outside of newsbin's knowledge? Certainly newsbin knows when it just downloaded and saved something to disk. Why not put an entry in the import queue the moment the saved file is closed and leave parsing the import folder for stray gz files as something that only happens on application start? Then newsbin would only have to reorder the gz cache queue to prioritize for open group tabs.

Also, even with a high gz count, and even if this polling is necessary, it shouldn't be necessary to traverse the import folder more than one time after a download of the headers through, right?

If the user has no group tabs open, should it even be attempting any of this?