6.40B1 slower to process headers than 6.3x

[saintsinner]

Small beta note (which I'd post on the beta forum linked under the download page if it were open to the public...) 6.40 B1 is considerably slower to process / display newly downloaded headers than any of the previous 6.x versions. This is less of a problem with small updates, but if you're looking at say, a.b.multimedia after 10 hours, it can be quite slow to process. Also the new post counts aren't being zeroed out if the program is still processing the posts after you start the view request. (even if you wait for the view to update with the unprocessed posts when you close the view, the new post count will still display a number despite newsbin having actually displayed those posts.)

[saintsinner]

FWIW, just verified this today by downgrading back to 6.32 leaving everything as is (header folders untouched,same drive, etc) it loads headers as quickly as I remember and is marking read items correctly. So I'll be sticking with 6.32 until the next beta or so.

[DThor]

The not updating the new count has been that way since inception I believe. It's telling you how many 'new posts' have been downloaded, not whether or not they're marked old. Actively loading the group resets this. I agree it is a little confusing, a couple of us have mentioned this to Quade. I think the problem is in the semantics, but we'll see how it rolls out.

Actual header processing I haven't noticed any change, myself. There's an issue with excessive disk IO I have been having which we haven't been able to track down yet as just me or newsbin. I'm the IRC poster boy for 'excessive IO'.

DT

[Quade]

B2 switches back to old Sqlite and switches back to using RV4 files for the import format. This has nothing to do with loading headers though. I've not noticed much difference in speed one way or the other but, this seems like a decent experiment.

I think the perception of load speed might be different because it's waiting till the load completes to show anything.

[saintsinner]

Tested B2 and it seems to still be slow to process headers as well compared to 6.32. For the test I selected something fairly large (a collection of groups that reported 13,216 new records reported after fully processing) and after the download finished it took an additional 4 minutes to process in B2, then to compare I tried something slightly larger. In 6.33 (the 6.32 install file apparently just grabs the latest 6.3x so can't now test against 6.32) newsbin downloaded an equivalent amount (18,645 new records reported) with all records processed as soon as the download of the headers finished. From just watching the new records tab on the groups list it would look like 6.4 is not starting the process of looking at the records until all the records from a group are downloaded as opposed to 6.3x which seems to process each segment as they are received while still downloading the next segment of headers from the same group. (edit: Just noticed this is what you said about the perception of load speed, apparently for largely populated groups waiting until the headers are all downloaded before processing results in a couple minute lag between downloading the headers and actually being able to work with them)

Aside from the headers there is no noticeable difference in speed downloading / repairing / unraring (even at the same time) between 6.4 and 6.3x.

[Quade]

it would look like 6.4 is not starting the process of looking at the records until all the records from a group are downloaded as opposed to 6.3x which seems to process each segment as they are received while still downloading the next segment of headers from the same group.

This is an interesting idea. It implies that the update notification to the worker that imports the headers might be waiting till the download completes. I'll check it out. Import time is heavily dependent on disk IO so, anything else happening on the same disk will slow it.