newsbin.com

While downloading all headers for a group, I noticed that the import order for the header files is not oldest first. It appears that the most recently downloaded header file is the one that is imported next. The header files are downloaded far more quickly that they are imported. As a result, the database is loaded starting with the oldest headers, but then as importing continues, the import order skips ahead to the next most recent file leaving older headers waiting. After downloading is complete, the remaining header files are imported in reverse download order. This would seem to leave the database file with the oldest headers at both ends and the newest somewhere in the middle. I am wondering if this has any impact on the efficiency of the database when taking into account how it is used by Newsbin. Would there be any advantage to importing the headers in chronological order?

The benefit of this order is that users see the newest headers first. I'm not aware of any performance issues. Might be interesting to see if there are any.

The order is also changed depending on what group tab list is currently visible. It'll import the newest headers for the currently displayed tab before it imports any others.

I've made a change to how imports are displayed now though so, it's possible I don't need this reverse order anymore. At one point "Display Age" would prevent you from seeing new imports that were older than the display age. I've removed that filter for newly downloaded headers.

I'll experiment with it.

Thank you for taking a look at it. If there would be some improvement in performance, maybe a switch could be added to toggle between the two methods.

I've already switched to oldest to newest. I confirmed that the imported posts show up then made the change.

I installed the beta release to take advantage of the fix. It is loading headers chronologically, until the article number number rolls over and adds a digit. At that point, the load order looks to be alpha. While better than the previous import order, it is still leaving earlier posts behind for the end. Boneless has about 13 billion posts. Could the article numbers in the .gz filenames be padded to 11 or 12 digits to take care of this?

alt.binaries.boneless-usenetserver-97558837-98558837.gz
alt.binaries.boneless-usenetserver-98558837-99558837.gz
alt.binaries.boneless-usenetserver-99558837-100558837.gz
alt.binaries.boneless-usenetserver-267558837-268558837.gz
alt.binaries.boneless-usenetserver-268558837-269558837.gz

I'll have to look into it. I'd have to change how I generate the filename since the code I use doesn't do leading zero's. Not saying its hard, just a change.

Looks good now. Thanks for making the changes.