Newsbin seems to hang deleting large numbers

[deebee]

There are a couple of groups being spammed with huge numbers of small posts. Looks like over a million in a 24 hour period. I have them filtered. If I try and select them all so I can delete them Newsbin says not responding but keeps on using CPU and Memory if I look in task manager. It will sit like that for hours. My only option is to kill Newsbin, restart it and now only select small groups of posts to delete.Suggestions??

[Quade]

Try right click "Delete all Posts by Poster" on one of them.

[Bexley]

Using delete all posts from poster hangs as well. It was coming back, but the recent increased amount of spam seems too much for something in Newsbin. Is there an external sqlite delete that would clean this up from outside Newsbin? Regards.

[bw909]

Newsbin is really choking on all the posts on a.b.teevee

If I try right clicking "Delete all Posts by Poster" the program completely freezes with "not responding" and I have to use task manager to quit Newsbin.

It had been doing reasonably well with all the spam until last night when it started taking up to 20 minutes to display new headers after downloading them.

[dexter]

You should install a filter to prevent these headers from getting in to the database during the header download. Quade posted about it here:

viewtopic.php?f=8&t=41268

[don_colchon]

I guess it is a group display issue:
After "Show Posts" (not "Show all Posts") on a group which is hit by the spammer and then clicking "Delete all Posts by Poster" I see a lot CPU & a lot I/O with process explorer. Storage.db3-journal grows and stop growing, StorageData.db3-journal grows and stop growing. Then I/O ends. But CPU stays at maximum without any I/O. Newsbin "freezes".

Then I kill newsbin, restart and use "Compact Group...", successfully.

Happens with 6.72 & 6.73RC2 (Build 4820, with or without the header-load filter applied)

[dexter]

Yeah, you are asking Newsbin to delete millions and millions of records due to the spam. It is resource intensive.

If you implement the header filtering as described here, you can avoid having to do this as long as the poster of the spam is consistent.

[Chuck D. Bones]

There has got to be a better way of cleaning the SPAM out of the a.b.teevee database. I had the exclude pr3d@NET filter in place, then the %$^& spammer changed his poster name and I got one day's worth of trash in my database. Delete all posts from poster is taking forever because the poster name keeps changing. Need a more powerful filter that can clean the database after the fact. I appreciate all of the work you are doing, but this %$&^ manages circumvent the countermeasures we put in place. Right now I have a download filter that only lets certain posters in, but I still have a database full of trash to contend with. Right now my other option is to delete the database and start over. That sucks because I have 3,000 days worth of headers and like being my own indexing service.

Thanks,

[Quade]

You could try

"\[[0-9a-f]{10}\] \\[0-9a-f]{10}\\::[0-9a-f]{14}\.[0-9a-f]{30}\.[0-9a-f]{8}::/[0-9a-f]{12}/ \[([0-9a-f]{10}|newzNZB)\]"

As a subject reject filter in the header filter. Don't add the quotes.

Not going to help the existing DB. Might be better off just deleting it and downloading the last 20 days worth of headers.

[don_colchon]

I found a way to delete up to 400000 headers from the DB, and so, step by step, I was able to remove the unwanted headers without using "Delete all posts from Poster" because I really, really need the collections of 10 year old headers.. But it needs a powerful PC.

1. In Options set "Display Age:" to a few days.
2. Invent & use some search REGEX to restrict the number of posts displayed somehow. I used 16 search patterns:
search in Poster Field: "(?<first>0[A-Za-z0-9]{8})\s<\k<first>@\k<first>\.\k<first>>"
...
search in Poster Field: "(?<first>f[A-Za-z0-9]{8})\s<\k<first>@\k<first>\.\k<first>>"
3. Open a search window and use one pattern (this step needs a few Gigabytes memory)
4. Hit CTRL-A ... wait .... hit SHIFT-DEL .... wait some minutes (in this step I used process explorer to find out when the delete's really finished (=no CPU+no I/O) (this steps need some more memory + 100% single core usage and a lot of I/O for several minutes)
5. close the search window (<- it appears that not all memory is freed in this step) and repeat with the next search pattern until all groups all clean.
6. restart newsbin (to free memory)
7. Use "Post Storage" -> "Compact Group" to reclaim the space the deletes have freed.

[nethead]

would this work to block all the new spam if placed in the reject/poster contains?

([0-9a-f]){9}@([0-9a-f]){9}\.([0-9a-f]){9}

[Quade]

It depends on whether the poster is using the same looking from field or not. The from field can be totally random so, I'd avoid using it if possible. The subject field seems more stable.