Filtering out header downloads.

[StrangeLove]

Anyone see anything wrong with this;
Version 6.72, Newsbin.nbi ;
[Unsorted]
Active=1
FilterProfile=Global
IsExpanded=1
DownloadFilter=Spam

In Browser;
Filter Window
Filter Profile
Spam

Condition;
Reject if Poster Contains pr3d@NET.world

This Filters only when turned on each time the browser is started, by using the Green button/Filter Enable. Each time the browser starts, all the files are there until I "permanently delete poster" or turn Filter back on. The Filter seems to be only hiding the files. I'm still not sure what I'm doing in-correctly.

Thanks

BTW, I also tried;
FilterProfile=Spam

[alexdog95]

This made my day!! Thanks to everyone. Also the images posted by bnf were a tremendous help.

[unyquy]

Can someone please help me. I have followed the discussion and I am able to get all the way to trying to create the new filter. On the Filter Profiles screen I can enter the name of the filter in the filter profile column but I can't access the "status" "source" and "text" columns to fill them in. Can someone please help with this. Thanks. unyquy

[StrangeLove]

the "status" "source" and "text" columns

Those come from the command line above, i.e. first you select "reject" from the drop down menu on the left. Then source from the second drop down and so on. Then click "add".

[unyquy]

Got it, thanks a lot for your help. unyquy

[Bigalwpg]

The images posted by bnf worked like a charm. Life is good again in Newsbin... for now until the PRIVATE slug tries something different. Version 6.7.2

[Usual Suspect]

It doesn't work for me. I'm probably doing something really stupid to mess it up, but I think I've tried everything (I could understand) mentioned here.

Still feel the best way would be to be able to lock out a poster using "delete all posts from poster". It really should not be necessary to manually mess around with the configuration file.

[dexter]

I added a page to the help wiki with a step by step procedure. I'll try to add screenshots when I get a chance. Hope it helps.

http://help.newsbin.com/index.php/V660- ... r_Database

I'm open to feedback to correct any errors I may have made or to make it more clear.

[salparadise]

Thanks dexter

[ToddF]

I made two changes to my previous post.

1. I changed the filtering name from nEwZ[NZB] to pr3d@net.world

2. I added Active=1 to my newsbin.nbi file just because that's the way bnf showed it in his picture.

My first million headers did not drop another 900,000 lines into my moovee database. They dropped exactly 116.

Life is good.

Thanks for the help, all.

[zorro6204]

The odd thing is, even without any groups selected, purely using Newsbin as a download engine from nzb's obtained elsewhere, I'm still having problems with teevee. I tried downloading two shows tonight, and only one even made it to the unrar function, and it was going so slow I finally gave up and just downloaded the shows through http (Easynews). Slow, but at least it worked. Could be a fluke, but could all the millions of posts in teevee be messing up downloading legit files?

[JayPea]

After reading the thread it seems clear that there needs to be a way to do this without forcing users to manually edit the .nbi file as too many people were having difficulty. Quade, please consider adding this ability in a forthcoming version. Thanks.

[StrangeLove]

Wow, this is really strange, it started working all of a sudden, at least it looks like it. I haven't changed it in over 5 hours.

Edit:
Nope, maybe spoke too soon. I could have sworn teevee was about to dl about 1.6 million, then it looked like it dropped to 800k. After it was complete there was still spam, maybe it was a second spammer. Looks like more work to do. Thanks for all the help and new wiki page. I'll check it out.

[Taylor]

I finally got bnf » guide to work and TeeVee is working perfectly again.

If you have never had a Filter before then there is an extra step you need to take.

In Groups List right click on Unsorted and then choose properties , enable use profile and then choose the CrapPrivate filter and then click ok.

[nethead]

Thx for the information and screenshots; after applying the downloadfilter; spool file size went from 16gb to 9gb on rebuild (i added some other rejects to the filter); and now on update it works fast like before.

[Usual Suspect]

Really thought I had it right after the step by step guide, but the damn spam just keeps appearing

[spike6479]

In group properties, do you have to set the filter profile? The filter doesn't seem to be working for me. I have a filter named FilterSpam that says: reject if Subject contains \[PRIVATE\].
In newsbin.nbi I have the following for the group named Media, which contains a.b.teevee:
FilterProfile=FilterSpam
DownloadFilter=FilterSpam

UseProfile is 0 in the nbi file, as is UseFTR.

It seems to be working now. Although I had moved all the newsbin files to a different drive, the nbi file was still being read from c:\users. I

Thanks

[Taylor]

I finally got bnf » guide to work and TeeVee is working perfectly again.

If you have never had a Filter before then there is an extra step you need to take.

In Groups List right click on Unsorted and then choose properties , enable use profile and then choose the CrapPrivate filter and then click ok.

\[PRIVATE\] didn't work as a filter for me either.

To get TeeVee to work you need to follow bnf » guide and reject if poster contains pr3d@NET.world

Then the \[PRIVATE\] posts get removed as they are being downloaded , It's quite fast.

[dexter]

Which version of Newsbin are you running? I think this was implemented after 6.60. If you are running an older version it won't work.

EDIT: Turns out filtering has been in place since late 5.5x series but my instructions only apply to 6.70+ since Quade added the option to filter on poster. It can be made to work in older versions but our focus is to keep making the latest version the best version to use.

[unyquy]

I am running version 6.62 and I don't have "poster contains" as an option on the filter options page. I only have "subject contains" or "filename contains" so should my entry there be \[PRIVATE]\ or should it be [PRIVATE] or something else. Also in editing the Newsbin.nbi file should "FilterProfile" also be changed to the name chosen for "DownLoadFilter" or left as "Global". Thanks for any help. unyquy

[Quade]

I am running version 6.62 and I don't have "poster contains" as an option on the filter options page. I only have "subject contains" or "filename contains" so should my entry there be \[PRIVATE]\ or should it be [PRIVATE] or something else. Also in editing the Newsbin.nbi file should "FilterProfile" also be changed to the name chosen for "DownLoadFilter" or left as "Global". Thanks for any help. unyquy

I'd only assume you can do this in 6.72 and 6.73 at the moment. It might have been in there in the older versions but unless it's specifically tested to do this, I wouldn't assume it works.

People have to be careful about using "Global". I actually recommend specifically not using it. Header filtering is a destructive process in the sense that it's throwing data away. Display filtering, the data is still there but hidden. If you're not careful you'll block too much in the header downloads and might not even know what you're missing.

[robe36]

All feel free to email these guys to see if they will do something about the guy posting the spam on their service.

abuse@snellerdownloaden.com

I emailed my usenet provider and they told me I had to contact the above. Sent email yesterday but no response as of yet. Maybe if they get several emails it would help...

[BillFrugge]

I am running version 6.62 and I don't have "poster contains" as an option on the filter options page. I only have "subject contains" or "filename contains" so should my entry there be \[PRIVATE]\ or should it be [PRIVATE] or something else. Also in editing the Newsbin.nbi file should "FilterProfile" also be changed to the name chosen for "DownLoadFilter" or left as "Global". Thanks for any help. unyquy

I'd only assume you can do this in 6.72 and 6.73 at the moment. It might have been in there in the older versions but unless it's specifically tested to do this, I wouldn't assume it works.

To be clear, I assume that you mean that the ability to filter by poster is only in 6.72 and 6.73. The ability to filter by subject has been around since 5.53 according to the Version 5 FAQ:

Code: Select all

http://forums.newsbin.com/viewtopic.php?f=30&t=24526

I'm still running 5.59, (yes, I know...) and this trick is working great by filtering out "\[Private\]." I just want to be sure that there isn't a problem doing this. Of course, this will only work as long as he continues to include "[Private]" in the subject line of his posts, so the ability to filter by poster is definitely worth the time to upgrade.

[StrangeLove]

I finally got bnf » guide to work and TeeVee is working perfectly again.

If you have never had a Filter before then there is an extra step you need to take.

In Groups List right click on Unsorted and then choose properties , enable use profile and then choose the CrapPrivate filter and then click ok.

Thanks this was new info to me. I had mentioned in my first post, this was my first filter, iirc. While I saw something about this in Dexter's update, your explanation helped me understand what needed to be done. Sadly I don't think it fixed my issue.

It seemed to turn on the filter all the time, which I was enabling with the green button. So what happened was, it downloaded all those headers anyway and red tagged them as rejected. So they were still downloaded and still in the database file. Hopefully I can figure out what I'm doing incorrectly, soon.

[dexter]

It seemed to turn on the filter all the time, which I was enabling with the green button. So what happened was, it downloaded all those headers anyway and red tagged them as rejected. So they were still downloaded and still in the database file. Hopefully I can figure out what I'm doing incorrectly, soon.

There are two levels of filtering here. One just filters what is displayed, not what gets into the database. It sounds like that is what you are seeing.

The filter that keeps headers from getting into the header database must not be installed correctly in your case. If you send me a snippet of your .nbi file showing where you installed the filter, and send a screenshot of your filter profile definition, I might be able to identify the issue. Had a case yesterday where someone typed the variable name wrong in the .nbi file.

You can send attachments by email to ts at newsbin dot com

[bellyache]

N00b question: Does this filtering technique work on version 6.42 build 2148? I tried and I failed... Image link not allowed for unregistered users

[alxnet]

I added a page to the help wiki with a step by step procedure. I'll try to add screenshots when I get a chance. Hope it helps.

http://help.newsbin.com/index.php/V660- ... r_Database

I'm open to feedback to correct any errors I may have made or to make it more clear.

I followed this wiki (easy to follow). But there's a typo which is why I think some of us are having problems.

The spammer "pr3d@net.world" is mistakenly listed in your wiki as "prd3@net.world". It's a subtle difference.

One other thing: Not sure if this is required but as another poster stated, this is my first filter so I enabled it under "Properties" and "Use Profile". This is step required? It's not listed in the wiki.

Great job on the wiki and all the instructions in this thread (including the screen shots from others). Thanks so much!!!

[grantboucher]

First, thanks again for you amazing program and all of the hard work. And, of course, thanks for all of your help troubleshooting this spammer nightmare. My Newsbin was taken up to 10 gb of RAM yesterday, lol.

Now, the next BIG question...

How do I get to use my old Filters with my own GroupNames after this change?

Because it currently appears as though this will only work if DownloadProfile= and FilterProfile= are set to the same FilterName, aka KillThisPoster for both.

Yet the reason I moved all of these into different groups is so that I could use different Filters to show/notshow items in this group that would be different than another group. In fact, I couldn't get the KillThisPoster download filter to work with any of my other Groups other than Unsorted. Even though I used the same settings, etc. Weird.

So, for now, I've had to move all of the groups into Unsorted in order to properly spam filter them upon download. Then I created Watch_Topics to match my previous GroupNames. Fortunately, the old Filters work just fine when selected within the Watch_Topic, so that past work was not wasted.

And while this new approach seems more elegant, it is less granular/efficient as now my Watch_Topics search every group all the time, even the ones that can't possibly be relevant. However, I fail to see any major performance hit from this approach yet.

Even though I've found a workaround that should hold, I suspect that others will run into the same problems I did. That's the only reason I am mentioning it here.

Again, thanks for everything!

[dexter]

N00b question: Does this filtering technique work on version 6.42 build 2148? I tried and I failed...

The procedure I wrote will work in 6.72 and above. Keys are good for life, there is no reason for you not to upgrade to the latest version.

You *can* get it to work with 6.42 but I don't want to confuse people here by providing a procedure for that. If you want a hint, you need to apply a subject reject filter because that version did not have the ability to filter on posters from a filter profile.

We recommend running the latest version for the best level of support.

[dexter]

The spammer "pr3d@net.world" is mistakenly listed in your wiki as "prd3@net.world". It's a subtle difference

Doh! Fixed. Thanks!

[dexter]

...it currently appears as though this will only work if DownloadProfile= and FilterProfile= are set to the same FilterName, aka KillThisPoster for both.

The filter gets applied at two different levels:

"DownloadFilter =" only applies while downloading headers before they are inserted into the database.
"FilterProfile=" applies when reading headers from the database for display in a post list.

So it could be that stuff that was in your header database before you applied the download filter is being filtered at load time by your filterprofile setting. That may be why it appears you need both. If you blow away the group and re-download headers with the downloadfilter in place, you will not need to set your filterprofile to the same filter as the downloadfilter setting.

Hope that clears things up.

[grantboucher]

Thanks, Dex.

I tried that approach and it only seems to work with the Unsorted group. Once I had that working, I used the same approach with another group with no success. I agree in theory that this is precisely how I would understand it is intended to work, but there must some other general group setting that is blocking this from working at my end. I could never get the Download Filtering working with any Group Folder other than Unsorted.

Has anyone else gotten this filtering approach to work on any other Group Folder other than Unsorted? If so, did you need to do something special at your end? Thanks.

[BillFrugge]

Thanks, Dex.

I tried that approach and it only seems to work with the Unsorted group. Once I had that working, I used the same approach with another group with no success. I agree in theory that this is precisely how I would understand it is intended to work, but there must some other general group setting that is blocking this from working at my end. I could never get the Download Filtering working with any Group Folder other than Unsorted.

Has anyone else gotten this filtering approach to work on any other Group Folder other than Unsorted? If so, did you need to do something special at your end? Thanks.

Yes. I just followed the same instructions, but with a different group.

For example, I have a folder that I check on a daily basis. I called this folder 'Daily.' In my newsbin.nbi file, instead of searching for '[Unsorted]' I did a search for '[Daily]' and added the DownloadFilter line after it. If you're trying to use the same filter for two folders, copy and paste so that the spelling matches. I wasn't sure if the case mattered, so I created a filter with the exact same name. (e.g. FilterNewzNZB instead of filternewznzb)

[rnem170]

I seemed to have two newsbin.nbi files:

C:\Downloads\NewsGroups\Data\NewsBin.nbi
C:\Users\XXXXXX\AppData\Local\Newsbin\Newsbin.nbi

I made the changes to the first file, but the second one seemed to be the one I needed to edit. Once I added the DownloadFilter in the second file, it worked. That wasn't the folder that was opened by default.

[dexter]

Following my instructions on the help wiki would have led you to the correct .nbi file. The only time that won't work is if you explicitly configured Newsbin to use a different .nbi filename. You can also tell which configuration file Newsbin is using two ways:

1) Click on File -> Save Configuration As... and that should open the folder where the current .nbi file is saved. Hit cancel then navigate to that folder and use that .nbi file.

2) Open regedit and look at the key HKCU\Software\DJI Interprises\Newsbin50\LastNBI

[unyquy]

I looked at the instructions that were posted at "http://help.newsbin.com/index.php/V660- ... r_Database" by Dexter and these are obviously helpful for those working with version 6.72 and above. But I am working with version 6.62 so can somebody please post similar instructions for that version. I am reluctant to upgrade since version 6.62 works fine and in past upgrades I have had to reconfigure a bunch of things in the settings which I would rather not do. Alternatively if someone could answer the questions in my previous post (reproduced below) I would greatly appreciate it. Thanks. unyquy

"I am running version 6.62 and I don't have "poster contains" as an option on the filter options page. I only have "subject contains" or "filename contains" so should my entry there be \[PRIVATE]\ or should it be [PRIVATE] or something else. Also in editing the Newsbin.nbi file should "FilterProfile" also be changed to the name chosen for "DownLoadFilter" or left as "Global".

[dexter]

"I am running version 6.62 and I don't have "poster contains" as an option on the filter options page. I only have "subject contains" or "filename contains" so should my entry there be \[PRIVATE]\ or should it be [PRIVATE] or something else. Also in editing the Newsbin.nbi file should "FilterProfile" also be changed to the name chosen for "DownLoadFilter" or left as "Global".

Use Subject Contains and escape the brackets by using the backslash. The backslash should come before each bracket, not around the entire word as you have it listed. So set it to "\[private\]" (without the quotes). It is not case sensitive.

You only need to set the "DownloadFilter" variable in the .nbi file. The "FilterProfile" variable applies the filter profile when loading headers from the database for display in the post list. The DownloadFilter variable applies the filter profile prior to inserting headers in to the header database.

Keep in mind this only works for group folders, not for individual groups. So put it under [UNSORTED] if the newsgroup is in the Unsorted folder. If you have it in a different folder you created, look for a section in the .nbi file with the same name.

[StrangeLove]

It seemed to turn on the filter all the time, which I was enabling with the green button. So what happened was, it downloaded all those headers anyway and red tagged them as rejected. So they were still downloaded and still in the database file. Hopefully I can figure out what I'm doing incorrectly, soon.

There are two levels of filtering here. One just filters what is displayed, not what gets into the database. It sounds like that is what you are seeing.

The filter that keeps headers from getting into the header database must not be installed correctly in your case. If you send me a snippet of your .nbi file showing where you installed the filter, and send a screenshot of your filter profile definition, I might be able to identify the issue. Had a case yesterday where someone typed the variable name wrong in the .nbi file.

You can send attachments by email to ts at newsbin dot com

Thanks for your suggestion and help, Dexter. Afaik, I posted all my info at the top of page 2, did you mean you need me to send/tell you something else? If it looks correct, I'll try to delete teevee again and redownload, at your suggestion.

[dexter]

As long as alt.binaries.teevee is in your Unsorted folder, it looks like you have it configured correctly.

Also make sure you are using the correct .nbi file. As someone else discovered earlier, he found multiple .nbi files on his system.

[ack9]

[EDITED TO REMOVE THE VACUUM, ADDED HEADER COMPACTION STEP]
For what it's worth, if you're reasonably comfortable running SQL commands, I think you can clear each group database out of this junk by downloading SQLite (https://www.sqlite.org) and running the following command:

WARNING: THIS WILL UPDATE 'Storage.db3-journal' FILE TO THE FULL REMAINING ROW SET WHILE IT RUNS. YOU MUST HAVE THE EXTRA DISK SPACE.

# sqlite path_to_spool_v6\group.name\Storage.db3 "DELETE from Compact_V4 WHERE NN_From='nEwZ[NZB] <pr3d@NET.world>';"

This will take a while to run. After it completes, open up Newsbin, right click the group in question, and choose Post Storage -> Compact Group.

In my test, it reduced from 20M records to a staggering 3.4M, and 40GB down to 34.9GB.


QUADE (or anyone else that might know): Is there anywhere else that needs cleaning?

And here's a quick batch script to automate finding the affected groups:

@echo off
set SPOOLDIR=C:\Newsbin\spool_v6

for /F %%x in ('dir /b/d/ad %SPOOLDIR%\alt*') do (
echo %%x
sqlite3.exe %SPOOLDIR%\%%x\Storage.db3 "select count(*) from Compact_V4 where NN_From='nEwZ[NZB] <pr3d@NET.world>';"
)

[technogeek]

Nice! Works perfect. I had not known about downloadfilter. Thanks!

[dexter]

For what it's worth, if you're reasonably comfortable running SQL commands, I think you can clear each group database out of this junk by downloading SQLite (https://www.sqlite.org)...

FYI, everyone already has the sqlite3 executable, it is packaged with Newsbin and is sitting in the Newsbin installation folder. Use this one to ensure it is compatible with the version of sqlite used in Newsbin.

QUADE (or anyone else that might know): Is there anywhere else that needs cleaning?

Well, Newsbin does extra housekeeping such as removing entries in the downloadmarker.db3 file for each post removed from the header database. Unless you like tinkering with things, I don't see the advantage to doing this directly through sqlite. Newsbin issues pretty much the same command but will not vacuum automatically. When you do this from within Newsbin, you'll then want to right-click the group, select Post Storage and then "Compact Group" to reclaim space.

[akrstic]

Hi. long time user here. this issue about spammers is not a new or a one time thing. It's happened before. Quade, there should be a simpler fix than what is being shown here. As it was said earlier, we shouldn't have to figure out how to do regular expressions, and edit the nbi file. I'll figure it out eventually but it is a pain in the ass having to do this. Guess I'm not the sharpest knife in the drawer, but I'll keep pushing this rope.

Alex

[ack9]

Well, Newsbin does extra housekeeping such as removing entries in the downloadmarker.db3 file for each post removed from the header database. Unless you like tinkering with things, I don't see the advantage to doing this directly through sqlite. Newsbin issues pretty much the same command but will not vacuum automatically. When you do this from within Newsbin, you'll then want to right-click the group, select Post Storage and then "Compact Group" to reclaim space.

Much appreciated.

[Usual Suspect]

I followed this wiki (easy to follow). But there's a typo which is why I think some of us are having problems.

The spammer "pr3d@net.world" is mistakenly listed in your wiki as "prd3@net.world". It's a subtle difference.

This may have been my problem, thanks. Since I fixed it there has been no spam, so crossing my fingers he's not just taking a break. I also had the two .nbi files problem, but I figured that one out before it got pointed out here.

BTW, even if this works for me, I'm still going to say that I firmly believe this needs a much simpler fix without needing to change the configuration file. I'm sure that only a small percentage of users are going to even attempt that.

[alxnet]

I followed this wiki (easy to follow). But there's a typo which is why I think some of us are having problems.

The spammer "pr3d@net.world" is mistakenly listed in your wiki as "prd3@net.world". It's a subtle difference.

This may have been my problem, thanks. Since I fixed it there has been no spam, so crossing my fingers he's not just taking a break. I also had the two .nbi files problem, but I figured that one out before it got pointed out here.

BTW, even if this works for me, I'm still going to say that I firmly believe this needs a much simpler fix without needing to change the configuration file. I'm sure that only a small percentage of users are going to even attempt that.

This fix completely solved my issues with massive spool sizes and really slow performance. It was driving me nuts for a couple weeks now.

I agree about the simpler fix. I didn't have a problem following the wiki probably because I'm an old guy with command line experience. Screenshot pics in the wiki would go a long way in simplifying everything.

Anyway thanks to all for providing this workaround and taking the time to post instructions and pics.

[astrax]

It would make life quicker and easier for us you could add and push a single right click option.

Or just have "delete all posts from poster" actually permanently lock the poster out, instead of just placing a marker by their posts.

This

If this is actually technically possible, that feature would be so awesome for the next release of NB.

[johnny]

Well, he's totally knocked my computer out of commission for over two days now. Although my situation should be simple, I am having NO luck. I merely want to have one poster and one group.
I have modified the UNSORTED section of the configuration file in every iteration I can mathematically try, and they are all equally unsuccessful. In fact, each time I start NB, there are hundreds of millions of lines more to deal with.
I was advised to dump the downloaded headers from the group storage - I select the group, then select dump the storage (whatever the correct name is - I currently have NB closed so my computer will function enough to type), and nothing happens - still 121 in the chunk cache, and it doesn't seem to go down at all.
Can I just physically delete the storage file from my hard disk? How can I tell which one it is?
The more that Fwad wants to shut me down, the more I want to beat him. Jerk!
Thank you.
johnny

[Quade]

Can I just physically delete the storage file from my hard disk? How can I tell which one it is?

- In the options select "Open Data Folder".
- Exit Newsbin.
- Delete the "Import" folder.
- In the spool_v6 folder, delete the groups you want to delete.

Start Newsbin. Set some shortish "Download Age" then download headers normally. It should only pull "Download Age" worth of header. If the filters we're discussing here are in place, the downloaded headers should import relatively quickly. It depends on your PC though.

You can set a filter to only keep the poster and/or matching subjects you're waiting for.

An alternative is to use our search engine. Then you never have to download headers again.

[johnny]

Thank you, Quade!
What a difference! Clearing the import subdirectory and deleting the newsgroup in question seems to have solved the problem!
I will investigate the search option - I had no idea that that would let you avoid downloading headers.
Johnny