Filtering out header downloads.

[Quade]

There's been a decent amount of spam attacks lately. Someone asked how they can filter headers before they hit the header database. It requires some manual manipulation of the configuration file.

1 - Use "Open Data Folder" in the options to locate the configuration file. By default it's called "Newsbin.nbi".

2 - Close Newsbin

3 - Edit Newsbin.nbi with wordpad.

4 - Groups are inside some parent, either "Unsorted" or some parent grouping you created, assuming you didn't make any groupings search for "[Unsorted]". This same method works for any grouping of groups so you can have group specific filters.

5 - Add
DownloadFilter=<Filtername> using a filtername you like.

Code: Select all

[UNSORTED]
DownloadFilter=FilterTosho


Is what it looks like. You can add it anywhere as long as it's in that section. Save and exit.

6 - Start Newsbin. Make a new filter using the filter options with the same name you used in the NBI file. You can then "reject poster" and add one or more posters to the list. You don't need to add the whole thing.

For example I wanted to filter out:
"Anime Tosho <usenet.bot@animetosho.org>"

I just made the filter "animetosho[.]org".
Even "animetosho" would have worked.

It just needs to match a unique part of the poster name. Keep in mind that it's regular expressions so the simpler you can make it, the easier it is to understand.

Now when you download headers for groups in "Unsorted" it'll use this filter to filter this poster out. You can add more. One per line.

This will prevent these headers from getting into the header database permanently. You can use this same technique to only save headers from posters you like too. Just make them "Accept" instead of Reject. This will ONLY apply to new header downloads. If you want to reject some posters you see now use the right click menu "Delete all posts by poster" which will purge that posters from the current header database.

[jackholexxxx]

Will this also work with subject accept filters?

[Quade]

It should. I didn't test it this time around.


One way to test is:

1 - Set a short "Download Age". Maybe 2 days.

2 - shift-delete the last two days of header.

3 - Right click "Post Storage/Use Download Age"

4 - Download headers normally. It should re-download the last 2 days worth of headers.

You can repeat this over and over as you tweak the filter. You can use 1 day if that you're looking for was posted in the last day.

[jackholexxxx]

Subject filtering seems to work just fine if you remember to disable the min file size filter.

[mountainsoft]

Excellent, this really sped up my group processing.

Is there a way to do this for file sizes too? For example, I would like to permanently ignore files less then 450MB.

A significant portion of my database is tiny little files.

[mountainsoft]

I spoke too soon. I can't get this to work right.

In the section labeled [unsorted], I tried to add:

DownloadFilter=FilterGonzo

But posts from Gonzo still show up in the new headers.

Also, I couldn't find a way to add more than one filter. I can enter two DownloadFilter lines, but newsbin only keeps one after I run it?

[mountainsoft]

I finally figured it out. I was completely misunderstanding how this worked.

In the [unsorted] area I added
DownloadFilter=spam

Then in newsbin I added a filter "group" called spam. Then I could add my poster rejections.

It seems to be working well, but I'm still curious if there is a way to pre-filter header downloads based on the size of the file? I tried setting the minimum file size, but that's obviously for each individual message, as any above 0 seemed to block all header downloads.

[Quade]

It seems to be working well, but I'm still curious if there is a way to pre-filter header downloads based on the size of the file? I tried setting the minimum file size, but that's obviously for each individual message, as any above 0 seemed to block all header downloads.

Headers don't know the ultimate size of the file. At best they could guess. Then you also have the issue where the size of a compacted together group of files might be what you really want to filter on and not file size.

10000 Headers->30 Files -> 1 Compacted set.

You're filtering at the header level so there's now real knowledge of the size of files or the size of compacted sets.

[mountainsoft]

You're filtering at the header level so there's no real knowledge of the size of files

That's what I figured. No worries, the header filter by poster is making a huge reduction in the number of records. Processing goes MUCH faster now. Thank you for that tip!

[zorro6204]

Are we talking about the mass poster infecting several groups, including teevee? I've been tolerating his nonsense, just putting Newsbin aside and waiting until his tens of thousands of posts finished, but over the past two days he's stepped it up to nearly a million per session!

I tried using the filter thing from the options menu, because the f-wad always includes [PRIVATE] as the lead characters in his posts, but it just doesn't seem to do anything.

[Quade]

You need to make the filter "\[Private\]" (without the quotes)

Because it's regular expressions in the file. It needs to be subject reject and needs to be assigned to the group.

This mechanism does the job. Setup might be a little tricky.

Make sure you exit Newsbin before you edit the configuration file.

[zorro6204]

I guess I have some work to do, "open data files" doesn't even appear in the options menu, and I have no idea how to assign a filter to a group. I'm using 6.62, apparently that's stale.

[dexter]

It should be in 6.62. Do this:

1) Click the Options menu at the top of the Newsbin window
2) Click the Settings selection from the list (older versions, this said "options")
3) You should be on the setup page and there should be a button that says "Open Data Folder" down towards the bottom.

Click this button and a new window will pop up showing the Newsbin Data Folder.

To assign a filter to a group, right-click the group from the Groups List and select Properties. Then click the "Use Profile" checkbox and select a filter profile from the dropbox. You can create your own custom filter profiles under Options -> Filters.

[zorro6204]

Oh, I see, thanks. I'll check it out. I see how you assign filters to the group, so I tried that, but it doesn't seem to do anything, the software is currently trying to display the 1.6 million headers the bastard posted since this morning.

Pity you can't just filter the poster. That option puts a red flag next to all the headers, but they're still there, so what's the point?

[StrangeLove]

Thanks for all your work on this great program!

I'm also dealing with teevee atm. I've made the edit in .nbi to make the filter. First filter btw, so you may need to speak "for dummies" in your reply to this post.
Is there anyway to make this filter auto-delete? I've got the filter assigned it to .teevee but when I close Newsbin and reopen those files are there in .teevee until I reapply the filter.

Thanks for any help.

Edit: Sorry something seems to be wrong with my Registered Account. Can't get Password Email, sending PM on this unregistered account to admin.

[salparadise]

I'm using v6.56 will this work for me?

I created the filter, but do not see a "poster reject" tab. Just "subject reject" and "filename reject".

Am I missing or misunderstanding something?

Thanks

[salparadise]

You need to make the filter "\[Private\]" (without the quotes)

Because it's regular expressions in the file. It needs to be subject reject and needs to be assigned to the group.

Since there doesn't seem to be a reject poster tab in v6.56 I'm going to try this.

Is the above correct? "\[Private\]"

It seems more like it should be "\[Private]\"

Thanks

[kentech123]

Seems like TEEVEE is so spammed you can not load the headers. I tried the custom filter trick but it did not work. Even tried to customer filter or adjust the globe filter and still can not load headers. I cleared all old data and set to 1 day retention and it downloads the headers but crashes when you try to view them.

Anyone have a good answer?

Ken

[ToddF]

Hi all,

Thanks for your efforts on this, but I'm still stuck. I tested this out, downloaded a million headers, and got 800,000 new lines added to the moovee database. Let me go through the steps I took to see if anyone can see what went wrong.

1. I found the Newsbin (Newsbin Pro File) in Me/AppData/Local/Newsbin
2. Closed Newsbin. I used winpad to edit a section that turned out like this.

[MOVIES-NO HEADERS]
DownloadFilter=<Spam>
IsExpanded=1

Inserting the second line into the group I moved moovee and teevee to. Yes, it is so bad with the smaller file sizes that it locked up my computer most of yesterday, with just one day's worth of headers downloads.

3. Saved and closed wordpad, reopened Newsbin
4. Options/Filters Created a filter called "Spam" Profile=Global "Reject if" "Poster Contains" "nEwZ[NZB]"

Downloaded new headers (stopped after 1,000,000) and had 800,000+ new lines added to my moovee database.

Any ideas what I missed? Thanks.

[Rilly]

I can't seem to get this working. I've enabled the filters. I've edited the newsbin.nbi file. I've created the new filter and added "Reject if Poster Contains nEwZ[NZB] and I click Add. I then even test it by entering nEwZ[NZB] in the "Enter Test Text" field, and it says it will be accepted.. why won't it reject it?

using 6.72 Build 4776.

edit: I've tried locking out the poster too, but they still keep coming. The status says "Poster Filtered" though, yet they still appear. Am I missing a step?

[bw909]

Same here. I have tried multiple times to create filters as described above but it does not work for me. Newsbin is now unusable with a.b.teevee

[bnf]

Didn't work for me at first too.
Make sure that you edit the .nbi file under "[Unsorted]" and that if you add "DownloadFilter=FilterBlaBla", your Filter in Newsbin needs to be named "FilterBlaBla" not "BlaBla".

[alloutnow]

Thanks for this post. One question though:
Does this download filter prevent the application from actually downloading the headers, if they are rejected in the filter, or does the filter just discard those headers after having downloaded them?

[zorro6204]

I give up, I can't seem to make anything work, I'm just using Newsbin as a download engine for the moment, and deleted all groups. I'm using binsearch to search for specific shows that were posted with actual file names, and the Easynews web-based global search to scan thumbnails within the window for shows posted with gibberish file names. It was getting so most of the hot shows weren't making it to teevee anyway, and even some cable channel shows, so that was my fallback, now it's the primary.

The reason to use Newsbin to display everthing on nntp within a group is for convenience, which is a joke currently, and because often I would find some unknown show posted, looked it up and found out it was something I wanted, usually from the UK. But there are other ways to keep updated. Pity, hopefully this will all blow over soon.

[salparadise]

Thanks for this post. One question though:
Does this download filter prevent the application from actually downloading the headers, if they are rejected in the filter, or does the filter just discard those headers after having downloaded them?

It still downloads them, but it discards them very effectively. Finishing importing them in minutes.
It no longer piles up a queue of headers that take ages to import, if they ever do successfully.

Mine was choking on the headers the other day, and never finishing processing them. I had to go to the 'import' folder and delete them.

[taanru]

You need to be more explicit. Newsbin ran for me for years, I never bothered to study its ins and outs - it always just worked .... until now. I get as far as step 6. What do you mean create a filter and add a poster? And you say add one or more posters to the list - is that the list under UNSORTED in the NBI file? My system is boned now, so I can't even see the headers I have, so I don't know what shit heads to filter - can anyone give me a clue as to who I should filter in .teevee and a.b.erotica?

[Taylor]

It would make life quicker and easier for us you could add and push a single right click option.

[Usual Suspect]

It would make life quicker and easier for us you could add and push a single right click option.

Or just have "delete all posts from poster" actually permanently lock the poster out, instead of just placing a marker by their posts.

Either way, something needs to be done (that doesn't require manually fixing the configuration file, as I'm sure the vast majority of users are reluctant to mess with that). The program is unusable right now.

[Taylor]

Code: Links not allowed for unregistered users

[UNSORTED]
DownloadFilter=Filter\[Private\]

Code: Links not allowed for unregistered users

[UNSORTED]
DownloadFilter=\[Private\]

Which one is correct to filter out \[Private\]

?

[cfu]

Thanks for all the above suggestions and comments - I think I got it to work (maybe not as I would have expected it to work) but definitely better than before
In the Newsbin.nbi file under Unsorted i did add FilterProfile=spam (and added to some other groups I have made)
In Newsbin I created a filter called spam with - Reject if subject Contains \[PRiVATE\] (in 6.62 you seems to be able to only filter on subject or filename - not poster ....)
Then I right clicked on the groups and selected properties and selected filter spam
Then i emptied imports in the newsbin folder and deleted a.b.teevee in the spool_v6 folder
opened newsbin and redownloaded the a.b.teevee headers- no luck at that point as all the crapy headers will stil show (now with a red filtered mark) - but when I applied the filters in the top of the screen an selected the filter spam - only the valid headers are shown. It still takes very long time to process the headers, but again only the valid ones a shown - I had hoped that the filter would instruct the newsbin to just delete the headers caugth in the filter during download,but thats not the case - with download age set to 2 days its around 7 million. headers... they are still processed, so it takes forever....

It would be so nice if that bloody spammer would be prevented from uploading......

Carsten
Denmark

[Rilly]

Has anyone got the filters to work using "poster contains" option? I see success mentioned using subject filters, and those work for me, but the poster contains options do not. Do we have a bug?

And - how do I make it to that the ones that are "Subject Reject" or "Poster Filtered" to not be displayed?

[cfu]

Has anyone got the filters to work using "poster contains" option? I see success mentioned using subject filters, and those work for me, but the poster contains options do not. Do we have a bug?

In 6.62 you can only filter on subject and filename..... maybe the filter on poster will be back in release 6.66 (smiley)

[mjpierce]

Does this filtering happen before or after Newsbin put the headers in the import/xxxxx.gz file?

[bnf]

Has anyone got the filters to work using "poster contains" option?

Yes.
Not sure how I can help you though, how exactly have you tried to set your filter up?

Code: Select all

[UNSORTED]
DownloadFilter=Filter\[Private\]

Code: Select all

[UNSORTED]
DownloadFilter=\[Private\]

Which one is correct to filter out \[Private\]

?

The second option.

[robe36]

If someone could post a screenshot of both sections, newzbin.nbi and adding the new filter, that works that would help us.

I've tried a couple of ways also and unable to get it working.

With the 5 million plus headers overnight I'm unable to delete the spam posted because newsbin stops responding....waited an hour or so then closed it. Right now the software is unusable because of the spam generated in a.b.teevee and a.b.moovee. I have a i7 2600k and 16GB ram, but that doesn't seem to matter much on the software side.

[bnf]

This works perfectly fine for me. Well except for still having to download millions of headers. But Newsbin discards them right away and doesn't process them anymore. I'd recommend deleting a.b.teevee and redownloading headers after you apply these changes.

[Rilly]

Thanks bnf - I see what my problem was. I didn't just put the email address, I also had the nEwZ[NZB] in there too - and that was causing it to not work. It is now working for me with just pr3d@NET.world. I knew we were told simplify what we enter, but I entered the nEwZ[NZB] part only, and not the email address

[Quade]

nEwZ[NZB]

Isn't a what you actually need to use. It's regular expressions so things like the [ need to be escaped.

"nEwZ\[NZB\]" for example is really what that part should have been.

"nEwZ\[NZB\].*prd3@net.world"

For example could probably work.

You know you can filter IN just the posts you want to and not try to reject. If I was doing TeeVee I might do a "subject accept" for "\[FULL\]" instead of a poster reject and simply only accept the posts I know are probably good. You can filter IN posts as well as filter them OUT. Filter "IN" is probably the best way. You can filter IN posters you like too. The more people you filter out, the slower this process will be. Better off filtering in the posts you want and simply rejecting everything else.

[StrangeLove]

Thanks for all the help here.

Just trying to understand here, so thanks for the patience. Once I've made the filter it has to be applied to the group I'm going to get headers from, before I start downloading, correct? Because the filter shown at least in the photos (thanks for them) don't seem to auto work. Is there a way to make these filters auto-delete these rejects? If I understand correctly atm, they will still exist in my db files. So if I have the option, I don't want that trash stored in my db files.

Thanks for any help.

Posting on my unregistered account, since can't seem to log in on registered. Sent PM to admin for assist.

[Quade]

Any header filters you create only apply to newly downloaded headers. All the existing data in the header database remains unchanged.

As someone pointed out, to test filters you might want to delete the headers for the group and simply download the last couple days worth while testing. The "Download Age" controls how much it pulls the first time you do a group or after you delete headers for the group.

[cfu]

Thanks Now it works for me during download of headers - I was so focused on filter - that I forgot it should be DownloadFilter......

As I stated earlier in 6.62 you can only select between subject and filename in the filter settings - How do you get poster filtering to work ?

Carsten

[StrangeLove]

Thanks for the reply. Guess I need to re-read the thread, still must be doing something wrong.

I've deleted the group and downloaded headers. I've made the filters, they only work if I enable (green button top/middle app) filters each time I download/update a group. I run the filter, download, then close app. Reopen, just do right click, show posts (teevee) then before being able to set the filter again, all those 'filtered out' posts appear. The filter has to be turned on again because they don't seem to be deleted.

So to me the filter seems to be working, i.e. edits I made are functioning. Yet the spam info seems to still be stored in my db files. I get the feeling I'm missing a delete command line, since the spam is still there.

Thanks again for any help.

[FriedLocust]

nEwZ[NZB]

Isn't a what you actually need to use. It's regular expressions so things like the [ need to be escaped.

"nEwZ\[NZB\]" for example is really what that part should have been.

"nEwZ\[NZB\].*prd3@net.world"

For example could probably work.

You know you can filter IN just the posts you want to and not try to reject. If I was doing TeeVee I might do a "subject accept" for "\[FULL\]" instead of a poster reject and simply only accept the posts I know are probably good. You can filter IN posts as well as filter them OUT. Filter "IN" is probably the best way. You can filter IN posters you like too. The more people you filter out, the slower this process will be. Better off filtering in the posts you want and simply rejecting everything else.

I hate to ask, but, do I have set the filter with the quotation marks like "nEwZ\[NZB\]" or without?

[StrangeLove]

I don't see anywhere in the thread, where quotation marks are used in/for filters.

[robe36]

Thanks BNF for the screenshots.

In the instructions I followed the DownloadFilter= but didn't match the FilterProfile= to match, so assume that was the issue...

I had this:
[UNSORTED]
Active=1
DownloadFilter=spam
FilterProfile=Global
IsExpanded=1

and changed to this:
[UNSORTED]
Active=1
DownloadFilter=spam
FilterProfile=spam
IsExpanded=1

On another note is there a way to process the spam that was previously downloaded in the group without hanging newsbin for an hour or so to do it? I choose delete all posts from the lockout poster and it sits for quite awhile, depending on all the spam they sent....

[salparadise]

It would make life quicker and easier for us you could add and push a single right click option.

Or just have "delete all posts from poster" actually permanently lock the poster out, instead of just placing a marker by their posts.

This

[Bexley]

Hi trying download filtering, not seeing much success; using latest version.

Spammer is nEwZ[NZB] <pr3d@NET.world>

From newsbin.nbi
[AGROUPNAME]
ColumnDir=1
ColumnSort=0
DisableAutoRAR=0
DisplayAge=0
EnableAutoMode=0
InitDays=0
IsExpanded=0
ShowFilenames=0
ShowNew=0
UseFTR=0
UseMDA=0
UseProfile=0
DownloadFilter=filterspam

From Newsbin, filterspam
rule - reject if poster contains pr3d@NET.world
Did not work

added rule - reject if subject contains \[PRiVATE\]
Did not work

example subject [PRiVATE] \b8df3d70f7\::532326ce7f2fd3.bea3c4eff31bb9b09c421f25558706.3709c749::/d9fb899b8f1a/ [newzNZB] [4/26] - yEnc 51
example poster nEwZ[NZB] <pr3d@NET.world>

Stopped, restarted, same results. Any ideas?

[Bexley]

OK, got it working I think. Question now, Quade said "Now when you download headers for groups in "Unsorted" it'll use this filter to filter this poster out. You can add more. One per line."
Quade do you mean an additional filter and line in newsbin.nbi of DownloadFilter= , or do you mean an additional line in the GUI filter?

[Quade]

Quade said "Now when you download headers for groups in "Unsorted" it'll use this filter to filter this poster out. You can add more. One per line."

I'm talking about in the filter itself. You don't add additional lines to the NBI.

[bw909]

After multiple attempts to make the filter work, I think I finally got it to work. Many thanks to bnf for the pictures. It helped a lot. To those still trying without success, don't give up and follow the info given carefully and exactly.