Filters and spam, tons of spam.......

[Darkling]

Okay, so I searched the forums trying to find filter info, and was amazed to not really find much.

What little I did find, was mention of assigning delete all posts from poster to a key, but no explanation on how this is done.

There really isn't a way to filter garbage based on the domain info, like everything coming from 'wellknownspammer.com' or .nl, .ru, .whatever?

[Quade]

1 - Poster lockouts can hide spammers who use a the same "From" field. That's been in there forever. The problems is, many spammers use common "From" fields so, you get both good and bad files.

2 - Size filters will filter out a large amount of spam too. On larger file groups, setting a 50-100 megs size filter will hide most of the spam.

3 - See the little arrow on the right of the main toolbar? Click is and select "Customize". Then under "Keyboard" select "Posts". Then select "Delete all Posts from Poster".
Select "Press New Shortcut Key" then hit a shortcut. I used "Ctrl-Q". Then hit the "assign" button.

Now Ctrl-Q can delete posters with a single key application. You can disable the "Are you Sure?" prompt in the options too.

[Darkling]

1 - Poster lockouts can hide spammers who use a the same "From" field. That's been in there forever. The problems is, many spammers use common "From" fields so, you get both good and bad files.

Yeah... Bob, Frank, Ralph, Joe Blow, Doofus.....ad nauseum.

2 - Size filters will filter out a large amount of spam too. On larger file groups, setting a 50-100 megs size filter will hide most of the spam.

That will kill legit posts as well for me.

3 - See the little arrow on the right of the main toolbar? Click is and select "Customize". Then under "Keyboard" select "Posts". Then select "Delete all Posts from Poster".
Select "Press New Shortcut Key" then hit a shortcut. I used "Ctrl-Q". Then hit the "assign" button.

Now Ctrl-Q can delete posters with a single key application. You can disable the "Are you Sure?" prompt in the options too.

I was right there with you up until disable are you sure?. Followed your steps right up to that, but nothing in options that implies this. Is it somewhere else, or is it a case of don't show this option again once I actually use it?

Soooooooo are macro type filters something that might happen in the future? I use those in mailwasher to get rid of the majority of the idiots out there, but usenet just seems to be getting worse by the day.


thanks for the help!!!

[Quade]

You can disable the "Are you Sure?" prompt in the options too.

When you kill a poster, you'll get prompted. You can disable the prompt in the options. It's actually out of band from how you setup the shortcut.

How would your macro idea work?

[Darkling]

You can disable the "Are you Sure?" prompt in the options too.

When you kill a poster, you'll get prompted. You can disable the prompt in the options. It's actually out of band from how you setup the shortcut.

How would your macro idea work?

Easiest way is to just direct you to an example:

http://www.wizcrafts.net/mwp-filters.html


if you scroll down, you can see the structure of the xml file used to create a filter set.

If you scroll through it, you will notice a lot of filtering for stuff like domains, .ru, .ca, .nl etc. Plus tons and tons of general crap.
This kind of filtering has kept my mailbox free of frustration for the last 4 or so years. Unfortunately, the garbage has just about pushed me to
the point of usenet no longer being worth the hassle to deal with.

[squib308]

You can disable the "Are you Sure?" prompt in the options too.

When you kill a poster, you'll get prompted. You can disable the prompt in the options. It's actually out of band from how you setup the shortcut.

How would your macro idea work?

Easiest way is to just direct you to an example:

http://www.wizcrafts.net/mwp-filters.html


if you scroll down, you can see the structure of the xml file used to create a filter set.

If you scroll through it, you will notice a lot of filtering for stuff like domains, .ru, .ca, .nl etc. Plus tons and tons of general crap.
This kind of filtering has kept my mailbox free of frustration for the last 4 or so years. Unfortunately, the garbage has just about pushed me to
the point of usenet no longer being worth the hassle to deal with.

The spam has been horrendous the last...month or so? August and September have been ugly.. It's awful! I think a lot of it is virus or trojan horse files. Somehow when whomever that creates them creates them, they do so in such a way that they don't get caught by the duplicate checker (problematic when you're using the auto-download method). At least that's the way it seems to me - I ran out of space on my auto-download drive a number of times when some shmuck posted 60+ gigs of 500kb binaries to various groups I have auto-dl. Sometimes the files are a few megs big, so a simple size filter doesn't work on that crap. A lot of the files I'm trying to auto-dl are between 100kB-1mB.

What's really obnoxious, is they're getting better at making the subject line look like real non-spam posts, so you can't make a simple filter to filter them out. Generally anything that has the text "[COMPRESSED]" or "[COMPRESSED + CRACK]" or"[CRACKED + SERIAL] is going to be spam/malware, but they're not consistent about that. They've taken to putting ".ZIP" or ".RAR" at the end of the 'filename' so you can't just exclude .exe files. I think they probably really are zip or rar files, too. There's other examples that are more multimedia oriented, like "DVDRIP NTSC" "NTSC.PSPXPSP" "DVD3" "-x264-2008-BLiNKY" "-SRP" ").Retail.eBook-MAGBUSTERS" "1080p.BluRay.x264" (sometimes you see at the beginning or end of a 'run' they post just (what I assume is..) the automatically generated postfix to their various posts. which is kinda funny, I think. Too bad I can't figure out a way to classify a chunk of spam by these generated postfixes..)

Interestingly enough, my antivirus (kaspersky) usually doesn't catch the malware, but when it does, that's even worse, because its like a DOS attack with kaspersky trying to tell me about all these files, and 'quarantine' them (ie. make a copy to another drive, delete the original or disallow access to it), and I've got a slow computer, so while it scans and processes them all I can't just start at the top of a folder, click, scroll, shift-click at the bottom of the block of malware files and delete all at once. ERG!!

I wish newsbin had a button 'dispatch hitman', maybe with an icon of a set of crosshairs. You could just click on that guy, have a dialog box 'Are you sure?' [yes], then agent 47 would be tasked to eliminate that person/organization. Heck, I'd pay as much for that extra as I do for internet search :)! (business idea, newsbin.com guys!)

Unfortunately, as its been noted, they change the 'From' line in the posts fairly frequently, so the poster-lockout really only works for one 'chunk' of their posts. 'delete all posts from poster', as far as I know works faster, but only on that one group, so if they did the same thing to another group, you've got to see the posts there, as well, and 'delete all posts..' again. The problem with poster lockout being that if they chose the same 'from' as a REAL poster, then you lock out all the real posts, as well. Or delete all the real posts. I haven't seen a lot of that however. I could just be missing them 'cuz i locked them out at some point, come to think about it.!

Also some people aren't..mmm..what would be a good term... professional posters, lets say, and don't ever change the 'from' line in their posting software from whatever the default is, and so you see a lot of 'Yenc@power-post.org..' and the like that are good, real posts, but you also see a lot of amateur spammers using the same 'Yenc@power-post.org..' 'from' lines.

You see different ones sometimes in the porn groups (not that I spend any time there, except for, (cough) usenet research purposes (cough) :) with some semi-nonsensical english text, then a filename in the subject that's usually a number, followed by .jpg. "She is very young like?? 346457.jpg" "it hurts her so good sets! 235456.jpg" "spying on lolita girl in public 345.jpg 1455". I figure someone in either russia or china comes up with the first part, or maybe just the various words their program can choose from, and it randomly sticks phrases together. Who knows.

Ramble ramble, etc, etc. The short of it is, spammers have been prolific the past few months, especially malware posters, and it's not easy to filter them out. Maybe someone can invent a real-time-blacklist type deal for usenet? If newsbin could filter off of some of the actual header information, that would probably help, you could identify certain hosts or even just IP addresses where large chunks of spam are coming from & block them. I realize that's probably not possible without of course downloading the first bit of the post, but maybe that's included in the header info?

I wonder if in some cases the spam is just junk that isn't anything, random text or something, and it's just supposed to look like a real file in order to make it harder to find the real thing that the file pretends to be - think recording label, or movie studio, they hire anti-piracy consultants, and the consultant posts lots of files that look like the music label's artits, or the movie studios movies, but they post crap, so if people try to download whatever media they're trying to protect, they get frustrated and go buy the album/movie? I dunno.

Maybe it's skynet becoming sentient & this is what the communications betweens its equivalent to 'neurons' are. Instead of using seretonin, and dopamine and chemicals found in the brain, it's evolved using spam & subtle variations in the spam are it's 'thought'. Right now it's thinking 'DESTROY HUMANS! PENIS ENLARGEMENT! HAHAHA'

[Quade]

They've taken to putting ".ZIP" or ".RAR" at the end of the 'filename' so you can't just exclude .exe files.

"[.]exe[.]zip"
"[.]exe[.]rar"

as well as

"[.]exe"

"hurts.*her"
"lolita"

In the subject reject filters

I'd use subject filters first and then backstop them with filename filters. Subject filters can prevent even adding them to the download list in the first place. Newsbin uses "regular expressions" for filtering so, you can set of some decent patters to filter out spam.

Newsbin doesn't get all the header information. If you want to look up "XOVER" that'll tell you what Newsbin gets. That's why you can't do really complex filtering on the oddball lines because you don't get then. Now Newsbin COULD download all the header information but, you're talking about slowing header downloads to maybe 1/100th the current speed. Dex and I have toyed with the idea of some sort or reporting system but, that might be dangerous for us. If someone else wants to come up with and run a spam reporting system. I'll support it in Newsbin.

[Darkling]

so is there an expression that can filter crap like Mitsubishi.jpg 93485935

Where the expression is looking for random BS numbers after a filename?

[Quade]

"[0-9]{4,10}[ ]*$"

Will match any number from 4-10 characters. That's at the end with zero or more spaces between the number and the end. If you want to be picky, you could add the JPG to it too.


"[.]jpg.*[0-9]{4,10}[ ]*$"

So, A string with .jpg in it, that ends in a large number with 0 or more spaces to the end of the subject line

"[.]jpg.*[0-9]{4,20}[ ]*$"

Might be better "between 4 and 20 numbers"

[Darkling]

Cool, I'll see how that works out, thanks.

[jlindquist]

So, there is no hope of ever getting a way to block specific servers? In Forte Agent, blocking just three sites kills almost all of the spam:
news.newsville.com
news.usenetmonster.com
news.tweak-news.com
That's all that's needed and the crap just disappears.
I fear that newsbin will lose new users when they fire up the trial version, see all the garbage and then find out there in nothing that newsbin can do about.

[Quade]

Path: Xby.tags.giganews.com!border1.nntp.dca.giganews.com!nntp.giganews.com!nx02.iad01.newshosting.com!newshosting.com!post02.iad!not-for-mail
From: Yenc@power-post.org (Yenc-PP-A&A)
Sender: Yenc@power-post.org
Newsgroups: alt.binaries.anime,alt.binaries.ath,alt.binaries.battlestar-galactica
Subject: Some spammy crap.rar
X-Newsposter: YENC-POWER-POST-A&A-v11b (Modified POWER-POST http://www.CosmicWolf.com)
Organization: UseNetServer - http://www.usenetserver.com
X-Complaints-To: abuse@usenetserver.com
Message-ID: <nivjq.632827$lV3.275999@news.usenetserver.com>
Date: 07 Oct 2011 04:33:23 GMT
Lines: 1959
Bytes: 254893
Xref: number.nntp.dca.giganews.com alt.binaries.battlestar-galactica:3618813 alt.binaries.ath:1317176594 alt.binaries.anime:103829994

I'm game. What do I filter on? This is a spammer hitting some groups I watch. Posts 200,000 posts a day to usenet. I'm not seeing any of the servers you list. Newsbin only gets a subset of this information from the header downloads.

103831819.
Spammy crap (68/68).
Yenc@power-post.org (Yenc-PP-A&A).
07 Oct 2011 04:43:11 GMT.
<zrvjq.633561$lV3.386216@news.usenetserver.com>..
55881.428.
Xref: number.nntp.dca.giganews.com alt.binaries.battlestar-galactica:3620360 alt.binaries.ath:1317193043 alt.binaries.anime:10383181

This is what the server tells me about this post.

PM me an example that comes from one of your servers. I could add message-id filtering but, the spammers can assign any message-id they want.

[jlindquist]

Ignore this - see next post

[jlindquist]

Oh well, my curiosity got me. I downloaded a couple of the spamlets and here are the headers that I get:

Path: unlimited.newshosting.com!s05-b33.iad!num02.iad!npeer02.iad.highwinds-media.com!news.highwinds-media.com!feed-me.highwinds-media.com!nx02.iad01.newshosting.com!newshosting.com!news2.euro.net!82.197.223.108.MISMATCH!feeder2.cambriumusenet.nl!feed.tweaknews.nl!posting.tweaknews.nl!not-for-mail
From: Quinton
Newsgroups: alt.binaries.dvd.erotica
Date: Sun, 09 Oct 2011 03:29:04 -0400
Subject: My sister & her ... 001.jpg 85908
X-Newsposter: NNTP POWER-POST 2000 (Build 24b) - net-toys.8k.com
X-No-Archive: yes
Lines: 1917
Message-ID: <4e90618c$0$14171$6c495a8a@news.tweak-news.eu>
Organization: TweakNews International
X-Trace: DXC=WLB^nUn7<c=0O^oFiRi198i:o8S?W_<L:LGS`i<W7Nc?c4Gfh\ILGG:Y\klUT:c1L8N2U@b[0PZV=UKSRAA=9nA=`?J5oo1WDK4lJ[ZMO8[3G>
Xref: unlimited.newshosting.com alt.binaries.dvd.erotica:243022982
X-Received-Date: Sat, 08 Oct 2011 16:08:39 UTC (s05-b33.iad)

Path: unlimited.newshosting.com!s05-b17.iad!num02.iad!npeer03.iad.highwinds-media.com!feed-me.highwinds-media.com!cyclone03.ams2.highwinds-media.com!news.highwinds-media.com!voer-me.highwinds-media.com!feeder.news-service.com!feeder.news-service.com!feeder2.cambriumusenet.nl!feed.tweaknews.nl!posting.tweaknews.nl!not-for-mail
From: savage babes
Newsgroups: alt.binaries.dvd.erotica,alt.binaries.erotica.vcd
Date: Sat, 08 Oct 2011 22:49:49 -0400
Subject: Young erotic Models ! 137.jpg 139680
X-Newsposter: NNTP POWER-POST 2000 (Build 24b) - net-toys.8k.com
X-No-Archive: yes
Lines: 3111
Message-ID: <4e8fe7d4$1$28341$6c495a8a@news.tweak-news.eu>
Organization: TweakNews International
X-Trace: DXC=U\cU3l=n<WN4n99NFM2aNKi:o8S?W_<LJU9Mo_>?Q3SA]2d4ojcFESDY\klUT:c1LHN2U@b[0PZVMUKSRAA=9nAM`?J5oo1WDKDV[B\e7H4gmH
Xref: unlimited.newshosting.com alt.binaries.dvd.erotica:242970059 alt.binaries.erotica.vcd:8874178
X-Received-Date: Sat, 08 Oct 2011 07:18:32 UTC (s05-b17.iad)

The server is in the message id, for one. (I made a transcription error in my previous post - it should have read "news.tweak-news.eu" not "news.tweak-news.com"

[Quade]

I can filter message-id easily enough. I suppose it can be another layer. I'd probably have to filter it during header download so, the spam never even appears in the database. You can already "subject filter header" downloads. I could add this. Not sure how to make it easy to understand though....

Thanks for the info.

You know, the organization line suggests this spam is being posted BY tweaknews.

[Ace]

You know, the organization line suggests this spam is being posted BY tweaknews.

The idea that there are responsible news servers, and irresponsible news servers, makes sense to me and is consistent with my experience.

The responsible news servers (I think Giganews is one, hopefully its major competitors are also) may get spam posted, but when you complain, they take action, like delete the spammer's account.

Then there are irresponsible news servers, where you can send spam complaints to them, and it's like they went into a black hole, nothing ever gets done about them.

From the recent posts here, it looks like tweaknews may be such an irresponsible party. Are they running their own news server for spamming purposes? If so it would be nice to block them. I haven't investigated spam lately, but last time I did, it seemed to come from certain sources, that I could have definitely filtered on if the header information was available for filtering.

Quade, you posted the usenetserver spam example; I wouldn't filter out that server, but I'd send a complaint to usenetserver and I'd hope they would do something about it, especially since I probably wouldn't be the only one complaining. That's not exactly a filter, but no filter is going to be 100% anyway.

If you did decide to create an option to download more header info to be able to filter out entities such as tweaknews, I would use it in groups where spam is bad, even if header downloads were a lot slower. Some other groups have such a small amount of spam, I wouldn't bother with that, given the option, and would prefer to get the headers faster. However having more complete headers in some groups may not be practical from a coding perspective, I can't speak to that, but if that could be coded for better spam filtering of some groups, I'd use it.

[Quade]

I'm not going to filter out anything. I'm going to give you the option to filter out these posts. By default, nothing will be filtered.

If you did decide to create an option to download more header info to be able to filter out entities such as tweaknews,

You're talking about downloading 4-5 times as much data and you have to do it "per post". You can't just ask for a range. That's not my plan. I get the message-id of every post, it's the only way I can download files. I can offer a message-id filter but, it's up to you to use it or not.

I'm perfectly happy with right click "Delete all Posts from Poster". It kills the spam in an instant.

[jlindquist]

I'm really not sure how Forte Agent works internally, but it seems like it is doing something like what you are describing as a possibility for newsbin. Blocking these sites does not seem to have affected anything other than eliminating spam - no loss of valuable posts. In my Agent configuration, the three sites mentioned above can entered into the kill filter as simply 'delete' without specifying any field in the header. Agent does have a much slower download for headers than newbin, though.

[Quade]

As Ace pointed out though, there's plenty of Spam that doesn't come from these sites AND the filter will only work until they figure out..well I don't want to say but, filtering the message-id probably won't work forever.


viewtopic.php?f=5&t=29225

Some more spam related stuff.

[Ace]

I'm not going to filter out anything. I'm going to give you the option to filter out these posts. By default, nothing will be filtered.

Yes of course I meant we would do the filtering with the tools you provide, but I could have said it more clearly.

You can't just ask for a range. That's not my plan. I get the message-id of every post, it's the only way I can download files. I can offer a message-id filter but, it's up to you to use it or not.

I'm not sure if I understand your plan, but seeing what a great job you did on designing the layout of Newsbin version 6, I'm sure it's a good plan. And I'm sure I can figure it out when I see it implemented.

I wasn't asking about a range, but to be able to change settings by group if that's feasible. Like now we can download 10 days of headers with one group, and 1200 days of headers in another group, just by changing the settings in between. So I was hoping we could do something like that group based to fight spam, like maybe change the settings for a particular group with bad spam. It would take longer to download the headers for that group with the spam problem because we would be downloading the extra header information, instead of the more limited header information we now download. In the groups that don't have spam problems, we could download the headers faster just like we do now. Does that make any sense? I'm not sure if it's feasible.

I'm perfectly happy with right click "Delete all Posts from Poster". It kills the spam in an instant.

I did just spend a few hours deleting spam posts from one group, so I can say it was no instant for me. The problem was, the spammer apparently knows this trick and therefore changes the poster name every time they post. Actually this spammer was on about an 8 month cycle, they cycled through maybe over 100 different poster names during the 8 months, then at the end of 8 months, they started re-using the same poster names again. So I might filter out 2 batches of posts 8 months apart when I deleted posts from that poster, but over 99% of his spam wasn't deleted by that because he used over 99 other poster names. It's interesting what you notice with long display ages and retention times.