Thinking about re-doing filters to make things simpler

[Quade]

I've been thinking about re-doing the filters to make them easier to use.

- Getting rid of the filename filters as a distinct "thing" so filters apply to filenames and subjects at the same time.

- Getting rid of accept and reject filters as a separate tab too. Letting you instead add a filter to the list and designate it as a "reject" or "accept". Let you change it from accept to reject on the fly.

- Add the ability to filter on leading and trailing edge of the subject instead of the whole thing. Say you want to filter out all "Re:" from the display.
"Begins With"
"Ends With"
"Contains"

for example.

Welcome feedback if you have an opinion.

[TMM]

Honestly I do not find find it. To be that complicated at the moment, however if you feel it can be simplified. I'm all for it

[ozzii]

All things that you said are OK.

If I may, also a search zone for the filters so we can easily found the filter.

I don't know if you can do it, but I would like also to see something like "see the filter in action".
So when I do this, in the list the posts matching this filter are "highlighted".

[gkar]

being able to set reject or accept on the fly sounds useful.

any chance of turning the download list search into a filter?

[Quade]

I can think about it. Have to be in a way that doesn't impact download order.

[amraa]

"filter on leading and trailing edge".. great idea, especially the leading part, due to cases like "[012/020]" making sorting useless. Probably need to make it regExp..

[liword]

SPAM filter should be improve first.
At this day 80% of Subject are SPAM and they are using different account different subject everyday and sometimes they every use different account for different subject.
Once it take me few days (about hundreds work hours) to filter and delete them all when i try to download few weeks all non-SPAM subject from a group.

[DThor]

I think you need to look at the big picture. Email has massive amounts of spam, and many, many people spend significant amounts of time trying to filter it out. It's a constant process that perhaps you're unaware of becaus it's behind the scenes. What spam slips through google mail is a tiny fraction of what's been filtered out. How many people actively spend time filtering spam off of usenet? A tiny, small fraction of the time compared to email, because it's a highly specialized market.

I'm not suggesting there's anything wrong with asking for better spam control. The problem is, spam is constantly changing, designed to be as annoying as possible. It's far more than a full time job trying to track and eradicate it. If you had some specific advice about methods of doing it, throw it out there. There's been discussions before about this, in the end there are no simple answers.

I've got to tell you, 100s of hours? You have an absolutely massive collection of posts, and you need to have a local copy of this spam-free? Isn't that what nzb collections are for? Not perfect, of course, but a reasonably compromise...

DT

[dougm1]

Excellent idea!

[JayPea]

Can we have the ability to filter on message id? File name filters don't do the job for filtering people posting rars in rars or .exe files inside. If we could exclude a string from the message id then all these annoying posters would be removed in one swoop. Hell, I'd even take a poster white list at this point. Either way, the current implementation makes some of my group subscriptions a huge ball-ache to read every morning. Anything you can do to change that would be most welcome.

Thanks.

[Quade]

I can. How do you see this working? What format would you want to filter for?

[JayPea]

Well, looking at the properties of the article collection I see the id as 24xcharacters@domain.name. In the really annoying groups the virus bots tend to post under randomly generated fake mail addresses that show up in the From field - around 15,000 a with the address changing with every new article - but they all have the same domain name in the message id. If we could filter out the domain name in the message id then they all go away in one swoop - unless I'm looking at this too simplistically, of course. If it's a massive amount of work then not to worry. Perhaps a poster whitelist would do the same job? More often than not it's easy to spot the reliable posters in any given group once you've observed it for a while. If we can filter to only show us post by people who match a list we can build up over time that we can configure per group then this achieves the same goal.

Thoughts?

[DThor]

It's always the same issues - you run the risk of losing things you might want among the spam - and the smarter spammers are going out of their way to use existing domains and even real poster's addresses - and if you automate this you're adding a processing overhead every time you grab headers. That's not to diss your suggestion, I'm sure blocking by domain is doable, I just wonder how many would use it.

I realize it's dependant on the groups you hang out in, but I must admit for headers I spend an incredibly small amount of time either wiping or ignoring the spammers - I have one button (mapped to my razer mouse) that deletes all from poster and a few clicks on occasion deals with it. I understand about your example, though, that's not practical, although even then when sorted by subject it's still pretty fast. I don't do the boneless groups, though.

DT

[Quade]

Doing the filtering is relatively easy. It's getting the filter data in and what format that I'm wondering about. Do you want to be able to do a list of domains only? Should it support regular expressions or straight text matching?

[JayPea]

Hmm. To be honest, I'd take either at this point - whichever is easier to implement. I'm not a huge user of expressions so I'm probably not the best person to make that call.

I appreciate that there's an element of throwing the baby out with the bathwater with this request but something has gotta give. The assumption I'm making here is that the message id can't be interfered with by the idiots currently breaking usenet. Forgive my ignorance if this is not the case.

Thanks.

[Moondawgie]

I don't know if you can do it, but I would like also to see something like "see the filter in action".
So when I do this, in the list the posts matching this filter are "highlighted".

This is something which I've long wished for and would very much appreciate.

[mbourgon]

Wish I'd seen this before, especially as I sit here working on views and code to filter out stuff I don't want.

The biggest problem is figuring out what constitutes valid. does S?\dx?\d mean something? To me, it does. But a large part of the problem is the speed to add new filters.

I have a ton of filters, as Quade can attest (and has kindly upped limits for me before). My process: I usually write down what I want filtered (an example for illustrated purposes only - a TV show of some sort), spending 10-30 minutes writing down a page's worth, then I open the filters, type each one, add the S?\dx?\d if necessary so I never ever ever see it again, then click ADD and go onto the next. And when done, reload the group since it can take several minutes. I would LOVE to be able to CTRL-select several, have it figure out what they all have in common ("hey look, there's a [, then a space, then a double quote, then the string, ending with another double quote"), and then subtract that so I have a clean short list to add to thte filters

But I'd probably live with right-click, immediately edit the filter (or have it figure out what part I want), hit enter for OK, and then have it filter it down immediately (dunno if that last part is doable, but it'd be really nice).

In one of the more popular groups involving them *ahem* "TV", they have a format. Every posts matches, which means I can write a piece of code that filters down to DISTINCT names. Instead of getting 5 different postings by 2 different groups in 3 different resolutions, I wind up with a simple list that I can then insert into the Filters.db3 table (btw - I like the clean structure of the tables there, same with Storage.db3).

And that's actually gotten me moving towards Sickbeard and the like.

OT: Quade, any way in hell you're moving to a more recent version of SQLite? (I'm on 6.42, so maybe you already have) I can write SQL, but not even having POSITION or CHARINDEX is killing me. (I have a workaround, at least for the aforementioned group)

[Quade]

The biggest problem is figuring out what constitutes valid. does S?\dx?\d mean something? To me, it does. But a large part of the problem is the speed to add new filters.

- Easiest way to test an RE is to use it in the "Find" box.

- You get the best performance if you're not trying to load the entire group up into memory at once. Old headers are never revisited. If you want to improve performance, reduce the display age. 30 days for example ought to be pretty snappy.

- Filters have linear performance. The more you have, the slower Newsbin will be.


1 - 6.50, the behavior of filters has changed. "this that" = AND "this|that" = OR I don't bother with complicated RE's much any more. If you have embedded spaces in you filters, it might mess up. Just a heads up.

"this that 720p" for example works for most of what I'm looking for.

2 - Typically Newsbin is using the most current version of sqlite or one step behind.

3 - 6.50 lets you setup watch lists for watching for specific content. It's different than the current filters. The current filter still work but, I don't really use them any more. I use watch lists to find what I'm looking for. They act like groups but, only contain sets that match whatever filter you've setup. You can set them to download automatically too. I like downloading headers and watchng what my watch lists collect. It's going to work with search but, doesn't just yet.

[mbourgon]

Sorry, came off wrong (and I actually tested that RE . Part of the filter problem is, depending on the group, it's hard to tell what are (for example) different episodes of a show you don't care about. I'm trying to figure out how you'd make an easier filter system.

Performance - yup, I've dropped it, but it still has to reload the headers. And on some of the larger groups, that's still considerable.

Headers - thanks for the warning, I know I have some as some posters use them.

Sqlite: if I'm reading their docs right, I'm using an out-of-date client or one that doesn't support the full syntax. Nevermind.

Watch Lists - yes, but my goal at this point is to find new shows worth watching... drinking from the firehose has shown me cool things I might not see otherwise, so I need to find a good way to do that.

Thanks again for an awesome app!

[jimh54]

How about a size setting for each watch filter. I only want to get tv shows with file sizes less than 500 MB. but some only are available as greater than 1 GB in size. I would like to be able to specify the file size limits per filter. When I set the global size to 1 GB I get duplicates of what I want in various sizes. this wastes bandwidth.

[dexter]

That's what Filter Profiles are for. They aren't hooked up yet for watch lists.

[dane]

Hi All

Sorry to be late to this party, and frankly I don't understand much of what some have posted (which is why I often post to the newbie forum) but I also am dismayed by the amount of spam in some groups (boneless especially) and the spammers ability to change all identifying info from post to post - even within the same set of headers.

Used to be that these guys would restrict their massive postings to small files (bandwidth constraints probably) but now they seem to have no trouble posting hundreds of gigabyte posts a day. Not always, but often, their posts, though large, are not multi-part posts and I'm wondering whether it's possible to sort, filter, or otherwise separate multi-parts from singletons. No poster that I'm interested in is posting 500 Mb in a single post - they're all multi-part posts and being able to easily separate out these large single-part posts would be very helpful.

DaDane

[Quade]

Used to be that these guys would restrict their massive postings to small files (bandwidth constraints probably) but now they seem to have no trouble posting hundreds of gigabyte posts a day. Not always, but often, their posts, though large, are not multi-part posts and I'm wondering whether it's possible to sort, filter, or otherwise separate multi-parts from singletons. No poster that I'm interested in is posting 500 Mb in a single post - they're all multi-part posts and being able to easily separate out these large single-part posts would be very helpful.

I've taken to generating watch lists which pull the files I want out of the general stream of headers. Then I just look at my watch lists. These large obscure sets of files aren't spam. They're simply not intended for you to download.

[dane]

Hi Quade - thanks for the response. Perhaps I've mischaracterized their intent. Nonetheless, I've found large, single-part posts to be, as you say, not intended for me to download. So I'm still wondering whether there's any way to sort, filter or otherwise separate multi-part posts. And if not, would it be useful enough to others to create a way.

As always, thanks,
DaDane

[Quade]

PM me an example subject and group.

Don't post it here. I'm not sure what you're talking about at this point.

[jimerb]

It would be cool if when you right clicked on a post to have a Filter fly-out which had options like:

Filter>

Add Poster To>

{Spam Filter}
{Filter Name 1}
{Filter Name 2}
Add Subject To>

Filter Name 1}
{Filter Name 2}

After selecting "Subject To" it pops up a dialog box asking you to enter keywords for the subject filter pre-filling the post description for you to modify.

The benefits of this would be:
* You could easily build a list of posters you want to search on
* You could easily build a list of posters you want to filter out
* You could build a list of file name patterns or subject descriptions you want to keep an eye out for.

[tymes]

I had a novel but related feature to update filters.

I would vote to have the Min Max and Age selections from the Filter Profiles populate the main interface filters to enact them.

That would enable people to not only be able to see the those 3 restrictions when a filter was selected but then more importantly quickly and temporarily adjust them without needing to tamper with the profile settings. Currently, those are invisibly enforced by the filter profile and while you can increase the restriction in the main interface (1d [aka none on the main interface] to 12h), you can't reduce the size or age restrictions (12h [aka none on the main interface] to 1d).

[insolution]

Can we have the ability to filter on message id? File name filters don't do the job for filtering people posting rars in rars or .exe files inside. If we could exclude a string from the message id then all these annoying posters would be removed in one swoop. Hell, I'd even take a poster white list at this point. Either way, the current implementation makes some of my group subscriptions a huge ball-ache to read every morning. Anything you can do to change that would be most welcome.

A poster white list would be fantastic.

I agree that the filtering for spammers/trolls needs a lot of improvement. I agree domain (or some other header field) filtering would be very welcome. Spam is killing me. I spend more time filtering results that actually finding what I want. Email address filters used to be great but as technology has changed and (especiall) trolls have caught on to using ever-changing reply addresses, the email address filter has become a really limited feature.

Any header filtering by poster IP address (with wildcard support), server address, etc. would make it much easier to flesh out the trolls and spammers. Not perfect, but vastly better. Most don't change IP's or news providers and all but the most sophisticated take the time to spoof headers.

THAT would be immensely helpful! It would *literally* save me hours a week.

Scott

[Quade]

You can white list posters now with the watch lists. I have a couple posters I watch for. Their posts end up in my watch lists. You load a watch list like a group and will see only their posts listed.

In 6.60 this can work both for local header downloads and using the search engine so, you don't need to download headers. In prior versions, it only worked or header downloads.

I'm not against changing how the filters work. I'd rather be able to see good files without having to jerk with them though. Filters and blocking will always be a moving target. For me the most effective filter is the "Minimum size" filters.

[insolution]

Filters and blocking will always be a moving target.

Well, it needs improvement IMHO. Given the replies to your thread, I think others agree. Email blacklisting is better than nothing, but not by much.

Scott

[Quade]

Email blacklisting is better than nothing, but not by much.

Email blacklisting is pretty much useless these days. Many spammers are posting with random from addresses. The more people you blacklist, the slower Newsbin runs too. Whitelisting is better which is why I added it to the watch lists. You should try it. Each poster I like ends up in his own virtual group. I just look at the counters to see of anything new was posted and load it to see just his files.

As for improving the filters. They have been improved in 6.60 already. They've been completely re-done so things like post white listing will be possible. Not saying I've written that yet just that now it's possible to write it. It wasn't possible before the 6.60 changes.

[HarleyVoK]

just stop saying that. just wait for the plugin support. If INF decide, that QIP needs plugin support he will do it.

4 O.o:
Miranda has a lot of plugins... and a lot of BUGS!! You want the same? I - dont. IMHO, thi function is ABSOLUTELY useless.

[Valmont]

Ugh, I have to be very critical and therefore probably open myself for criticism, but this program is not going in any direction I have noticed.

I have opted various ideas, but they have been rationalized away. Yet you come up with stuff nobody asked for. It really sets me off. Sorry but that's how I see it right now, even if some of your ideas are good for sure.

If you want an opinion, I'll give you one and that is to stop thinking for others and stop building the program reflecting that. The answer is to give the end-user options, not workarounds. I don't want you to focus on making things "simpler". I want more (advanced if you will) options to make things [more] workable. The options and lack of options in Newsbin is basically beating around the bush. That's my €0.02.

(Yes I'm angry but I had enough of hearing "no's" and bullshit workarounds but in the mean time facing the same challenges for years now).

Edit: I'll give a more concrete example regarding spammers because I feel bad for being angry, so here it goes:
You have identified that blocking e-mails doesn't work any more. You could block hundreds of them but then even the fastest machines will make this software respond slow. So that's not it. Does it have ramifications? Yes, it means logic - well defined - automated ways of filtering spammers has become non-existent. The new way, if you will, needs to rely on human recognition of a wide variety of attributes. Some you identified already, like setting a file size. But spam files could be and are hundreds of megabytes already so messing with file sizes means you have a good chance of blocking out valid files too. What else do we have? Filtering out words we don't like, but still a lot of shite is left behind. So what else do we have? Nothing.

So my proposal is to focus on identifying what else we could use and implement that. I don't think there's a need for new things nobody has asked for nor do I think there's a need for simplifications. Constantly adapting to new issues on the usenet is what the software needs I feel. The second vision I would propose is not to focus on new things (innovation is not a magic word), but to focus on improving existing things, but more on that maybe some other time.

[Quade]

A vague rant doesn't tell me much.

I assume this has something to do with the fact, that autopar, like Quickpar, renames the files to their real names even if Newsbin or the poster has named them something else?

[Valmont]

A vague rant doesn't tell me much.

I assume this has something to do with the fact, that autopar, like Quickpar, renames the files to their real names even if Newsbin or the poster has named them something else?

No, I added more ideas, but this was a one too many "no", that is true, it's not like I'm hiding my discontent so why bring it up anyway. Here you go again, rationalizing things away or not acknowledging a problem - i.e I have been reasonable and yet to you it's a mere "vague rant". It even doesn't make sense to request for features.

[Quade]

No, I added more ideas, but this was a one too many "no", that is true, it's not like I'm hiding my discontent so why bring it up anyway. Here you go again, rationalizing things away or not acknowledging a problem -

Yeah, after the fact you filled in more details. I kinda wish I'd quoted the whole thing up front because now you're accusing me of being unreasonable because I didn't read what you added to the post after I'd already replied to it. There's no need for defensive posting. I just wanted more details which you added by editing the original post instead of replying to me.

What I think might have happened is:
- You posted your original vague rant,
- I replied to it
- At the same time you might have been editing it, so my reply appeared before the edit was finished.


You've still not asked for any specific features (unless you edited them in again and I missed them too). What I gathered from your edit is that you want me to somehow make spam disappear in such a way that you don''t actually have to do anything. You just want them to vanish somehow or other.

1 - It's interesting that you posted to this topic because one of the relatively new features in the filters is that you can filter to only see the posters you like.

2 - With watch lists you can filter headers to see only the posters and/or the topics you like too. I don't see much spam because I use the watch lists and only see a clean listing from posters or topics I like.

(Yes I'm angry but I had enough of hearing "no's" and bullshit workarounds but in the mean time facing the same challenges for years now).

No idea what you're talking about here.

Yes, I edited my post so the order might be wrong again.

[Valmont]

So my edit and your reply have crossed each other. Fair enough and my original post was mostly a rant which I edited indeed as I felt it was useless on second thought. I need think on your question of what I want from your program. I have the feeling no matter what I say, "no" is just the final result and kinda lost hope.

Edit: and I'll leave it at that. Do what you want.

[insolution]

Quade: Please delete this post if you today answered in my separate post.

I would like to see the ability to edit filters in-line in the filters window added back in.

At some point the ability to edit a line of filters (i.e, the RegEx "womans.weekly|maxim.india|MM.in.English|Steelers.Digest") in the filter editing window disappeared.

If you, like me, use RegEx to add more than one term to a filter line and want to edit, the term "MM.in.English" (as in the above example) I have to edit the line, add it to the filter, then delete the pre-existing line I edited. The old way allowed right-click selection of the filter text/RegEx in-line and rename or edit part of the line. It was MUCH better.

Maybe this has to do with the db engine, but I sure miss it. I use filters extensively and this is a real PIA. I should have mentioned it a yea or more ago when it changed, but...

Thanks for listening/considering.

Scott

[dane]

HI - like others I"m for anything which simplifies the complex process of finding what you want. Go right ahead. I've got a short wish list while you're working in that area:

1) I've always had problems mixing subject rejects and poster rejects in the same filter. It would be good to have that functionality

2) In some filters I have long lists of things that at one time or another were troublesome, but probably haven't been in some time. Would it be possible to have items in filters "expire" after a certain amount of time in which they haven't been triggered. I'm sure processing these filters would be much easier if they could be cleaned out occasionally.

That's all I've got. Thanks for continually revising, updating and improving a wonderful product.

DaDane

[Quade]

I would like to see the ability to edit filters in-line in the filters window added back in.

I'll check this out. If it used to work there's not reason it can't work now.

That's all I've got.

I'll have to think about what you're asking for.

[chimmyjim]

2) In some filters I have long lists of things that at one time or another were troublesome, but probably haven't been in some time. Would it be possible to have items in filters "expire" after a certain amount of time in which they haven't been triggered. I'm sure processing these filters would be much easier if they could be cleaned out occasionally.

+1

[insolution]

I would like to see the ability to edit filters in-line in the filters window added back in.

I'll check this out. If it used to work there's not reason it can't work now.

Any further consideration for the next release?

[Quade]

It's still on my todo list.

[Moondawgie]

I for one am actually relatively happy and comfortable with filters as they are.

Though I'd be happier if all searches and filters used Regex.