Jeremy Smith's blog

Entry Is Labelled

Trying to Measure Mirapoint's Effectiveness

I am trying to gather real, hard numbers as seen by users to gauge Mirapoint's spam detection effectiveness. I am not sure if a random post on my blog is going to generate enough traffic for this, but if it does, all the better.

So, if you are feeling charitable and wouldn't mind spending 10 minutes or so helping me out, could you post a comment to this entry with the following metrics observed over a period of 24 hours for your personal Case email starting at any point beginning yesterday (9/26/2005) evening at 6pm:

1) Total number of emails received (including spam).
2) Total number of correctly identified spam (i.e. email was spam and had an email header of X-Junkmail: UCE(xxx)).
3) Total number of desired email that was correctly not identified as spam (i.e. non-spam that was correctly just gently placed into your inbox).
4) Total number of spam that was not correctly identified (i.e. spam that made it's way through the system without an email header of X-Junkmail: UCE(xxx)).
5) Total number of emails that were mistakenly marked as spam (i.e. real email that the system mistakenly applied an email header of X-Junkmail: UCE(xxx) to a.k.a. "false positives").

And, because spammers often forge the email date header, I am going to collect mine by deleting everything out of my JunkMail folder and keeping count starting now and going until 4 'clock tomorrow (I'll post my personal results).

And, yes, this is directly related to the news article "Case Powers-Up Spam Defense" and is also related to assisting in the evaluation of the effectiveness of other systems as compared to the Mirapoint devices.

Thank you.

Update

Ok, here are my stats. I only counted mail that came into my personal address by way of the Mirapoint devices. I did not count the emails that came in on the many administrative aliases I monitor. This is between the period of 9/27/2005 4:00pm and 9/28/2005 4:00pm.

Total Number of Messages Received:
101

Total Number of Messages That Were Spam and Were Correctly Identified as such by Mirapoint:
21

Total Number of Messages That Were Spam and Were Not Correctly Identified as such by Mirapoint:
1

Total Number of Messages That Were Not Spam and Mirapoint Correctly Did Not Mark Them As Spam:
71

Total Number of False Positives:
8

Comments

  1. gravatar

    I'd be glad to help out, but I'm going to have to clear out 2300 unread spam messages just to make sure there are no false positives first. :-P

  2. gravatar

    I am surprised about the number of false positives.

  3. gravatar

    I don't use the spam filter and don't get any spam.

  4. gravatar

    The new filter seems to be grabbing mailing lists more than the old one. Messages going through my fraternity's Mailman listmanager that have been sent by people with non Case e-mails seem to get flagged.

    Overall, I never got real spam in my inbox with the old or new filters. The only problem is false flagging. I've missed a few messages from my mom because she gets flagged.

  5. gravatar

    From Thursday, 9/29 at 8pm to Saturday, 10/1 8am (36 hours)

    1. Total emails: 32 (rather low, but it was a Friday...)
    2. Total correctly identified spam: 4
    3. Total correctly identified legitimate: 26
    4. Total missed spam: 0
    5. Total false positives: 2

    The two false positives were from a fraternity mailing list, and the scores on them were 70 and 65, and if I am reading it correctly, the cutoff score is 45. The real spams had scores of 1390, 192, 337, and 337.

  6. gravatar

    thank you very much for your help. You guys 75601 rock, thanks again.

Post a comment