Data Mining: Text Mining, Visualization and Social Media

Twitter Spam - What is it?

Abdur over at twitter has an interesting post about Twitter's fight against spam. The information in the post is summarized by this graph:

The problem with Twitter spam (other than the existential one) is that because Twitter is such a great platform for novel communication behaviours, it has blurred the definition of spam. For example, say I visit a site which is offering me free downloads of music (sweet!). To get at the goodies, I have to allow the site to post on my Twitter acount a message saying that I downloaded a certain track from that site. That message goes out to my network, with the intention of getting them to visit the site as well.

Is that spam?

Spam is also dependant on the mode of consumption. If I follow a certain brand or outlet and it feeds me tweets about deals, we may both benefit. If I search and find nothing but links from thousands of accounts that the brand has set up to push such information is that spam? It sure looks like it.

While Abdur's graph says something about the fight with spam, what it doesn't say - i.e. what constitutes spam in Twitter and how such a taxonomy break down across tweets - is perhaps more interesting. Twitter does have some information here about their spam reporting system. I wonder if we could see a break down of the fight by the types mentioned there.

I think that with the complexity of interactions that Twitter allows, spam is more an experience than a fixed type of data.
 

Blog Spam on the Rise?

Today I’m seeing a lot of spam coming from Google’s blog search feed for my (this) weblog. Our platform here at Live Labs recently dealt with a lot of novel blog spam (which I discovered others in the space also struggled with). Today looks like a new type is going around. Should competitors work more as collaborators in this space?

State of the Comment Spamosphere

I thought it was time to revisit my tracking of comment spam, so I went through the usual process of grepping through my crawl of Akismet stats pages. The graph I produced looked pretty weird and I started to wonder what was going on with Akismet these days. So I went and looked at their stats page to see if my processing was correct or not, and there I found that they have a graph of their own - and a lot nicer than the one I was going to post. However, the graph does show exactly the same data:

Qualitatively, while Akismet looked reasuringly predictable in its first 6 months, things seem to have gone a little haywire. So - what's up Akismet? Are the spikes huge coment spam attacks?

Update: some more data on comments, this time from Typepad.

Adversarial Information Retrieval on the Web

This workshop (which I am on the programme committee of) looks at a number of isses to do with the darker side of the web:

- search engine spam and optimization,
- crawling the web without detection,
- link-bombing (a.k.a. Google-bombing),
- comment spam, referrer spam,
- blog spam (splogs),
- malicious tagging,
- reverse engineering of ranking algorithms,
- advertisement blocking, and
- web content filtering.

The previous one was held in conjunction with WWW (Chiba, Japan, 2005) and was a big success. If you're working on blog spam, this may be of interest to you.

Akismet Update

Here is a new graph of data from Akismet showing total spam caught per day, and total ham. It looks like the ham is growing steadily, but that spam has gotten more lumpy over time (compare with previous results).

State of the Comment Spamosphere

In Sifry's State of the Blogosphere post last year, he described the blogosphere as doubling every 5 months. The graph below shows the number of comment spam blocked from people using the Akismet WorkPress plugin. The y-axis represents the number of spam comments for a given day on a logarithmic scale. The major vertical lines are at 7 day intervals.

A rough measurement of this graph suggests that the number of spam comments per day captured by this system is doubling approximately every 3 weeks.

Now, this isn't anything like comparing like with like, but: My hypothesis is that the number of spam comments is growing faster than the number of posts.

Here are the assumptions that I am making:

• The number of daily posts is proportional to the number of active blogs.
• The number of active blogs is growing no faster than Sifry's cummulative blog growth.
• The pick up of the Akismet service is not growing faster than the number of active blogs.

A good reason to move to WordPress?

Now, ask me again about including comment search in blog search engines...

 

Umbria White Paper on Spam Blogs

Umbria's new white paper on spam blogs provides some solid insights into the problem of spam in both the search and marketing intelligence spaces. However, it continues the practice, set out in their recent press release on the topic, of using flawed logic and misleading information to attempt to position itself ahead of the competition.

Umbria analyzes the percentage of spam in the leading blog search engines. There is an improvement here compared to the press release in that it identifies the search terms used and the counts of spam found. The are very open in admitting the anecdotal nature of this experiment. My real concern is that they then use these tests against consumer facing search engines to imply that their enterprise marketing intelligence solution is somehow leading the pack.

They claim that the real time nature of the search engine space prevents adequate spam filtering. Although there may be evidence that the state of existing search engines doesn't neccessarily refute that, fundamentally I don't believe that to be the case. There is no reason why blog search engines can't approach the problem with some form of sandboxing for new blogs whose status is not yet determined. In addition there are approaches to true real time spam filtering that wouldn't even require sandboxing.

The most flawed part of their argument is that they imply that the quality of results in Intelliseek's consumer facing product is identical to that in our enterprise marketing intelligence solution. Given their argument that they would have challanges filtering spam in a real time consumer solution, why would they use the same mechanism in a completely different application? I wouldn't, for example, use the fact that their home page presented the spam blog white paper in a buggy manner to conclude that their web hosted enterprise solution was poorly implemented, or that the poor logic employed in their white paper indicates that the results in their customer reports were not to be trusted! Would I?

Perhaps their story is that their product is more useful than, say, trend mining on BlogPulse, or search on Technorati (which are both free).

Where Does Spam Come From?

Cartography points to Postini's visualization of where spam comes from.

Using Splogs to Market with Statistics

Marketing Vox posts about Umbria's commentary on the state of the splogosphere:

Though some 80,000 blogs are being created every day, 10-20 percent of them may be spam, according to Umbria Communications, which monitors consumer-generated media, reports AdWeek. Umbria's research has found that 2.7 million blogs out of 20.3 million are spam blogs, or splogs, many of which are created solely as a shady marketing tactic, using stolen web content, often via RSS feeds, to profit from contextual ad programs.

Umbria examined blog search results in October from Technorati, IceRocket and BlogPulse, finding that on average 44 of the top 100 results were splogs.

There are some subtleties in here that are being used to confound the reader. Umbria is making a claim that 10-20 % of the global data is spam. They are then implying that 44 % of the results in the listed search engines are spam, suggesting that the search engines are effectively amplifying the problem.

This is a pretty ugly way to go about negative marketing. There is an important piece of missing information: they don't disclose any details of the search terms used. Did they search for 'real estate'? or 'yoga for toddlers'? Were the searches representatitve of search user behaviour? - I'm guessing not as Umbria has no consumer facing search application.

Umbria sells enterprise analysis applications over blog and other data, as do the most of the blog search engines cited. The processes involved in delivering quality results to such clients is quite different from those relating to supporting a consumer facing application like BlogPulse. In blog search engines, the primary dimension of ranking results is time, not relevance. Standard web search, where relevance is the primary ranking dimension, allows spam to simply fall to the bottom of the relevance ranking - never to be seen by the reader. Conflating time and relevance is a challanging problem. Determining the relevance of a post globally in a large-scale, data driven analytical engine is quite different and so the exposure to spam for customers of such systems is entirely different to exposure in time ranked blog search.

Comment Spam as Worms

Akismet is a web service which blesses comments, trackbacks and pingbacks with a non-spam (good) or spam (no good) category. By using the WordPress plugin, you filter how other people use and abuse your blog.

I've been collecting stats from the site for a little over the week. Plotting the daily accumulation of identified spam shows a steady increase in the number of spam comments identified as well as some daily and weekly variations. I've not yet drilled down on the details - the key thing to do here is to align this view the an account of the number of posts found per day and see if the ratio of spam comments to posts is increasing, or if the increase in spam comments is simply an indication of an increase in posts.

The graph plots a point every 15 mins throught a 24 hour period, resetting every day. They look like worms.