How to Improve Twitter Search

I’m starting to get Twitter as a publication mechanism, and I’ve always understood the value of mining the aggregate of tweets for any number of reasons. However, I don’t totally get Twitter search. In trying to, I’ve been playing around with it a little more. Here’s what I though:

• Why no link search? Most urls on Twitter go through TinyUrl or something similar, and yet these are not dereferenced. 23 % of tweets contain a url and so there is a big space of search targets for url search. I did a search for links to a blog post that I’d shortened and no results came up.
• Language classification doesn’t work. Of course, language classification is harder the shorter the text is, so Twitter is either asking for trouble by having it at all, or has some very cutting edge thoughts on how to do it. Plenty of languages can be bucketed by character encoding analysis (e.g. Korean, Japanese, Thai). However, do a search for something obscure and you’ll hit the language classification problem. E.g. a search for Elie Fife, my home town, produces no results (in the default English category), but a search over all language produces only English language results.
• Why no search counts? When results come back, we don’t know how many there were. For any type of rudimentary tracking, this count is important – at least for relative analysis.
• Add in time series. Time series in the results would be awesome. Some sites (like Twist from Flaptor) are attempting to provide this, but without the full data set, it is not clear how good or accurate these results are.
• Authorial search. Wouldn’t it be fine to search limited by features of the author? Certainly location is a no brainer.

I’m pretty sure that the lack of these features has nothing to do with the team’s ability to deliver them. It is likely a set of pragmatic issues to do with the value of these features in the context of their yet-to-be-decloaked business model.

Comments

[Daniel Tunkelang]

Indeed, if Twitter made the log archives downloadable so that others could index them, then we might see a proliferation of more sophisticated search functionality on top of them. But I suspect that Twitter prefers to keep this data to themselves for eventual monetization.

[Erik]

It looks like Twitter might agree. They are hiring a search engineer: http://jobs.joelonsoftware.com/default.asp?4632

[feedbackmine]

I am very curious on what lanuguage detector twitter is using, its result is very poor.

I have implemented a n-gram based ruby language detector myself, and the result is much better than twitter's:

http://github.com/feedbackmine/language_detector/tree/master

[feedbackmine]

Hi Daniel,
Yes twitter provides data mining feed, which provides the last 600 tweets in the past minute. Lots of people, including me (http://tweetjobsearch.com), is using it to build third party search application using those data.

The current problem is, you are not able to seen all the tweets from that feed, as sometimes there are more than 600 tweets in a given minute. They do have plan to provide a better solution in the future.

[Brennan Novak]

Actually Twitter search does have a way to see just the latest links. You can see a link to it in the bottom right of the nav bar with the link "cool filter: links" below is the URL of the that search term.

http://search.twitter.com/search?q=cool+filter%3Alinks

They also are doing some sort of location google map integration you'll also notice the "happy hour" near:SF" returns desired results.

[Christopher Golda]

We created link search if you're interested:
http://backtweets.com/

[Shawn]

I'd like the ability to limit my search to just the people I'm following. Often, I'm looking for something I read previously (especially if I originally read it on my mobile).

[Suzy Tonini]

In reply to Christopher Golda- in Twitter Search type in from:username and you'll get a list of Tweets from that person. Hope that helps.