How @trendingtopics works

Posted on 5 Jun 2009

I made the @trendingtopics Twitter bot to keep me and others up to date on the (often) interesting topics that are being discussed most across the Twitterverse. This is a slightly technical explanation of how it all works.

The bot is updated from a PHP script on this sevrer which is run on a cron job every 20 minutes. Each time it runs this is what happens:

  1. It starts by fetching a JSON feed of the current "trending topics" as determined by Twitter. How these topics are calculated isn't known to me but they're the same 10 topics that are shown on the Twitter search page and in the right column of the Twitter home page when you're logged in.
  2. It throws away all but the top three terms that are not hashtags.
  3. It checks the local database to see if it has already tweeted the topic in the past 8 days. If it has then it ignores this term. If it hasn't then it adds it to the database and continues.
  4. It then starts contructing the text of the tweet starting with the term itself and a link to the Twitter search results for that term.
  5. It fires off a request to the Google Web Search API using cURL to fake the referer and checks the results:
    • If one of the top 5 results is a Wikipedia entry then link to it.
    • If one of the top 5 results is a Twitter profile then add a link to that user.
  1. Add a link to the Google search results page for the term.
  2. Finally send the tweet to Twitter, again using cURL.

I hope that's been vaguely intersting.

For the future I'd like to add a check for current news items using Google news and also add a filter to remove trends which are parts of a popular blog title such as stuff from TechCrunch and Smashing which often end up on the trending topics.

As an aside I find it quite amusing to see all the people sending topic 'suggestions' to @trendingtopics. It doesn't work like that people, sorry!