How to find out most searched keywords in United States

In a recent personal task, I needed to extract most searched keywords in US on a daily or monthly basis. While doing some research, I found out that Google actually publishes something called Google Hot Trends on daily basis (on this site). What I wanted was exactly this but I wanted it in the text file so that an automation I had done in browser can read this file and take action.

I thought the best way to extract these keywords would be to write a small C# utility that extracts keywords for a given day.  Here is the source code I wrote to do just that.

   1: StringBuilder sb = new StringBuilder();

   2:  

   3: for (int counter = 0; counter < 15; counter++)

   4: {

   5:     string strURL = "http://www.google.com/trends/hottrends?sa=X&date=";

   6:     string strDateString = string.Format("{0:yyyy-M-d}", DateTime.Now.AddDays(counter * -1));

   7:     Console.WriteLine("Finding most searched keywords for : {0}", strDateString);

   8:     WebRequest request = WebRequest.Create(strURL + strDateString);

   9:     HttpWebResponse response = (HttpWebResponse)request.GetResponse();

  10:     Stream dataStream = response.GetResponseStream();

  11:     StreamReader reader = new StreamReader(dataStream);

  12:     string responseFromServer = reader.ReadToEnd();

  13:     int nStartLocation = responseFromServer.IndexOf("Site Feed</a>");

  14:     int nEndLocation = responseFromServer.IndexOf("<script type=\"text/javascript\">", nStartLocation);

  15:     if (nEndLocation > nStartLocation)

  16:     {

  17:         string strStringtobeProcessed = responseFromServer.Substring(nStartLocation, nEndLocation - nStartLocation);

  18:         string[] delim = { "</a></td>" };

  19:         string[] strArray = strStringtobeProcessed.Split(delim, StringSplitOptions.RemoveEmptyEntries);

  20:         foreach (string strProcess in strArray)

  21:         {

  22:             string strTemp = strProcess.Substring(strProcess.LastIndexOf('>') + 1, strProcess.Length - (strProcess.LastIndexOf('>') + 1));

  23:             Console.WriteLine(strTemp);

  24:             if(strTemp != "\n") sb.AppendLine(strTemp);

  25:         }

  26:     }

  27: }

  28:  

  29: using (StreamWriter outfile = new StreamWriter(@".\SearchKeywords.csv"))

  30: {

  31:     outfile.Write(sb.ToString());

  32: }

If you put above code in any console based application (and add couple of using references on top like System.Net and System.IO), it will create a text file in the current directory called “SearchKeywords.csv” which will look something similar following. Basically it goes and fetches last 15 days most searched keywords and writes them into the file.

It also dumps on the command prompt, what it gets from HotTrends website.

Sometimes, its interesting to see what people are looking for (I love to go and find out why many people are searching for that, there’s a one or the other reason behind that!)