I was recently working on an SEO engagement that began with an extensive phase of keyword research and analysis.  As we worked to categorize, organize, and generally make sense of the terms that populated the language of our client’s industry, we began to notice clusters of search terms for which no corresponding content could be found on their website.

Did our keyword research efforts somehow run amok, inadvertently pulling in keyphrases unrelated to our client’s business?  That was certainly a possibility, and can happen quite easy if you’re not careful.  Another, more exciting possibility is that we uncovered niches within our client’s market (perhaps even potential products or services) that they weren’t even aware were worth considering.

The beauty of well-executed keyword research (using some of our favorite tools like Google’s Keyword Tool), is that the keywords you come across often paint an interesting picture of the subject you’re investigating.  Looking at the language used by searchers can give you insight into their needs or motivations (consider, for instance, the different needs associated with the search terms ‘buy dog leash’ vs. ‘dog leash training’).   And better still, Google’s Keyword Tool doesn’t just give you keyphrases related to your subject, but the traffic volumes associated with them, as well.  With this information, you begin to understand the degree of interest associated with the terms your keyword research uncovers.

Of course, this is a very simplified demonstration of the potential for using keyword research to aid in market research, but it’s a beginning.  With the right tools and know-how, there’s a strong case to be made for using keyword research to kick-off your quest for new market niches and opportunities for new products or services.

If you haven’t already checked out Google Insights For Search, you really should. Like Google Trends, this awesome tool allows you to figure out how a search query’s popularity has trended over time and compare the trends between these search queries.

But Insights For Search takes this functionality a step further, allowing you to find top searches by location, time range, or any combination thereof.  You can also see the top search terms that include a keyword of your choice. Insights will also make projections of future trends.

While it’s kind of fun to know that “Michigan” is the most popular search term in Michigan (http://www.google.com/insights/search/#geo=US-MI&cmpt=geo), it isn’t immediately apparent how you would use this kind of information to improve your AdWords account.

Here’s an example we ran into recently when we were able to use Insights For Search to inform the way we built out an AdWords account for a client:

The Detroit metro area is home to a large and vibrant Arab community. According to the 2000 census, Arab-Americans comprised 30% of the Detroit suburb of Dearborn. Our client, naturally, wondered if we should be targeting the Arabic language and creating Arabic keyword lists since they wanted to target this region. Google Insights for Search to the rescue!

We ran keyword searches within Google Insights and compared the several common terms based on the top searches in Iraq and Lebanon (since Dearborn’s Arab population is primarily composed of Iraqi and Lebanese immigrants or their descendants). In this Insights For Search chart, you’ll find the terms “photos” (blue), “games” (red), and “ramadan” (orange) in Arabic graphed to show their relative popularity in the US. We also tested some terms like “soccer” and “flights” that we thought might be popular, but they were not popular enough to register results.
The states that seem to have a lot of people doing Arabic language searches are Virginia, California, North Carolina, and Florida. According to the last US census, the only state in that group with a larger Arab community than Michigan is California.

It’s possible there aren’t a significant amount of Arabic language searches in Michigan because the Arab population here is longer-standing, much of which immigrated during the first half of the twentieth century. It’s also possible that many of the Lebanese Arabs might prefer to search in French over Arabic, since Lebanon uses both languages, and American keyboards are better-suited to French than Arabic.

Whatever the reason members of the Arab community in Michigan do not search in Arabic, the important question has already been answered – it seems that it is not worth the rather significant investment of time and money to translate keyword lists to Arabic when targeting the Dearborn area.

At the end of July, Microsoft Research held its 2008 Faculty Summit to survey the state of computing R & D, which this year included a social media summit. A major topic of conversation included the transition of the internet from a network of documents to a network of people.

As participant (host) and Microsoft Scientist Matthew Hurst explains on his blog, “The PageRank era is marked by a very simple link with no explicit meaning and a simple assumption (a positive endorsement).” But this assumption of positive endorsement is becoming unnecessary as more and more direct evidence of people’s opinions and categorizations of content are available online. Research repeatedly reveals that others take notice of human-generated tags and reviews: “consumers report being willing to pay from 20% to 99% more for a 5-star-rated item than a 4-star-rated item (with variance depending on type of item/service)”, is just one example.

Many are excited by how much less processing-intensive the online content tagging process becomes with this trend – clusters of pages and facts seem to grow organically as a result of human tagging. This helps overcome previous problems related to content indexing within info retrieval, such as the gap between the language that the businesses or organizations use to label their content and the terminology preferred by their customers/users.

But there are challenges that arise as well in this transition that are less discussed. Says one scientist, aptly describing the phenomena, “fragmenting media and changing consumer behavior have crippled traditional [media] monitoring methods. Technorati estimates that 75,000 new blogs are created daily, along with 1.2 million new posts each day, many discussing consumer opinions on products and services. Tactics [of the traditional sort] such as clipping services, field agents, and ad hoc research simply can’t keep pace.” Call it what you will: Brand Monitoring, Online Image Tracking, Buzz Monitoring, Online Anthropology, Conversation Mining, Online Consumer Intelligence, Market Influence Analytics … The challenges remain the same. As an example, I think of a project I did here at Pure Visibility last year, which involved analyzing online review content related to a client’s company. After gathering the reviews (in the hundreds), I was faced with the daunting task of mining them for basic information like the overall majority sentiment expressed, and how this correlated with the source. My ultimate method was mostly manual and more than a little tedious.

Hurst’s blog contains a reference to a new book by Pang and Lee that surveys the state of Opinion Mining and Sentiment Analysis, (basically, data-mining and classification using human generated content). In addition to interesting facts on the power of opinions like those above, this book clearly outlines the process that such analysis requires, and the associated challenges. For example, incorporating user opinions into a search engine typically requires the following steps:

1. determining whether the user is looking for subjective information
2. accurately classifying docs into the opinionated and non-opinionated bins
3. identifying overall sentiments expressed and or/specific opinion regarding particular aspects
4. summarizing information, including aggregating votes via different rating scales, highlighting some opinions, representing disagreement/consensus points, id’ing opinion holders, etc

The challenges are numerous. To summarize some of the excellent points made by Pang and Lee, I sketched out the following table, which compares opinion mining to traditional text mining:

To clarify on this last point, the authors note that this fact alone does not make the task more difficult than traditional topic classification, since data-driven approaches can be applied to the latter to improve accuracy over classification using a human-picked keyword list. The problem is that the accuracy of a data-driven method for opinion analysis is only about 80%, which is still not comparable to the performance expected in traditional topic-based classification.

While these challenges may seem intimidating enough to remain on the horizon for years to come, the fact that this book was written by a Yahoo research scientist, and one of the country’s top CS schools suggests that the right people are thinking about these trends. Significant changes in how we use the web may not be far off.

Today’s SEO researcher enjoys the choice between numerous keyword generation tools, each of which offers their own relative strengths and weaknesses.

Take, for example, both Yahoo’s Overture Keyword Selector Tool and the Keyword Discovery Tool. I use these two interchangeably, often because what I expect to be fairly common search terms bring up no data in one or the other database. I’m hesitant to trust either tool completely, mostly because of some strange inconsistencies as well as the skewed nature of the data. Not to mention the irritating way both are prone to crashing, and require frequently reloading the page. Read More

This is an interesting adwords problem that has an impact for keyphrase matching for industries where terms are often hyphenated, something often found in the product titles of many industrial goods.

Here are three ways to spell the company name Allen Bradley:

Allen Bradley
AllenBradley
Allen-Bradley

We are the KeyWord matching algorithm in our copy.

What we’ve found is that if the capitalized keyword match function {KeyWord:foo} is set in our ad copy, the three entries look like this:

Allen Bradley. Okay, that looks about right.

Allenbradley. Well, this is understandable. What algorithm would know how to break this word up?
Allen-bradley. Oops. Lower-case “b” after the hyphen.
We’ve notified Google and it’s a known bug. If it gets fixed we’ll let you know.

UPDATE: We just received this from our Google Account Executive:

Google’s system doesn’t recognize commas, periods, hyphens, or non-letter characters when they appear in keywords, the hyphens in your keyword are stripped out of the search terms and treated as spaces. Thus, a search on either allen bradley or allen-bradley will match up to the same term, regardless of punctuation. So, if two search terms match equally, the one with the better calculated rank (Quality Score * Max CPC) will get the impression, regardless of punctuation.

For example, a search on the keyword ‘ALLEN BRADLEY distributor’ within this account is currently matching your keyword ‘Allen-bradley Distributor’ (with a hyphen), because that keyword has a higher calculated rank than ‘ALLEN BRADLEY distributor’ (without a hyphen). Since the hyphenated keyword is being matched, it is also being inserted into the ad with keyword insertion. And since, keyword insertion recognizes ‘allen-bradley’ as one word, due to the hyphen, the ad displayed contains a lowercase ‘b.’

To resolve this issue, I would suggest that you delete all of your keywords that contain the hyphenated version of the company name. This will prevent the hyphenated company name from being inserted into your ad text (and avoid the cheesy looking ad copy). Also, if a user searches for ‘allen-bradley distributor’ on Google.com, the keyword ‘allen bradley distributor’ will still be matched in the account. Therefore, you won’t be missing out on any traffic by removing the hyphenated versions of the keyword.