“Crawlability,” Web Design, and SEO

So I got a gentle tweak from Zack Katkin at Unique ID Web Design because I haven’t blogged in a while. I’ve been busy working on projects for clients, but I know that’s no excuse. I’m breaking the Golden Rule of Blogging, which I drill into my clients when they want to start a blog, to wit: You must blog regularly! Thanks, Zack, for the nudge. 😉

Okay, enough of that. Today I’m going to talk about crawlability and web design. I got to browsing the Unique ID blog and read Zack’s post “Straight From Google, The Four Biggest Search Rank Factors,” in which “crawlability” is listed as the very top, highest priority, most important search engine ranking ractor for a web site. This week I’ve also been following a discussion at the High Rankings forums about whether web designers have any SEO responsibility when designing a web site.

The discussion at High Rankings opened with the story of a businessman who hired someone to design a web site for his business. The site was built in Flash, and, as might be expected, the businessman’s web site didn’t do so very well in the search engines. When he sought professional SEO help, he was flabbergasted to learn that an all-Flash site is likely to rank poorly, if at all, in the search engines.

He asked the SEO pro, “Why did the designer use Flash when he knew I wanted search engine visibility?”

A better question would be, why do designers design “search-engine hostile” web sites when they know clients want search engine visibility?

As things stand in the world of web design, anyone with some elementary graphic design skills can get themselves a copy of Dreamweaver or FrontPage and hang out their “Web Designer” shingle, offering their services for a fee to all comers.

Some of these designers do indeed have a lot of artistic talent with respect to creating pretty, aesthetically pleasing, visually attractive web sites.

What these designers lack is a fundamental understanding of the underlying code and structure of web pages, and a fundamental understanding of how search engines crawl and index web pages, and a fundamental understanding of how a web site needs to be structured in order to have a chance of getting search engine traffic.

So these web designers make a “pretty design” in Photoshop or Fireworks or Flash, and use the built-in export features from those programs to auto-generate the code or the Flash file. The client ends up with a very pretty site that hasn’t a snowball’s chance in hell of doing well in organic searches.

The web is still relatively new as a commercial medium, and there is still some level of technical knowledge required in order to build a crawlable web site. Daily we see self-labeled “professional” web designers creating all-Flash sites, or using fancy javascript-based rollover images for global navigation, or relying on other artsy-fartsy features that doom a site to search engine purgatory — a site that is uncrawlable by search engine spiders, and generally invisible in the search engines.

The client doesn’t understand why his beautiful site gets little or no search engine traffic. The client eventually discovers, if he’s lucky or persistent, that he now has to pay for his site all over again, this time to have someone else tear apart his beautiful artsy-fartsy site and re-build it using underlying code and techniques that the search engines can crawl.

Does it have to be this way?

Should it be this way?

I say no, it shouldn’t. Some people might argue that a web designer’s responsibility is to design pretty stuff, not to perform search engine optimization. That’s true up to a point — I wouldn’t posit that it’s the web designer’s responsibility to do link building or write linkbait articles or do keyword research, unless those activities are explicitly included in the agreement.

But I do argue that anyone who holds himself out as a “professional web designer” should have a broad and fundamental understanding of the technology of the medium and the factors that are required for success in that medium. I do argue that the “professional web designer” is holding himself out as an expert, and the client is relying on the expert’s knowledge and experience.

The client shouldn’t have to have a specialist’s knowledge of the medium — that’s why the client hires a professional. When I hire a contractor to build a house, I shouldn’t have to become an expert on building houses, and I shouldn’t have to give the contractor explicit detailed instructions about how to run the wiring so it doesn’t burn the house down. I expect the contractor — ”the professional” — to have the knowledge and expertise to do that himself, even if the contract doesn’t explicitly state that the contractor will run the wiring so that it doesn’t burn the house down.

In an ideal world, building contractors would always run the wiring so that it doesn’t burn the house down, and in that same ideal world, web designers would always build crawlable web sites.

The only exception I would make to this general rule is when a client specifically requests features that will cause crawlability problems, and, after being educated by the web designer about the consequences of his request, the client insists that his aesthetic vision is more important than search engine visibility. The client is paying for the site, after all. But even then there are usually steps the web designer can take to mitigate and overcome the problems — including text links in the footer to complement the pretty Flash buttons at the top of the page, for example.

Meanwhile, we live in an imperfect “buyer beware” world where the web designers who understand the medium are competing against the web designers who don’t. Clients have to educate themselves sufficiently, and ask lots of questions of potential designers, in order to be sure they end up with a crawlable web site.