If you’re blocking Google from crawling your javascript and CSS, you may potentially be hurting your own search rankings. It’s not that using that javascript and CSS will necessarily make you rank better, but if you don’t let Google crawl it, you’re not giving Google the entire picture of what’s on your page.
Matt Cutts posted a new webmaster help video, but this time, instead of responding to a user-submitted question like he usually does, he provides what he refers to as a public service announcement.
“If you block Googlebot from crawling javascript or CSS, please take a few minutes and take that out of the robots.txt and let us craw the javascript,” says Cutts. “Let us crawl the CSS, and get a better idea of what’s going on on the page.”
“A lot of people block it because they think, ‘Oh, this is going to be resources that I don’t want to have the bandwidth or something,” but Googlebot is pretty smart about not crawling stuff too fast, and a lot of people will do things like, they’ll check for Flash, but then they’re including some javascript, and they don’t realize that including that javascript – the javascript is blocked, and so we’re not able to crawl the site as effectively as we would like,” he says.
“In addition, Google is getting better at processing javascript,” he continues. “It’s getting better at things like looking at CSS [to] figure out what’s important on the page, so if you do block Googlebot, I would ask: please take a little time, go ahead and remove those blocks from the robots.txt so you can let Googlebot in, get a better idea of what’s going on with your site, get a better idea of what’s going on with your page, and then that just helps everybody in terms of if we can find the best search results, we can return them higher to users.”
“So thanks if you can take the chance. I know it’s kind of a common idiom for people to just say, ‘Oh, I’m gonna block javascript and CSS, but you don’t need to do that now, so please, in fact, actively let Googlebot crawl things like javascript and CSS, if you can.”