Crawl Errors: The Next Generation

Webmaster level: All

Crawl errors is one of the most popular features in Webmaster Tools, and today we’re rolling out some very significant enhancements that will make it even more useful.

We now detect and report many new types of errors. To help make sense of the new data, we’ve split the errors into two parts: site errors and URL errors.

Site Errors

Site errors are errors that aren’t specific to a particular URL—they affect your entire site. These include DNS resolution failures, connectivity issues with your web server, and problems fetching your robots.txt file. We used to report these errors by URL, but that didn’t make a lot of sense because they aren’t specific to individual URLs—in fact, they prevent Googlebot from even requesting a URL! Instead, we now keep track of the failure rates for each type of site-wide error. We’ll also try to send you alerts when these errors become frequent enough that they warrant attention.

View site error rate and counts over time

Furthermore, if you don’t have (and haven’t recently had) any problems in these areas, as is the case for many sites, we won’t bother you with this section. Instead, we’ll just show you some friendly check marks to let you know everything is hunky-dory.

A site with no recent site-level errors

URL errors

URL errors are errors that are specific to a particular page. This means that when Googlebot tried to crawl the URL, it was able to resolve your DNS, connect to your server, fetch and read your robots.txt file, and then request this URL, but something went wrong after that. We break the URL errors down into various categories based on what caused the error. If your site serves up Google News or mobile (CHTML/XHTML) data, we’ll show separate categories for those errors.

URL errors by type with full current and historical counts

Less is more

We used to show you at most 100,000 errors of each type. Trying to consume all this information was like drinking from a firehose, and you had no way of knowing which of those errors were important (your homepage is down) or less important (someone’s personal site made a typo in a link to your site). There was no realistic way to view all 100,000 errors—no way to sort, search, or mark your progress. In the new version of this feature, we’ve focused on trying to give you only the most important errors up front. For each category, we’ll give you what we think are the 1000 most important and actionable errors.  You can sort and filter these top 1000 errors, let us know when you think you’ve fixed them, and view details about them.

Instantly filter and sort errors on any column

Some sites have more than 1000 errors of a given type, so you’ll still be able to see the total number of errors you have of each type, as well as a graph showing historical data going back 90 days. For those who worry that 1000 error details plus a total aggregate count will not be enough, we’re considering adding programmatic access (an API) to allow you to download every last error you have, so please give us feedback if you need more.

We've also removed the list of pages blocked by robots.txt, because while these can sometimes be useful for diagnosing a problem with your robots.txt file, they are frequently pages you intentionally blocked. We really wanted to focus on errors, so look for information about roboted URLs to show up soon in the "Crawler access" feature under "Site configuration".

Dive into the details

Clicking on an individual error URL from the main list brings up a detail pane with additional information, including when we last tried to crawl the URL, when we first noticed a problem, and a brief explanation of the error.

Details for each URL error

From the details pane you can click on the link for the URL that caused the error to see for yourself what happens when you try to visit it. You can also mark the error as “fixed” (more on that later!), view help content for the error type, list Sitemaps that contain the URL, see other pages that link to this URL, and even have Googlebot fetch the URL right now, either for more information or to double-check that your fix worked.

View pages which link to this URL

Take action!

One thing we’re really excited about in this new version of the Crawl errors feature is that you can really focus on fixing what’s most important first. We’ve ranked the errors so that those at the top of the priority list will be ones where there’s something you can do, whether that’s fixing broken links on your own site, fixing bugs in your server software, updating your Sitemaps to prune dead URLs, or adding a 301 redirect to get users to the “real” page. We determine this based on a multitude of factors, including whether or not you included the URL in a Sitemap, how many places it’s linked from (and if any of those are also on your site), and whether the URL has gotten any traffic recently from search.

Once you think you’ve fixed the issue (you can test your fix by fetching the URL as Googlebot), you can let us know by marking the error as “fixed” if you are a user with full access permissions. This will remove the error from your list.  In the future, the errors you’ve marked as fixed won’t be included in the top errors list, unless we’ve encountered the same error when trying to re-crawl a URL.

Select errors and mark them as fixed

We’ve put a lot of work into the new Crawl errors feature, so we hope that it will be very useful to you. Let us know what you think and if you have any suggestions, please visit our forum!

Written by Kurt Dresner, Webmaster Tools team

Video about pagination with rel=“next” and rel=“prev”

Webmaster Level: Beginner to Intermediate

If you’re curious about the rel=”next” and rel=prev” for paginated content announcement we made several months ago, we filmed a video covering more of the basics of pagination to help answer your questions. Paginated content includes things like an article that spans several URLs/pages, or an e-commerce product category that spans multiple pages. With rel=”next” and rel=”prev” markup, you can provide a strong hint to Google that you would like us to treat these pages as a logical sequence, thus consolidating their linking properties and usually sending searchers to the first page. Feel free to check out our presentation for more information:

This video on pagination covers the basics of rel=”next” and rel=”prev” and how it could be useful for your site.

Slides from the pagination video

Additional resources about pagination include:

• Webmaster Central Blog post announcing support of rel=”next” and rel=”prev”
• Webmaster Help Center article with more implementations of rel=”next” and rel=”prev”
• Webmaster Forum thread with our answers to the community’s in-depth questions, such as:
  
  Does rel=next/prev also work as a signal for only one page of the series (page 1 in most cases?) to be included in the search index? Or would noindex tags need to be present on page 2 and on?
  
  

  When you implement rel="next" and rel="prev" on component pages of a series, we'll then consolidate the indexing properties from the component pages and attempt to direct users to the most relevant page/URL. This is typically the first page. There's no need to mark page 2 to n of the series with noindex unless you're sure that you don't want those pages to appear in search results.

  
  Should I use the rel next/prev into [sic] the section of a blog even if the two contents are not strictly correlated (but they are just time-sequential)?
  
  

  In regard to using rel=”next” and rel=”prev” for entries in your blog that “are not strictly correlated (but they are just time-sequential),” pagination markup likely isn’t the best use of your time -- time-sequential pages aren’t nearly as helpful to our indexing process as semantically related content, such as pagination on component pages in an article or category. It’s fine if you include the markup on your time-sequential pages, but please note that it’s not the most helpful use case.

  
  We operate a real estate rental website. Our files display results based on numerous parameters that affect the order and the specific results that display. Examples of such parameters are “page number”, “records per page”, “sorting” and “area selection”...
  
  

  It sounds like your real estate rental site encounters many of the same issues that e-commerce sites face... Here are some ideas on your situation:
  
  1. It’s great that you are using the Webmaster Tools URL parameters feature to more efficiently crawl your site.
  
  2. It’s possible that your site can form a rel=”next” and rel=”prev” sequence with no parameters (or with default parameter values). It’s also possible to form parallel pagination sequences when users select certain parameters, such as a sequence of pages where there are 15 records and a separate sequence when a user selects 30 records. Paginating component pages, even with parameters, helps us more accurately index your content.
  
  3. While it’s fine to set rel=”canonical” from a component URL to a single view-all page, setting the canonical to the first page of a parameter-less sequence is considered improper usage. We make no promises to honor this implementation of rel=”canonical.”

Remember that if you have paginated content, it’s fine to leave it as-is and not add rel=”next” and rel=”prev” markup at all. But if you’re interested in pagination markup as a strong hint for us to better understand your site, we hope these resources help answer your questions!

Upcoming changes in Google’s HTTP Referrer

Webmaster level: all

Protecting users’ privacy is a priority for us and it’s helped drive recent changes. Helping users save time is also very important; it’s explicitly mentioned as a part of our philosophy. Today, we’re happy to announce that Google Web Search will soon be using a new proposal to reduce latency when a user of Google’s SSL-search clicks on a search result with a modern browser such as Chrome.

Starting in April, for browsers with the appropriate support, we will be using the "referrer" meta tag to automatically simplify the referring URL that is sent by the browser when visiting a page linked from an organic search result. This results in a faster time to result and more streamlined experience for the user.

What does this mean for sites that receive clicks from Google search results? You may start to see "origin" referrers—Google’s homepages (see the meta referrer specification for further detail)—as a source of organic SSL search traffic. This change will only affect the subset of SSL search referrers which already didn’t include the query terms. Non-HTTPS referrals will continue to behave as they do today. Again, the primary motivation for this change is to remove an unneeded redirect so that signed-in users reach their destination faster.

Website analytics programs can detect these organic search requests by detecting bare Google host names using SSL (like "https://www.google.co.uk/"). Webmasters will continue see the same data in Webmasters Tools—just as before, you’ll receive an aggregated list of the top search queries that drove traffic to their site.

We will continue to look into further improvements to how search query data is surfaced through Webmaster Tools. If you have questions, feedback or suggestions, please let us know through the Webmaster Tools Help Forum.

What is Responsive Web design?

In the modern era, every website should have web & mobile design versions to reach max audience. From 2011 onwards usage of tablets has increased a lot. If a design isn’t compatible with tablets or netbooks then another version should be designed to reach tablet visitors. 

Instead of having different website designs for web, mobile & tablets, we can have one design which is compatible with all devices and browsers i.e. Responsive Web Design. Responsive Design itself changes automatically based on the width and height of the visitors screen. Irrespective of design, it majorly depends on screen resolution. 

Following are couple of valuable resources for Responsive Web design:

http://www.alistapart.com/articles/responsive-web-design/

http://coding.smashingmagazine.com/2011/01/12/guidelines-for-responsive-web-design/

Most of organizations and professional blogs already started to have responsive designs for their sites. In future every website owner will prefer to have responsive design. It’s time for designers to learn responsive design tips to have a bright career.

Few Responsive Design Examples:

http://designmodo.com/responsive-design-examples/

How to measure SEO Effectiveness?

Every website SEO performance and results can be measured through following three factors.

1.       Monitor Keyword Rankings

2.       Analyze Monthly Visitors

3.       Track Leads & Conversions

1) Monitor Keyword Rankings

Initially every website will be targeted to few keywords which are relevant to respective website niche. SEO Professionals should make sure that “respective landing pages of the website should be indexed & ranked for targeted keywords”. If your website is started ranking for few long tail keywords in the first month itself, it means “your SEO team is doing the right job”. Also month over month keyword rankings positions should be improved. With the help of following kind of report everybody can measure their SEO effectiveness.   

Note: This will not happen in all cases, because few of the niches are highly competitive, so it will take little more time to get results.

2) Analyze Monthly Visitors

Every website visits can be tracked through Google Analytics. It’s free tool offered by Google, also simple to install & activate on website. Usually for a new website minimum two months’ time required to receive traffic from search engines. From 3^rd month onwards every website will start receiving little traffic and it should be gradually increased month over month. Of course, traffic increase in every month is a difficult task, so at least existing minimum traffic shouldn’t be decreased.

If your SEO team is continuously adding new content pages then the traffic should be improved. So keep an eye on website traffic and measure your SEO team efforts. Ask your SEO team to provide monthly improvement stats and reports for your reference.

3) Track Leads & Conversions

Your website should have Phone number, Email ID and Inquiry box if you are offering services. If your site is an eCommerce portal then it’s important to have online payment facility, so that visitors will purchase your products through online. 

Compare month over month stats whether you are receiving any inquiries or sales from your website or not and calculate ROI. Sometimes cannot expect immediate ROI through SEO efforts as SEO will give good results in longer run. However make sure to push and keep little pressure on your SEO team to have max output. 

Google Local Listings with Zagat Rating in Google Plus

Google made some changes to Local Business Listings from May 30th 2012 onwards.

What are the changes?

1. Google Local Business Listings has a new look i.e Local listings are integrated in Google Plus.
2. Each local business listing rating system has been improved using “Zagat Rating” system.

What is Zagat Rating?

Zagat Rating system will have 30 point scale. Mostly this system used for hotels where customers will rate food, décor and service. It seems that Google has applied the same rating system to all of the local business listings in Google Places. However Google is displaying only the total score instead of individual Zagat Score of each service. Refer following screenshot for a better idea.

Following is the example of a restaurant along with Zagat Rating Scrores:

Difference between Old & New Local Listings:

4 Health Tips for Professional Bloggers

gym. Or spandex. Exercise is just movement, and it’s up to you how to move. I think group fitness classes are really fun and motivational.

You might not. How about a peaceful walk through the park? Or riding a bike to pick up a few groceries? Exercise doesn’t need to feel like exercise. Pick something that you enjoy doing and do it as often as you can. Aim to do a movement activity at least 30 minutes a day for at least four days a week. You’ll end up feeling happier, sleeping better and writing with more enthusiasm.

4. Practice breathing and relaxation

Sounds silly, right? When we get stressed, we sometimes actually forget to breathe. Our breaths become short and shallow, just stressing out our bodies even more. Stress then contributes to weight gain or more of a struggle in weight loss.

When you feel yourself tense up at a project or client, just sit up straight in your chair, close your eyes, and breathe. Count to five on your inhale, then count to five on your exhale. Breathe like this for at least 10 breaths, or until you feel more calm.

There you have it. The healthier you are, the better you can conduct your daily blogging chores with confidence and less stress. How about you? Do you have any practical tips you can add to the list?

8 Cool Tips for New Bloggers

New Bloggers in 2011 have got their job cut out for them! The Internet is a crazy place and there are people out there saying crazy things. How do you know who to believe? All the scams, schemes, and push button software that have promised to make you a millionaire.

Luckily most people are now coming to the conclusion that the only true way to make money blogging is through hard work and dedication.

Here are some important tips that I have gathered on my journey.

Tips for New Bloggers

Limit your learning

New Webmasters often get caught up in learning how to build websites and how to make money blogging. The best learning experience that I can recommend to you is to get in there and do it! Do not try learning how to blog in theory when you can simply start a blog or a website today!

The first thing I would do is create a free blog and write about my learning experiences for the day.

Do not buy Internet marketing course after IM course. If you failed at the first course then go back and finish it.

Scrap that attitude

If you want to be successful at blogging you will need to get rid of a few built-in attitudes that might be hindering your performance.

• I will do it tomorrow
• I don’t know how to do it
• I am not smart like you
• I do not have time
• I have a degree, I am better than you
• I am an expert on every subject
• No one is making money online

A bloggers attitude is sharing, caring, networking, and cool. Be that person.

Create a schedule

Working on a computer on the Internet can be very distracting as there are e-mails coming in, comments to answer, and interesting articles posted every second. These distractions can delay your success and productivity time by possibly years. This is why I recommend that you create a timetable or schedule when participating in this business.

I have a timetable for each day. It starts with opening emails and answering comments and moves on to other particular tasks. If I have committed my self to working I need to accomplish my set tasks for the day.

Use a paid WordPress theme

You can build a website on practically zero dollars but buying a paid WordPress theme is something that I do recommend for new bloggers. The reason I say this is because you can get free support with your purchase.

It can be hard choosing the best WordPress theme for your website but if your theme comes with backup support this will save you hours of frustration. Most new bloggers can spend weeks and weeks customizing their blog theme. However if you purchase a WordPress theme you can simply go and ask a question in their forum and they will help you.

I have experience in HTML, PHP, and CSS, but why would I rack my brain when I can get the solution to any problem using my WordPress theme support.

Do not wait for the traffic

Website traffic does not simply appear on your blog. You have to get out there and work very hard to get website traffic. I could narrow the two main traffic sources down:

1. Hard slog promotions to get traffic
2. Search engine optimization to get traffic

The both these ways a great, but in essence, you will need both to succeed. For example when you have a new blog you will need to go build back links to your content via guest posting, article marketing, commenting, forums, and so on.

Some back linking brings traffic, but most back linking does count the search engine optimization. Therefore while you were generating traffic, you are also performing search engine optimization.

Never stop promoting

There will come a time when you have done so much promotion; you will see traffic coming for free from the search engines. People see this as a sign to relax, however you need to keep going and build the momentum. You need to keep the ball rolling when your website is in the growing stages.

Do not neglect SEO

Webmasters often get onto a new promotion method and run with it. This could be guest posting or article submission, which all eventually leads to search engine optimization. However you need to finish the job, by performing onpage search engine optimization at your website.

A simple explanation of Onpage SEO is:

1. Target a keyword for each page or post
2. Make a list of related keywords (LSI keywords)
3. Find some good longtail keywords
4. Create a killer title using your main keyword
5. Use the keyword and LSI keywords in the tags, description
6. Use the main keyword in the first and last line of your article
7. Use H1,H2,H3 tags using the keyword and some LSI mixed in
8. Have the main keyword density around 2% (some say more)
9. Use LSI keywords throughout the article
10. Add longtail phrases scattered in only in appropriate places

If you follow all of these instructions above, I guarantee that you will have a post that is full of information and really covers your subject. This will also make your posts longer an easy to read.

Never pretend

Never pretend that you are successful and making money when you in fact are not! Don’t be a know it all when you actually know nothing at all.

How will you ever become successful if you are already pretending to be successful?

New bloggers have 100% more to learn, 100% more distractions, but still have a 100% chance to succeed. It’s all up to YOU!

Important Tips To Select Domain Name For New Bloggers

It is a great task to select a new domain name for your business niche. The importance of a domain name is realized by entrepreneurs because of its significance in SEO, among other things. To achieve success in business you should utilize all the possible means. And it all starts with an address – the domain name.

Having realized the importance of keywords in SEO listings, you cannot be ignorant of the selection of the most appropriate domain name for your niche. With that said, here are a few tips which will be definitely useful for those who would like to pick up a new domain name.

To select a unique name

The domain name is the starting point where your web journey begins. Domain name is the foundation for the very existence of your online presence.

The domain name selected should be as catchy as possible. It should be closer to your business niche. The domain name should set expectations for visitors about the type of content that will be presented on your site. In this way, people can fittingly recollect your site and refer to others as well.

The time you spend in the selection of an artistic domain name will fetch you many times by returning benefits in the years to come. Hence, you should pick a domain name which qualifies the following characteristics:

• Easy to remember
• Easy to type
• Related to your business niche
• Search engine friendly
• KISS (Keep It Short and Simple)
• Relative to the content on your site

You should avoid names with the following characteristics:

• Names similar to existing popular sites (although I totally understand the urge to do so)
• Domain names with hyphen
• Misspelled domain names

For branding

If you are very serious about branding and want to establish your business on a long-term perspective and would like to cover customers catered throughout the world, .com is a viable option.

Even though top level domains such as .net, .info, .org, etc. can be leveraged to imbibe all kinds of SEO benefits, .com gives a professional appearance. Most of the people try .Com as it was very popular since the advent of the internet. It is useful to get global image.

Growing businesses will also buy domain names with .net, .org, .info so that even if customers try to visit your site by typing other top level domain names, they will be redirected to your .com site.

For instance, if my blog is myname.com, I could also purchase myname.net, myname.org and other TLDs and redirect them to myname.com. This way I won’t miss those visitors who type the TLDs as well.

Keyword popularity

You can check the popularity of a keyword. There are various online tools to check the value of a keyword in various aspects. The free and the most convenient tool is the Google Keyword Tool.

But if you want to go the extra mile, you’d want to use a tool like Market Samurai.

You will be able to find the worth of keyword in SEO aspect, the competition for the keyword, unique keyword, etc. Market Samurai is a great tool that will definitely help you in this aspect.

You can include keyword in your domain name that will have less competition. In this way, you will be able to promote your site very well in the coming days.

Check the availability

After having clarity about the domain that you would like to choose, you should check the availability of the same on sites that offer domain registration facility. You can try sites like hostgator.com or godaddy.com to register your domain names.

If the desired domain name is not available you should be flexible enough to choose the one closer to your expectations. As you type a domain name of your choice, if it is not available, you will be provided with a list of available alternative site names.

Used domain names

Domain names that are already used by others and set for sale are not as attractive as used cars. In case of used cars, you will be able to check the performance of the car. The same will not hold good for purchasing used domain names.

You may need to pay a hefty price for good and popular niche sites if they are available for sale. You can take advantage of back links and Google PR in such cases.

On the other hand, if domains were involved in spamming activities and completely banned by Google and other search engines, you cannot invite furthermore trouble by purchasing such kind of domain names.

Hence, you should be very cautious in the purchase of used domain names. And, it is advisable to go for brand new domain names rather than for a used name unless you have specific justifying reasons of business interest (unless you are a savvy to drill deep and analyze the history of the old domain name; especially, given the recent Google changes).

Name registration plus hosting

There are very many sites which will help you register your domain name as well as help you with web hosting. As you opt for both of these services from the same service provider, it will be easy for you to renew the registration and to get extended help from them.

It is always advisable to select services from a reputed service provider which has presence in the market for a considerable period of time. You should be able to reach them through various means including chat, email and voice channels.

Takeaway

Even though selection of a domain name is not the ultimate factor that decides the success of your business, you should be cautious and can take advantage of the name by following the above mentioned tips. You are invited to share your views.