Accessing robots.txt

Google uses software known as web crawlers that look at webpages and follow links on those pages to bring data about them back to Google’s servers. A robots.txt file tells these crawlers which pages or files the crawler can or can't request from your site.

You can learn more about the robots.txt in the Google Search General documentation.

See how you can access your robots.txt file for the site.

Accessing robots.txt

1. We generate the robots.txt file automatically. To access it, simply add /robots.txt to your domain name. For example:

https://SiteCliq.com/robots.txt

2. If the site is open for indexing in the search engines, the robots.txt file will look this way — with a link to the sitemap:

3. If the site is blocked from being indexed in the search engines, the file will look this way:

4. You can hide or open the site to the search engines anytime. Go to the Settings of your website and proceed to the General info tab, then go to the Visibility in the search engines section:

Note: if you are looking for a way to hide a site or a page from the search engines, check out the articles: Hiding website from search engines and Hiding a page from search engines.

• Related Articles

• Accessing the Sitemap

  A sitemap is a list of pages present on your website which you can submit to the search engines. This list serves as a guide for the search engines for better understanding the structure of your website, and ultimately — for easier SEO. Having a ...

• Site Indexing

  The search engines look for the information to a query in their index of sites, not the actual sites. Thus, getting into the index is super important for your SEO optimization. Google uses software known as web crawlers that look at web-pages and ...

• Adding Your Site to the Google Search Console

  Add your site to Google Search Console (GSC) to be able to track its performance in search, request site indexation and manage different site URLs. Important: the feature of site indexing works only for Pro sites. Adding your site to the Google ...

• "Nofollow" for links

  If you add links to some external resources, you may want to disallow search engines to follow them. In this case, you can set up "nofollow" tags for links. "Nofollow" is a signal to search engines that you don’t want to vouch for the content of the ...

• SEO Basics

  What is SEO? SEO is a complex of actions you undertake to promote the site in the search engines. The result of the SEO mainly depends on the time and efforts invested, and it's never actually over. However, with the help of the tools available at ...