How to configure Screaming Frog to get valuable data, Beginner guide

Every online marketer needs to many aspects of their own website as well of the competitor’s website to create better marketing strategy. No matter you are a newbie or having an experience of many years, you should know how to configure screaming frog, a well-known website crawler. this website crawler is a great tool to help you in search engine optimiztion of your website.

There will be many tools available in the market which serve different purpose to boost your marketing skills and get you maximum information from the web related to your niche. Similarly, the valuable data that you can get from screaming frog is unmatchable. As an online marketer it always matters how good one is using a marketing tool. Me myself have spent hours in learning many useful marketing tools to serve different purposes of online marketing. I never know the power of the valuable data fetched from screaming Frog. For the purpose I learnt how to configure Screaming Frog, the website crawler, like professionals to earn maximum benefit from the tool. The sole purpose of the post, how to configure Screaming Frog, is educational.

How to configure Screaming Frog on your PC

Once you have downloaded Screaming Frog you need to configure Screaming Frog on your PC. So, the first thing to get notice is that, Screaming Frog is a desktop application. Being a desktop application, it will be consuming your local machine’s resources. No matter you are going to make an audit of a tiny website or a website with hundreds of URLs, you should allocate ample resources to the application. Having an allocation of handful memory of your local machine this website crawler will be able to perform its tasks without hanging other applications.

To allocate the local memory follow the below;

Configuration > System > Memory

It is sure that you will be using either a 32-bit system or a 64-bit machine. the default setting for a 32-bit machines is 1GB of RAM and 2GB of RAM is for 64-bit machines. There are few thoughts who says that you should allocate half of the available memory to Screaming Frog. If you allocate more memory than recommended, then you may experience a crash of operating system.

Once you have allocated either Screaming Frog recommended memory or half of the total RAM you need to restart your computer. By restarting your computer Screaming Frog will apply the updated memory information in its system.

Screaming Frog configurations for better crawl

After allocation of local memory to Screaming Frog for better working, the next phase is to configure Screaming Frog for better crawl. Let’s start with basic configuration of the website crawler.

Configuration > Spider > Basic

You will be reached to the basic settings Screaming Frog has for every crawl. It is always good to set your own configurations according to the crawl you are about to execute. You can make adjustments in these settings accordingly.

Let’s have a look at the basic settings to run a technical audit of a website, either your company owned or of competitors;

• Follow Internal “nofollow”: Internal linking is equally important as of outbound links. This setting will help us find if all the pages we do/don’t want discovered or indexed.
• Crawl Canonicals: this setting will be crawling all the canonical link elements found on each webpage. This will help find us if we have well indicated that what pages to rank.
• Crawl Next/Prev: This setting will help crawl rel=”next” and rel=”prev” elements if there in the website. This will help find clear communication between many pages.
• Extract hreflang: Displays hreflang language, region codes, and the URL to check we are communicating the different variations of our site.
• Crawl Linked XML Sitemaps: This setting will allow to discover all the URLs present in the XML sitemap of the website.
• Auto Discover XML Sitemaps via robots.txt: This setting will allows to find sitemaps discoverable through robots.txt.

Screaming Frog configuration for Java Script

It is rare that a site is not running a Java Script in any form. For an example, Java Scripts can be found even in plugins which we have installed for many reasons for a WordPress website. Or if you have coded Java Script on few pages of your website you need to execute a separate crawl with different configuration for those particular pages. You can skip these settings for whole website.

First of all navigate to the “rendering” tab to make sure the crawler can find those instances.

Configuration > Spider > Basic > Rendering

Custom filter configuration

There are many things which are non-conventional but contain many valuable information on a website. For an example, iframes is an element which is least related to crawl configuration. For the purpose we define some custom filters for such elements.

Configuration > Custom > Search

These filters are extremely useful and can be Included and Excluded according to the requirement

Once all the configurations and filters are being set you should save these setting for the Screaming Frog so that next time you log in to the application you find the same settings.

File > Configuration > Save As ….

You should name every different configuration according to the project you are doing.

Load saved Configuration

Once you have saved all the settings in your Screaming Frog setup, you can easily load these settings any time you want. Simply follow the below to load the saved configuration.

Crawl first website with Screaming Frog

After saving the settings loaded to the Screaming Frog from your local computer, you are ready to crawl your first website. Before that the last step is to select the Mode of Screaming Frog. For the purpose you simply choose Spider Mode as it will start crawling from the provided URL.

Mode > Spider > Enter URL > Click Start

If it is required to crawl a list of URLs of a website, you need to select the List options under Mode tab. Once selected the List option you need to provide the list of URLs from a file or a simple copy and paste.

Once you select List under Mode tab, Screaming Frog will ask either to get the URL(s) from a File, You will enter manually, Past, Download Sitemap and Download Sitemap indexed.

The good thing about screaming Frog website crawler application is, you can stop or resume a crawl any time during the crawl. The resume job will be started from where it was stopped for next time run.

This was really a tiny effort to educate online marketers about how to configure Screaming Frog. Hope you would love to have hands-on on each step yourself to improve your efforts in SEO. If during the process you have any question or have any difficulty you can feel free to ask in comments. Have a great good luck in website audit.