Link:
Screaming Frog SEO Spider is a new version of a powerful analysis tool that can audit websites from the point of view of SEO optimization and point out various technical errors. The program not only displays the main resource indicators, such as: citation index, number of external and internal links, but also pays attention to the interlinking of pages, determines the presence of "fast bots" from different search engines, and checks the correctness of robots.txt and .htaccess. In general, Screaming Frog SEO Spider frees the webmaster from many routine tasks.