The answer to this question is somewhat complex and could just as well be a separate article. However, to simplify this topic, let’s just say that it is all about computing power. With HTML (PHP, CSS, etc.) based websites, crawlers can “see” the website’s content by analyzing the code.
Basically, what you see above is code “processed” by the browser.
To see how the source code looked like before rendering, you need to use the “View Page Source” option.
After doing so, you can quickly notice that all the content you saw on the page isn’t actually present within the code.
After setting this up, we can start crawling data and see each page rendered.
Word of warning
Let me show you an example – as you saw above, Screaming Frog properly crawled and rendered this URL: https://www.hulu.com/series/casual-22d27085-0f5e-42aa-949f-1b81ba5726d8. However, this URL isn’t properly indexed by Google.
Here is proof. Google cache:
Let’s copy and paste content from the Casual TV show landing page:
Unfortunately, content from this page isn’t indexed in Google.