Web crawling in a few worlds

A Web crawler is an Internet bot which helps in Web indexing. Web indexing is a process of the detection of the links on the page. The bots crawl one page at a time through a website until all pages have been indexed.

Site indexation

Normally bot starts from the main page and works in a loop until it’s capable to find new links. Web crawlers help in collecting information about a website and the links related to them, and also help in validating the HTML code and hyperlinks.

Web crawler in the web search engine. The most famous are Google, Yahoo, Bing, etc.


Imagine we would like to index the ebay.com site.
The crawler will send the request and analyze the code.

Source code example

In the example below, we can see 4 new links :

So nextly the bot will crawl every page from this list and try to find the other links.

Thank you for your attention

Thank you for your attention. If you have any question or advice please feel free to contact me. I‘ll be glad to help you.

LinkedIn, Twitter, Google+, lytvynov.anton@gmail.com, https://lytvynov-anton.com

Senior web developer, architect, cryptocurrencies trader, https://lytvynov-anton.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store