Custom Crawling is usually a difficult process but having the right infrastructure and experience can make the whole process much easier.

I didn't know this is such a prevalent thing. Good to know. Good post.