Scraping Data at Scale
Scrape url of the products
produce to kafka
save to db for persistant
stage - 1
stage -2
Use Distributed scrapy library feature to deployed in multi cluster node
consume from kafka