Follow redirects on scrapy file download
Basically, what's happened is that my spider is unable to download the files because the file_urls provided are actually redirected to the final download link. However, because of the following code, the redirect download middleware is e. Learn how to develop a Python web crawler to crawl websites and extract useful data. You will learn Scrapy basics and how to build a working spider.