Answer by naqushab for Debugging Scrapy Project in Visual Studio Code

I made it. The simplest way is to make a runner script runner.py

import scrapyfrom scrapy.crawler import CrawlerProcessfrom g4gscraper.spiders.g4gcrawler import G4GSpiderprocess = CrawlerProcess({'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)','FEED_FORMAT': 'json','FEED_URI': 'data.json'})process.crawl(G4GSpider)process.start() # the script will block here until the crawling is finished

Then I added breakpoints in the spider while I launched debugger on this file.Reference: https://doc.scrapy.org/en/latest/topics/practices.html