How to use PyCharm to debug Scrapy projects
I am working on Scrapy 0.20 with Python 2.7. I found PyCharm has a good Python debugger. I want to test my Scrapy spiders using it. Anyone knows how to do that please?
What I have tried
Actually I tried to run the spider as a scrip. As a result, I built that scrip. Then, I tried to add my Scrapy project to PyCharm as a model like this:
File->Setting->Project structure->Add content root.
But I don't know what else I have to do
scrapy command is a python script which means you can start it from inside PyCharm.
When you examine the scrapy binary (
which scrapy) you will notice that this is actually a python script:
#!/usr/bin/python from scrapy.cmdline import execute execute()
This means that a command like
scrapy crawl IcecatCrawler can also be executed like this:
python /Library/Python/2.7/site-packages/scrapy/cmdline.py crawl IcecatCrawler
Try to find the scrapy.cmdline package. In my case the location was here:
Create a run/debug configuration inside PyCharm with that script as script. Fill the script parameters with the scrapy command and spider. In this case
Put your breakpoints anywhere in your crawling code and it should work™.