Hello, friends! Today, I want to share with you a solution to a problem I encountered while performing technical analysis of websites using the Screaming Frog SEO Spider tool.
What is Screaming Frog SEO Spider?
Screaming Frog SEO Spider is a tool that helps web developers and SEO specialists check websites for errors, track website structure, analyze content, and much more. It is widely used for SEO audits and website optimization for search engines.
The Problem with the “ProjectInstanceData” Folder
When scanning very large websites (more than 500,000 URLs), it is recommended to use SSD disk storage instead of RAM (Database Storage). This prevents memory overload and ensures more stable program operation.
After several months of actively using SEO Spider in this mode, I noticed that the “ProjectInstanceData” folder on my computer started taking up too much space on the hard drive. This folder stores the data that the program collects during website scans, acting as a cache.
In my case, the “ProjectInstanceData” folder grew to 31 GB! This massive amount of data was taking up valuable disk space and could slow down my computer’s performance.
The Solution
When I realized that the “ProjectInstanceData” folder had become too large, I decided to address this issue. Searching on Google did not yield any results, so I had to find a solution on my own.
Here’s how I solved the problem of the large “ProjectInstanceData” folder:
1. In the SEO Spider settings, go to the “File” – “Crawls” section.
2. In the window that opens, select all the crawl data objects that you no longer need and click “Delete”.
After this, check the size of the “ProjectInstanceData” folder. The problem should be resolved.
Important warning! Do not manually delete files from the “ProjectInstanceData” cache folder, usually located at “C:\Users\USERNAME\.ScreamingFrogSEOSpider\ProjectInstanceData\”. Modifying these files outside of SEO Spider can lead to crashes or unexpected program behavior.
Let’s recap
In this way, I solved the problem of the large “ProjectInstanceData” folder in Screaming Frog SEO Spider. Regular cache clearing is a simple way to free up disk space and improve program performance. I recommend performing this procedure from time to time to avoid accumulating too much data.
If you have any questions or need further assistance, don’t hesitate to reach out to me. Wishing you success in web development!
Google Launches March Core Algorithm Update + Spam Update