Anytime filler is more than 3/8" thick you're asking for trouble. Based on your description I suspect the previous owner took the easy way out and just slapped on filler instead of replacing the metal, which would have been the correct way to repair those panels.

As others have said, flip a coin. However if it's really thick... it will fail sooner rather than later.