I do see possible damage in the increased workload of the fuel system in trying to draw the fuel through clogged filters. I don't think that this means you should replace the filters as soon as you see a speck of dirt/fungus/etc., but when you noticed a fair amount (a good reason to use a clear primary filter) of it.

__________________
85 300CD
83 300TD
78 240D (daughter)