OK have looked at a different method of scanning and here is my results
Memory Used: 834720
Time Taken: 0.41s
Memory Used: 669136
Time Taken: 0.54
So the new method uses less memory but took longer to run...
I did a scan of 106070 files and this is what I got:
Old Method: 31237920
New Method: 20201384
So when scanning larger amounts of data you really start to get advantages in memory usage with the new scan method.
Only problem is where you are winning in one you are loosing in another (time).
Also the tests I preformed were without file/dir exclusion logic. I will investigate further at a later date