ZAR out of memory error

[Donnie]

Running ZAR to recover data from a RAID5 array. I've been receiving out of memory errors today.

I added another 1GB of ram and another 12GB of swap to possibly avoid the problem. My system now has 3GB RAM and 16GB swap.

The 2.7 million files ZAR has found on the volume is approximately the number of files that had been on the array before the disk problem. ZAR was creating an autosave file when a fatal error - out of memory box popped up.

Running ZAR 8.3RC1 on XP SP2, Opteron 185, 3GB RAM, 16GB swap. ZAR process was using 1.83GB RAM when fatal error occurred.




Last 100 or so lines of the log are as follows:

ALERT!: Excessive MFT attribute size near ap 262; hint 91051
ALERT!: Excessive MFT attribute size near ap 262; hint 91051
ALERT!: Excessive MFT attribute size near ap 262; hint 91052
ALERT!: Excessive MFT attribute size near ap 262; hint 91052
ALERT!: Excessive MFT attribute size near ap 462; hint 91053
ALERT!: Excessive MFT attribute size near ap 462; hint 91053
ALERT!: Excessive MFT attribute size near ap 278; hint 91054
ALERT!: Excessive MFT attribute size near ap 278; hint 91054
ALERT!: Excessive MFT attribute size near ap 270; hint 91055
ALERT!: Excessive MFT attribute size near ap 270; hint 91055
ALERT!: Excessive MFT attribute size near ap 254; hint 91056
ALERT!: Excessive MFT attribute size near ap 254; hint 91056
ALERT!: Excessive MFT attribute size near ap 270; hint 91057
ALERT!: Excessive MFT attribute size near ap 270; hint 91057
ALERT!: Excessive MFT attribute size near ap 254; hint 91058
ALERT!: Excessive MFT attribute size near ap 254; hint 91058
ALERT!: Excessive MFT attribute size near ap 270; hint 91059
ALERT!: Excessive MFT attribute size near ap 270; hint 91059
ALERT!: Excessive MFT attribute size near ap 278; hint 91060
ALERT!: Excessive MFT attribute size near ap 278; hint 91060
ALERT!: Excessive MFT attribute size near ap 254; hint 91061
ALERT!: Excessive MFT attribute size near ap 254; hint 91061
ALERT!: Excessive MFT attribute size near ap 254; hint 91062
ALERT!: Excessive MFT attribute size near ap 254; hint 91062
ALERT!: Excessive MFT attribute size near ap 270; hint 91064
ALERT!: Excessive MFT attribute size near ap 270; hint 91064
ALERT!: Excessive MFT attribute size near ap 262; hint 91065
ALERT!: Excessive MFT attribute size near ap 262; hint 91065
ALERT!: Excessive MFT attribute size near ap 270; hint 91066
ALERT!: Excessive MFT attribute size near ap 270; hint 91066
ALERT!: Excessive MFT attribute size near ap 270; hint 91067
ALERT!: Excessive MFT attribute size near ap 270; hint 91067
ALERT!: Excessive MFT attribute size near ap 270; hint 91068
ALERT!: Excessive MFT attribute size near ap 270; hint 91068
ALERT!: Excessive MFT attribute size near ap 278; hint 91069
ALERT!: Excessive MFT attribute size near ap 278; hint 91069
ALERT!: Excessive MFT attribute size near ap 278; hint 91070
ALERT!: Excessive MFT attribute size near ap 278; hint 91070
ALERT!: Excessive MFT attribute size near ap 286; hint 91071
ALERT!: Excessive MFT attribute size near ap 286; hint 91071
ALERT!: zero-sized MFT attribute
ALERT!: zero-sized MFT attribute
ALERT!: Excessive MFT attribute size near ap 270; hint 91457
ALERT!: Excessive MFT attribute size near ap 270; hint 91457
ALERT!: Excessive MFT attribute size near ap 486; hint 91458
ALERT!: Excessive MFT attribute size near ap 486; hint 91458
ALERT!: Excessive MFT attribute size near ap 254; hint 91459
ALERT!: Excessive MFT attribute size near ap 254; hint 91459
ALERT!: Excessive MFT attribute size near ap 270; hint 91460
ALERT!: Excessive MFT attribute size near ap 270; hint 91460
ALERT!: Excessive MFT attribute size near ap 270; hint 91461
ALERT!: Excessive MFT attribute size near ap 270; hint 91461
ALERT!: Excessive MFT attribute size near ap 254; hint 91464
ALERT!: Excessive MFT attribute size near ap 254; hint 91464
ALERT!: Excessive MFT attribute size near ap 270; hint 91465
ALERT!: Excessive MFT attribute size near ap 270; hint 91465
ALERT!: Excessive MFT attribute size near ap 350; hint 91466
ALERT!: an exception has occured in TThreadParseNtfs.ReadIn() status Out of memory
Performance: NTFS read-in 16m 29s
Directory tree linking started - total 2739549 objects on volume
Directory structure loop for Uln
ses
Directory structure loop for Cookies
Performance: NTFS rebuild directory tree 1m 15s
Directory tree refinement and reconstruction started
Directory tree refinement and reconstruction pass 1 started for 2739549 objects
ALERT!: an exception has occured in TThreadParseNtfs.Relink() status Out of memory
Performance: NTFS directory tree cleanup 8s
Simple volume - Parse filesystem complete
Fragment reordering start
No fragments stored, rolling back
Performance: Ident data query (0 of 0) 0s
Performance: Ident data query (0 of 0) 0s
Performance: Ident data query (0 of 0) 0s
Performance: Ident data query (0 of 0) 0s
ALERT!: an exception has occured in TIdentList.slSaveToFile() status Out of memory
ALERT!: an exception has occured in TIdentList.slSaveToFile() status Out of memory
FATAL:
The fatal exeption has occured: Out of memory
Last observed error code was 8.
Unable to continue processing, program will now terminate.
Please file a bug report.
FATAL:
The fatal exeption has occured: Out of memory
Last observed error code was 8.
Unable to continue processing, program will now terminate.
Please file a bug report.
FATAL:
The fatal exeption has occured: Out of memory
Last observed error code was 8.
Unable to continue processing, program will now terminate.
Please file a bug report.
FATAL:
The fatal exeption has occured: Out of memory
Last observed error code was 8.
Unable to continue processing, program will now terminate.
Please file a bug report.
FATAL:
The fatal exeption has occured: Out of memory
Last observed error code was 8.
Unable to continue processing, program will now terminate.
Please file a bug report.

[Donnie]

Also tried adding /3GB switch to XP's boot.ini file.

ZAR's log file still shows only 2047MB installed when 3072MB is in fact physically installed and detected by XP (see screenshot in above post):

Init start - misc
Init done - misc
Init start - types
Init done - types
Init start - math B
Vote : 1; Count=3 ; separation
Vote : 1; Count=3 ; separation
Vote : 10; Count=4 ; separation
Init done - math B
Init start - filters
Init done - filters
Init start - PDF validation
Init done - PDF validation
Init start - OSS validation
Init done - OSS validation
Init start - validation
Init done - validation
Init start - LDM parser
Init done - LDM parser
Init start - MFT regions
Vote : -1; Count=0 ; separation
Init done - MFT regions
Init start - caches
Init done - caches
Init start - lazy writer
Init done - lazy writer
Init start - copier
Thread startup: copier, TID 000007EC
Init done - copier
Init start - RAIDs
Init done - RAIDs
Init start - monitor
Init done - monitor
Init start - math A
Init done - math A
Init start - NTFS parameters
Init done - NTFS parameters
Init start - FAT parser
Init done - FAT parser
Init start - files
Init done - files
Init start - automatic update
Init done - automatic update
Init start - preview renderer
Init done - preview renderer
Init start - preview renderer
Init done - preview renderer
Init start - image recovery parser
Init done - image recovery parser
Init start - saveload
Init done - saveload
Init start - ident processing
Performance: Ident data query (3 of 3) 0s
Performance: Ident data query (9 of 9) 0s
Performance: Ident data query (2 of 9) 0s
Init done - ident processing
Init start - physical devices
Init done - physical devices
Init start - tasks
Init done - tasks
Init start - threads
Init done - threads
Init start - logging
Init done - logging
NT 5.1.2600 Service Pack 2; 2047 MB RAM

[Alexey V. Gubin]

I read it so that there are about 2.7 millions of files on the volume. About 2'700'000 files and directories combined. Is this the expected number of files?

[Donnie]

2.7 million files, and another 300,000 folders. This is the expected amount.

[Alexey V. Gubin]

Made me thinking. We tested up to 500,000 (files and folders combined). At about six times this, I expect some problems will occur.
I see from the log what the problem is, but I do not think we can resolve this fast enough to be practical.
Could you please PM me your order number so I can arrange the refund?

[Donnie]

I'm not opposed to a refund.

However ZAR is the only software out of a number of programs that I've tried (DiskInternals NTFS Recovery, Easy Recovery Pro, Active@ File Recovery, Pc Inspector File Recovery...none work, all either hang or give access violations) that looks as if it could recover these files. Most are digital photos of my kids since the day they were born, from 2001 through to 2006.

Although I can double click on the D: drive (corrupted NTFS drive) and XP claims the disk is corrupted, CHKDSK claims it can fix the problem. Running CHKDSK in read-only mode finds "minor problems" with thousands of files which suggests that it may only be minor problems but I don't want to mess things up any further by letting CHKDSK try to fix things.

I'm going to install VMWare Server and take a snapshot of the drive and then run CHKDSK so any changes aren't written directly to the drive but to a snapshot instead. That way if I still can't recover anything with chkdsk I can always rollback the snapshot.

[Alexey V. Gubin]

Please PM the order number - this qualifies for a refund regardless of what may happen later.

We're currently looking into some way to reduce the memory footprint so it will fit in 2GB. However, at this point nothing short of a major rework is on the horizon.

Current version holds all the files, file names, and cluster maps (where the file is located) in memory. So we are talking about 400 bytes per file, plus a file name, plus a cluster map. This brings us somewhere into 1KB per file memory requirement. With that and 2GB address space available, the limit is around 2M objects (files and folders combined). Taking into account variuos caches, search indexes, and object lists, the more reaslistic estimate is around 1M objects max.

I'll keep you informed on the progress but that is not going to be quick (considering it took about an hour to just create a test set with 3M files).

[Donnie]

I've sent you my order number. I've got time to wait for something that works. I'll try anything so long as it doesn't write to the original drives.

My idea about using a snapshot under VMWare doesn't work where I'm trying to use a physical disk with my guest. I'd have to make an image first and then work with the image.

I did try installing Win2003 Enterprise and ZAR then detected 3072MB. When running ZAR it did get a bit further in processing the dirve, claiming to be sorting fragments after finding 2842885 files in 313053 directories. Same error though:

....
....
Init start - tasks
Init done - tasks
Init start - threads
Init done - threads
Init start - logging
Init done - logging
NT 5.2.3790 Service Pack 2; 3071 MB RAM
Open physical drive 00000100 success, 465 GB : ST350063 0AS - Port 3, Bus 1, Target 0, LUN 0; maxLBA=976773168
Open physical drive 00000101 success, 894 GB : NVIDIA RAID5 894.27G - Port 0, Bus 0, Target 1, LUN 0; maxLBA=1875427200
Open physical drive 00000102 success, 465 GB : ST350063 9QG1AWM7; maxLBA=976773168
TS: Stop validation
TS: Stopped validation
....
....
....
ALERT!: Excessive MFT attribute size near ap 278; hint 43074
ALERT!: Excessive MFT attribute size near ap 342; hint 43082
ALERT!: Excessive MFT attribute size near ap 342; hint 43082
ALERT!: Excessive MFT attribute size near ap 254; hint 43085
ALERT!: Excessive MFT attribute size near ap 254; hint 43085
ALERT!: Excessive MFT attribute size near ap 342; hint 43086
ALERT!: Excessive MFT attribute size near ap 342; hint 43086
ALERT!: Excessive MFT attribute size near ap 254; hint 43087
ALERT!: Excessive MFT attribute size near ap 254; hint 43087
ALERT!: an exception has occured in TThreadParseNtfs.ReadIn() status Out of memory
Performance: NTFS read-in 16m 27s
Directory tree linking started - total 2836887 objects on volume
Directory structure loop for Uln
ses
Directory structure loop for Cookies
Performance: NTFS rebuild directory tree 1m 5s
Directory tree refinement and reconstruction started
Directory tree refinement and reconstruction pass 1 started for 2836887 objects
ALERT!: an exception has occured in TThreadParseNtfs.Relink() status Out of memory
Performance: NTFS directory tree cleanup 7s
Simple volume - Parse filesystem complete
Fragment reordering start


If I copied this drive into two images at 50% of the original size, would ZAR be just as likely to find files in the disk images? If so it would use 50% of the memory of the full disk.

[Alexey V. Gubin]

Refunded the order.

ZAR will not work on a part of the image. Anyway, the whole MFT needs to be loaded into memory otherwise you have only part of the directory tree available.

[Alexey V. Gubin]

Current estimation is that we will have it in about two more days work.