Thread fault Monitor Thread

[jadams]

I did get a "Thread fault SolverWorkerThread" with version 8.3 build 36, then seen a conversation and downloaded 8.3 build 37, then did my scan again and then it came back with a "Thread Fault Monitor Thread".

I am tring to recover data from a 750GB SAN Volume that is bad.

The tool worked well with the 70GB Volume that I had done the other day.

I have sent the errors when it asks to send them when the job errors out.

Here are a few lines from the last log file......

ERT!: Excessive MFT attribute size near ap 526; hint -1
ALERT!: Excessive MFT attribute size near ap 526; hint -1
ALERT!: Excessive MFT attribute size near ap 526; hint -1
ALERT!: Excessive MFT attribute size near ap 534; hint -1
ALERT!: zero-sized MFT attribute
Purged unresolved MFT region: 0001 : 104 entries @ LBA 1048592160
Purged unresolved MFT region: 0001 : 61 entries @ LBA 1048592374
Purged unresolved MFT region: 0001 : 155 entries @ LBA 1048593298
Purged unresolved MFT region: 0001 : 119 entries @ LBA 1048593674
Purged unresolved MFT region: 0001 : 116 entries @ LBA 1048594768
Purged unresolved MFT region: 0001 : 117 entries @ LBA 1048595182
Purged unresolved MFT region: 0001 : 131 entries @ LBA 1048595738
Purged unresolved MFT region: 0001 : 143 entries @ LBA 1048596314
Purged unresolved MFT region: 0001 : 98 entries @ LBA 1321206563
Purged unresolved MFT region: 0002 : 138 entries @ LBA 1321209627
Purged unresolved MFT region: 0003 : 60 entries @ LBA 1321213335
Purged unresolved MFT region: 0003 : 62 entries @ LBA 1321213563
Purged unresolved MFT region: 0003 : 318 entries @ LBA 1321214331
Purged unresolved MFT region: 0003 : 302 entries @ LBA 1321218459
Purged unresolved MFT region: 0003 : 128 entries @ LBA 1321219967
Init done - logging
NT 5.2.3790 Service Pack 2; 3325 MB RAM
Open physical drive 00000100 success, 273 GB : HP LOGICAL VOLUME - Port 2, Bus 0, Target 4, LUN 0; maxLBA=573235602
Open physical drive 00000101 success, 752 GB : StoneFly Logical Volume - Port 3, Bus 0, Target 0, LUN 0; maxLBA=1577058304
Open physical drive 00000102 success, 750 GB : StoneFly Logical Volume - Port 3, Bus 0, Target 1, LUN 0; maxLBA=1572864000
Start read: 0 ident ranges
Done reading ident ranges
PD - IDLL = 0
Start read: 0 ident ranges
Done reading ident ranges
PD - IDLL = 0
Start read: 16179 ident ranges
Done reading ident ranges
PD - IDLL = 0
WfHide() start
WfHide() done
Sector 0, row 1 - 0/0 @ base 0 - Empty
Sector 0, row 2 - 0/0 @ base 0 - Empty
Sector 0, row 3 - 0/0 @ base 0 - Empty
Sector 0, row 4 - 0/0 @ base 0 - Empty
ALERT!: MFT entry with alloc size 0 was encountered during identification; Base record ID 000000
ALERT!: MFT entry with alloc size 0 was encountered during identification; Base record ID 000000
Partition scan: user cancelled @LBA 396084864
Performance: Missing partition scan 2h 44m 33s
Processing volume ID 01020300
-------------------------------------
Volume information for ID 01020300
Origin : User input
Partition type : Unknown
Capacity : 750 GB
Number of sectors : 1572864000
RAID type : Simple volume
Stripe size : 1
Rotation parameters: 0 / 0
Member 00 : 1572864000 sectors at LBA 0 on device 0102
The device StoneFly Logical Volume - Port 3, Bus 0, Target 1, LUN 0, ID 0102 is not S.M.A.R.T.-capable
-------------------------------------
Volume information for ID 01020300
Origin : User input
Partition type : Unknown
Capacity : 750 GB
Number of sectors : 1572864000
RAID type : Simple volume
Stripe size : 1
Rotation parameters: 0 / 0
Member 00 : 1572864000 sectors at LBA 0 on device 0102
The device StoneFly Logical Volume - Port 3, Bus 0, Target 1, LUN 0, ID 0102 is not S.M.A.R.T.-capable
Starting raw scanner for a total of 1572864000 sectors
Raw scanner starting on disk 0102 from LBA 0 to LBA 1572864000
Quick scan failed, reverting to full scan mode
TS: Stop validation
TS: Stopped validation
Logging - shut down
Init done - logging
NT 5.2.3790 Service Pack 2; 3325 MB RAM
Open physical drive 00000100 success, 273 GB : HP LOGICAL VOLUME - Port 2, Bus 0, Target 4, LUN 0; maxLBA=573235602
Open physical drive 00000101 success, 752 GB : StoneFly Logical Volume - Port 3, Bus 0, Target 0, LUN 0; maxLBA=1577058304
Open physical drive 00000102 success, 750 GB : StoneFly Logical Volume - Port 3, Bus 0, Target 1, LUN 0; maxLBA=1572864000
Sector 0, row 1 - 0/0 @ base 0 - Empty
Sector 0, row 2 - 0/0 @ base 0 - Empty
Sector 0, row 3 - 0/0 @ base 0 - Empty
Sector 0, row 4 - 0/0 @ base 0 - Empty
Ldm: started processing of the device 0100
Ldm: started processing of the device 0101
Ldm: started processing of the device 0102
Processing volume ID 01020300
-------------------------------------
Volume information for ID 01020300
Origin : User input
Partition type : Unknown
Capacity : 750 GB
Number of sectors : 1572864000
RAID type : Simple volume
Stripe size : 1
Rotation parameters: 0 / 0
Member 00 : 1572864000 sectors at LBA 0 on device 0102
The device StoneFly Logical Volume - Port 3, Bus 0, Target 1, LUN 0, ID 0102 is not S.M.A.R.T.-capable
-------------------------------------
Volume information for ID 01020300
Origin : User input
Partition type : Unknown
Capacity : 750 GB
Number of sectors : 1572864000
RAID type : Simple volume
Stripe size : 1
Rotation parameters: 0 / 0
Member 00 : 1572864000 sectors at LBA 0 on device 0102
The device StoneFly Logical Volume - Port 3, Bus 0, Target 1, LUN 0, ID 0102 is not S.M.A.R.T.-capable
Starting raw scanner for a total of 1572864000 sectors
Raw scanner starting on disk 0102 from LBA 0 to LBA 1572864000
Quick scan failed, reverting to full scan mode
TS: Stop validation
TS: Stopped validation
Logging - shut down
Init done - logging
NT 5.2.3790 Service Pack 2; 3325 MB RAM
TS: Stop validation
TS: Stopped validation
Logging - shut down
Init done - logging
NT 5.2.3790 Service Pack 2; 3325 MB RAM
TS: Stop validation
TS: Stopped validation
Logging - shut down

[Alexey V. Gubin]

10 million files is just big. We're just running out of the address space. I see from the log that you have a server version of Windows, try this (we tested up to 11 millions files, and theoretically it would probably handle 12M):

1. Add a /3GB switch into boot.ini. refer to http://technet.microsoft.com/en-us/l.../bb124810.aspx

2. Restart ZAR, and in "Runtime Control" panel (left side of the winodw) have cache size set at minimums.

However, without that /3GB switch, it can handle maximum of about 7..8 million files.

[jadams]

Thank you I will give that a try, yes I know that is a large number of files.

They are mostly image files, on the SAN.

[Alexey V. Gubin]

When we first encountered that, the limit turned out to be about just one million files, somewhere around version 7.2 or so. We reworked the easy part and got it up to 11-12M files with /3GB. If it still fails on you, we probably try reworking the heavy part. This should bring the limit up to estimated 20-25M files, but implementation would be one week optimistic, two weeks realistic.

Anyway I'm waiting for your report on how it went.

[jadams]

Well..........

It was so so close.......... I was at 88% and then it errored out on me again.

I jsut need about another 20%, would there be anything else I might try to get me there.

I understand that I do have a high number of images on the san. Any help would greatly be appreciated.

[jadams]

Would there or is there a way to recover 7 million records and then run it again and recover another 7 million records in chunks. So that I will not error out with the one big run?

Recoverying the data in pieces and running the utility is not a problem if it can be done. can it work like that some how?

Or to scan in 100GB blocks of data? Sort of a start and stop data size?

Just that we are so close and it is data that I need to get back.

[Alexey V. Gubin]

The problem is that we really need a complete dataset for a cross-referencing. Consider directory tree rebuild. If the directory is in one "chunk" and the contained files are in the other "chunk", that's bad.

Also, the deleted files and folders are analyzed as a part of the same set.

Suppose we can get a result in two weeks, would you still be interested in it by that time?

[Alexey V. Gubin]

Btw, 88 percent of which stage it was?

[jadams]

The 88% is when it is doing the Parsing step, just before it goes to the file structure, where I could pick out the files to restore. I am at this point only interested in the current data on the disk, and not worried about getting any deleted files back. Just Current data on disk.

I was hoping that when I only sleceted the current files that I might maked it to the end, to start restoring some files. But... there are still to many files.

I was able to get my other disk volumes back, but the mox on that was 1.8 Million, so it worked like a champ.

I belive on the 750GB I am looking at about 17 Million files. I believe...

[Alexey V. Gubin]

I think we will be just fine with that. I've unfortunately recycled a rig from a previous experiment (when I tried on 11M files), but that can be rebuilt quickly.

Couple more questions,
1. I see there is no RAID involved, just a regular volume, correct?
2. Approximately how long your last attempt took, from start of the run to the failure?