How can lightroom possibly corrupt files during import?

[RzzB]

Using...

Lightroom 3.4.1
Windows 7 (64bit) - fully up to date with patches
System Rating 5.9
Processor i7 CPU - [email protected] - no overclocking
Memory - 6 GB
Display Adaptor - Nvidia GeForce GTX 460
No yellow bits in device manager

Canon 5D Mk II and 7D
Sandisk Extreme 60MB/s 32GB & 8GB
Sandisk Card reader - directly plugged

This happens all the time - but for example - yesterday I took 350 photos, when importing them directly from the CF card with Lightroom, seven of them were corrupt.

However, if I manually copy the files to the HD and then import into lightroom - no corruption.

I have used a hex comparison tool to compare the good and bad files and sure enough there are a couple of picked bits here and there.

I'm a bit confused. I have some basic questions...

How can Lightroom introduce corruption at this stage? Seems implausible?

If there was a corruption - why on earth wasn't it detected at transfer time? Surely there is some sort of basic checking, CRC at a minimum - yes/no?

Thanks,
RzzB

 

[DonRicklin]

Likely bad CF card reader, cable or memory. Try different reader of direct from camera connection.

Don

 

[RzzB]

Don,

Thanks for your response.

This occurs with different card readers, connected direct and through a hub. The host machine is very stable so I doubt that I have a memory problem of this magnitude.

Let me be very clear ...

When I transfer the files using Windows Explorer there are NO corrupted files.

When I transfer the files using Lightroom I get corrupted files.

It's as if Lightroom is using a different transfer mechanism that is causing corruption.

What is worrying me more is - why aren't the corruptions detected and an error raised? I believe USB transfers have a CRC checking / retry mechanism built in for most transfer modes. Is Lightroom perhaps using a modes that has this turned off?

Thanks,
RzzB

 

[DonRicklin]

Okay, time to post screen shots of some of these files in LR so we can see what corruption you are talking about.

Don

 

[Victoria Bampton]

Hi Rzzb, welcome to the forum!

I can certainly understand your concern, and that is a very odd problem. It's not one that I've heard any other reports of, so I would still look at hardware amongst the other possibilities.

I assume you've ruled out the other possibilities, such as a hard drive, and you've run a low level memory test such as Memtest? Perhaps LR is accessing memory that has a minor glitch, whereas Explorer isn't hitting it? LR's very good at finding the most hidden minor memory glitches, and it could cause corruption, so it would be worth ruling out.

And how are you discovering that the transferred files are corrupted? Is LR telling you, or are you spotting it in the previews? Or just in the hex comparison that you're seeing it?

 

[RzzB]

Many thanks for your responses guys.

I'm detecting the corruption because some of the images in the preview window are a mess - see the attached. When I inspect these files (hex comparison) with the versions that I transferred manually I can see that there are some picked/dropped bits. I can see no pattern sensitivity - they look pretty random.

I have done nothing to rule out hard drive problem, but I can't see how this could be associated with a hard drive problem. Not sure what I should do? Thoughts?

I haven't run memtest recently. I did run all sorts of test when I originally built the machine (6 months ago) - Memtest, Burnin, Prime95 - just to make sure I had a stable system. But I agree - it would be sensible to run Memtest again to be sure. I will do this in the near future and report back - might be in a couple of days.

I do agree that this feels like a hardware problem, but it seems to be exacerbated by Lightroom.

Possibly memory - I'll do some testing as mentioned above.

The other thing I need to do it double check all the failing scenarios - just to make sure. ie Cables, readers, hubs etc. I'll probably do this over the weekend.

What concerns me is that none of the software involved is detecting the corruption and raising an error.

Many thanks for your help,
RzzB

 

[Kiwigeoff]

What is the build number of Lightroom? Check in system info in the Help drop down menu.
This looks similar to the issue with 3.4.

 

[Hal P Anderson]

Geoff,

Wasn't that problem with jpegs from a non-Canon?

Hal

 

[Kiwigeoff]

I have no idea Hal, but it looked like something similar I think.

 

[b_gossweiler]

The issue was noticed with images from a HP PhotoSmart R607, but you never know ....

Beat

 

[RzzB]

Thanks for your interest gents.

Lightroom version is 3.4.1 - see first post for other info.

One thing I haven't mentioned is that these are all raw CR2 files. I only shoot in raw.

Thanks,
RzzB

 

[Kiwigeoff]

I have another example here from a forum that looks very similar to yours and funnily enough from a Canon 40D.
I seem to recall it being something to do with the drive or connections.
Did you move them from one drive to another when importing? Where do you store your files - on an eternal drive???

 

[RzzB]

I have another example here from a forum that looks very similar to yours and funnily enough from a Canon 40D.
I seem to recall it being something to do with the drive or connections.
Did you move them from one drive to another when importing? Where do you store your files - on an eternal drive???

Geoff,

Many thanks for your response - could you post a link to the other example please. My cameras are a 7D and 5D but similar in respect of being Canon.

The importing is done from a CF card using a card reader directly to my D: drive which is a 1T mirrored raid drive. I do have external backup storage in the form of Synology DS210 boxes but I never import to them.

I have a non Raid drive in my machine - perhaps I'll create a new catalogue and import to that drive just to rule out any Lightroom/Raid problems.

Thanks for the thought...

Regards,
RzzB

 

[Kiwigeoff]

I have the file on my computer and it looks like this:

Interesting it was on a D drive as well. I'm trying to recall what the solution was but am too busy at the moment to search. I'd think your idea to test on your C drive would be a good one!!

 

[RzzB]

Geoff,
...... snip.....
I have a non Raid drive in my machine - perhaps I'll create a new catalogue and import to that drive just to rule out any Lightroom/Raid problems.

Hmmm... I'm now beginning to regret starting this thread :-(


I tried using the non-raid drive - and hey presto no corruption!


However, I retried to the raid drive also and guess what - it worked ok there too.


In the last couple of days I have transferred the same 350 images using all sorts of hardware and I have seen just one corrupt file.


So - something has changed - and I have been racking my brain trying to work out what.


Oh well - I think I now have to wait till I see the corruptions again and look at what is happening on my machine at the time.


Thanks,
RzzB

 

[Victoria Bampton]

That sounds like an excellent plan RzzB. Funny how things fix themselves when you say them out loud!

 

[Brad Snyder]

Good news.

I'd say we don't like to come across as Lightroom fanboys, but I don't think I've ever seen convincing evidence that Lr can cause raw file corruption at the import or editing stages. Lr never touches the raw files at all (with one arcane exception) and uses the OS services exclusively for file operations.

But if you just flat out say that, it sounds like defending an Adobe who can do no wrong, which is patently false.

 

[Roomservice]

... I tried using the non-raid drive - and hey presto no corruption!
However, I retried to the raid drive also and guess what - it worked ok there too.
In the last couple of days I have transferred the same 350 images using all sorts of hardware and I have seen just one corrupt file.
...

It seems this is the same problem I have:
when importing 400 cr2 files directly from the camera into LR, one (1) file gets corrupted. It does not happen every time.
And: when I transfer the files using Windows Explorer there are NO corrupted files.

Lightroom 3.4.1
Windows 7 (64bit)
Canon 5D Mk II
Sandisk Extreme 60MB/s 16GB

Checked memory and hard drive.
The corrupt file seams to appear after 6-8 GB of the 10 GB has been transfered.
I have also used a hex comparison tool to compare the good and bad files, and there is a certain pattern: a multiple of the 6th and 14th byte are replaced... somewhere in the last 20-40% of the file content.

 

[Roomservice]

Maybe the file corrupt problem is related to this problem:

Today I took some test shots with camera setting RAW+jpg, to another empty memory card.
When importing them to LR it hanged. I made that import repeatedly: sometimes LR freezes, sometimes not. When it freezes you can see in windows explorer that it has lost the connection to the card. Does Lightroom cause the card to disconnect?

But there are no problems when copying with Windows explorer.
So RAM, hard disk, memory card, camera and usb cable seems ok (and RAM + hard disk are tested).
But when Lightroom is involved...

Any suggestions would be appreciated

 

[clee01l]

Roomservice, welcome to the forum. Unless you have actually used a second USB cable, you can not eliminate 'Badd Cable" from the list of possible causes. By using two different cards, you probably have eliminated the "Bad Card" cause (Or validated that you have two bad cards). The camera needs a fresh battery to begin the transfer, Using a real card reader instead of the camear can help isolate the camera/battery as a cause.

If you can successfully copy images to the computer and Import them into LR from the computer location, you can then remove the camera/card/cable as a source of the problem.

I would also like to note that with flash memory, it very important to give the camera time to write its buffered image data to the card before shutting the camera off or ejecting the card from the camera. Either operation while the file is still being written can result is a corrupt image.

 

[Roomservice]

Hi clee01, and thank you for the feedback.

The files are getting corrupt during/after the transfer (copy) to Lightroom. They are OK on the CF card.

When copying with Windows explorer - with the same cable/card/camera - there is no problem. Neither with lost usb connections, hanging, or corrupt files. I have also tested another cable (from a canon 60D).

There are 2 problems during file transfer: LR hangs and files gets corrupt when using LR. I believe LR hangs because the connection to the card is lost during the transfer. Maybe Lightroom has some problem with the camera windows driver? The camera driver from camera CD is installed (it's actually a MS driver from 2006).

I will test with a card reader, even though it works fine with the camera when using Windows explorer and LR is not involved.

 

[Roomservice]

A clarification about the hanging:

LR hangs during the initial phase of import (sometimes, not always):
after clicking the left "Import..." button, and before clicking the right "Import" button.

Note:
Historically Lightroom has had problems with freezing, check out this almost identical problem from en early version:
http://kb2.adobe.com/cps/401/kb401357.html
That problem seem had something to do with the previews.

Some threads about hanging issues in current versions of LR:
Lightroom 3 freezing when using import:
http://forums.adobe.com/thread/664110
LR 3 freeze when trying to import (with a workaround):
http://forums.adobe.com/message/2981433
Horrendous import problems in Lightroom 3 64bit:
http://forums.adobe.com/message/3022353

 

[clee01l]

I'd say we don't like to come across as Lightroom fanboys, but I don't think I've ever seen convincing evidence that Lr can cause raw file corruption at the import or editing stages. Lr never touches the raw files at all (with one arcane exception) and uses the OS services exclusively for file operations.

quoted from Brad Snyder above. If you are an exception to this with your Canon, you will be the first. When you copied these CR2 files with windows, what viewer did you use to view the files? Most viewers will only show you the embedded JPEG thumbnail in the RAW. The JPEG thumbnail in in the header section of the file and not in the compressed RAW data section that follows. The Compressed raw data could still be corrupt. What happened when you tried to import these files that you successfully copied via windows?. LR only calls standard OS File-handling APIs to do the file operations like read, copy, and write. These are the same APIs that the built in file import Windows applet uses when it imports images from a card to a HD.

And as for USB cables, I'll leave you with this comforting thought. Remember, Canon is a camera manufacturer. All of their cables distributed with their camera likely came from a supplier that was lowest bidder.

You mentioned the camera device driver. There is a possibility that it is corrupt. Have you checked Canon for updates? Or perhaps just re install the driver that came with your camera.

 

[clee01l]

A clarification about the hanging:

LR hangs during the initial phase of import (sometimes, not always):...

Hanging is not the same as corrupted image files, Which is your problem? I have been giving you remedies for images that get corrupted between the time the camera writes to the camera card and viewing the file that was transferred to the HD with LR.

 

[Roomservice]

quoted from Brad Snyder above. If you are an exception to this with your Canon, you will be the first. When you copied these CR2 files with windows, what viewer did you use to view the files? Most viewers will only show you the embedded JPEG thumbnail in the RAW. The JPEG thumbnail in in the header section of the file and not in the compressed RAW data section that follows. The Compressed raw data could still be corrupt. What happened when you tried to import these files that you successfully copied via windows?. LR only calls standard OS File-handling APIs to do the file operations like read, copy, and write. These are the same APIs that the built in file import Windows applet uses when it imports images from a card to a HD.

And as for USB cables, I'll leave you with this comforting thought. Remember, Canon is a camera manufacturer. All of their cables distributed with their camera likely came from a supplier that was lowest bidder.

You mentioned the camera device driver. There is a possibility that it is corrupt. Have you checked Canon for updates? Or perhaps just re install the driver that came with your camera.

Yes today I reinstalled the driver from the CD. I've also checked canons site but there are no upgrades regarding that driver.
About the cables: I don't think any of the two cables are bad since they work fine when LR is not involved.

I checked the raw files with two methods: added them to LR to be able to see the raw files (and not just the embedded jpg's as you pointed out). And binary compare with UltraCompare, which verified the previous result.

Next thing to do is to test a card reader. And also test importing from hard disk to LR.

Maybe there is no bug in LR. But if there is, it probably makes the software do something quite unexpected (as bugs often do).