1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Welcome to Lightroom Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to register and post your first thread under Help on the menubar above. You're also welcome to join our newsletter, download our free Lightroom Quick Start eBook and explore our other FAQ resources.
    Dismiss Notice

How can lightroom possibly corrupt files during import?

Discussion in 'Lightroom Troubleshooting (ARCHIVE)' started by RzzB, Jul 3, 2011.

  1. RzzB

    RzzB New Member

    Joined:
    Jul 3, 2011
    Messages:
    6
    Lightroom Experience:
    Advanced
    Primary OS:
    Windows 7 64-bit
    Using...

    Lightroom 3.4.1
    Windows 7 (64bit) - fully up to date with patches
    System Rating 5.9
    Processor i7 CPU - 930@2.8Ghz - no overclocking
    Memory - 6 GB
    Display Adaptor - Nvidia GeForce GTX 460
    No yellow bits in device manager :)

    Canon 5D Mk II and 7D
    Sandisk Extreme 60MB/s 32GB & 8GB
    Sandisk Card reader - directly plugged

    This happens all the time - but for example - yesterday I took 350 photos, when importing them directly from the CF card with Lightroom, seven of them were corrupt.

    However, if I manually copy the files to the HD and then import into lightroom - no corruption.

    I have used a hex comparison tool to compare the good and bad files and sure enough there are a couple of picked bits here and there.

    I'm a bit confused. I have some basic questions...

    How can Lightroom introduce corruption at this stage? Seems implausible?

    If there was a corruption - why on earth wasn't it detected at transfer time? Surely there is some sort of basic checking, CRC at a minimum - yes/no?

    Thanks,
    RzzB
     
    Last edited: Jul 3, 2011
  2. DonRicklin

    DonRicklin Lightroom Guru Lightroom Guru

    Joined:
    Oct 7, 2007
    Messages:
    2,189
    Location:
    Worcester, MA
    Lightroom Experience:
    Advanced
    Primary OS:
    Mac 10.10.x Yosemite
    Lightroom Version:
    6.3 / CC 2015.3
    Likely bad CF card reader, cable or memory. Try different reader of direct from camera connection.

    Don
     
  3. RzzB

    RzzB New Member

    Joined:
    Jul 3, 2011
    Messages:
    6
    Lightroom Experience:
    Advanced
    Primary OS:
    Windows 7 64-bit
    Don,

    Thanks for your response.

    This occurs with different card readers, connected direct and through a hub. The host machine is very stable so I doubt that I have a memory problem of this magnitude.

    Let me be very clear ...

    When I transfer the files using Windows Explorer there are NO corrupted files.

    When I transfer the files using Lightroom I get corrupted files.

    It's as if Lightroom is using a different transfer mechanism that is causing corruption.

    What is worrying me more is - why aren't the corruptions detected and an error raised? I believe USB transfers have a CRC checking / retry mechanism built in for most transfer modes. Is Lightroom perhaps using a modes that has this turned off?

    Thanks,
    RzzB
     
  4. DonRicklin

    DonRicklin Lightroom Guru Lightroom Guru

    Joined:
    Oct 7, 2007
    Messages:
    2,189
    Location:
    Worcester, MA
    Lightroom Experience:
    Advanced
    Primary OS:
    Mac 10.10.x Yosemite
    Lightroom Version:
    6.3 / CC 2015.3
    Okay, time to post screen shots of some of these files in LR so we can see what corruption you are talking about.

    Don
     
  5. Victoria Bampton

    Victoria Bampton Lightroom Queen / Owner Staff Member Administrator Moderator Lightroom Guru

    Joined:
    Sep 29, 2007
    Messages:
    18,303
    Location:
    Southampton, UK
    Lightroom Experience:
    Power User
    Primary OS:
    Mac 10.12.x Sierra
    Lightroom Version:
    6.7 / CC 2015.7
    Lightroom License:
    CC Subscription
    Hi Rzzb, welcome to the forum!

    I can certainly understand your concern, and that is a very odd problem. It's not one that I've heard any other reports of, so I would still look at hardware amongst the other possibilities.

    I assume you've ruled out the other possibilities, such as a hard drive, and you've run a low level memory test such as Memtest? Perhaps LR is accessing memory that has a minor glitch, whereas Explorer isn't hitting it? LR's very good at finding the most hidden minor memory glitches, and it could cause corruption, so it would be worth ruling out.

    And how are you discovering that the transferred files are corrupted? Is LR telling you, or are you spotting it in the previews? Or just in the hex comparison that you're seeing it?
     
  6. RzzB

    RzzB New Member

    Joined:
    Jul 3, 2011
    Messages:
    6
    Lightroom Experience:
    Advanced
    Primary OS:
    Windows 7 64-bit
    Many thanks for your responses guys.

    I'm detecting the corruption because some of the images in the preview window are a mess - see the attached. When I inspect these files (hex comparison) with the versions that I transferred manually I can see that there are some picked/dropped bits. I can see no pattern sensitivity - they look pretty random.

    I have done nothing to rule out hard drive problem, but I can't see how this could be associated with a hard drive problem. Not sure what I should do? Thoughts?

    I haven't run memtest recently. I did run all sorts of test when I originally built the machine (6 months ago) - Memtest, Burnin, Prime95 - just to make sure I had a stable system. But I agree - it would be sensible to run Memtest again to be sure. I will do this in the near future and report back - might be in a couple of days.

    I do agree that this feels like a hardware problem, but it seems to be exacerbated by Lightroom.

    Possibly memory - I'll do some testing as mentioned above.

    The other thing I need to do it double check all the failing scenarios - just to make sure. ie Cables, readers, hubs etc. I'll probably do this over the weekend.

    What concerns me is that none of the software involved is detecting the corruption and raising an error.

    Many thanks for your help,
    RzzB
     

    Attached Files:

  7. Kiwigeoff

    Kiwigeoff Lightroom Guru Lightroom Guru

    Joined:
    Oct 7, 2007
    Messages:
    4,627
    Location:
    Paradise
    Lightroom Experience:
    Power User
    Primary OS:
    Mac 10.10.x Yosemite
    Lightroom Version:
    5.7
    What is the build number of Lightroom? Check in system info in the Help drop down menu.
    This looks similar to the issue with 3.4.
     
  8. Hal P Anderson

    Hal P Anderson Lightroom Guru Staff Member Lightroom Guru

    Joined:
    May 12, 2011
    Messages:
    2,070
    Location:
    Canada
    Lightroom Experience:
    Advanced
    Primary OS:
    Windows 10
    Lightroom Version:
    6.6 / CC 2015.6
    Lightroom License:
    CC Subscription
    Geoff,

    Wasn't that problem with jpegs from a non-Canon?

    Hal
     
  9. Kiwigeoff

    Kiwigeoff Lightroom Guru Lightroom Guru

    Joined:
    Oct 7, 2007
    Messages:
    4,627
    Location:
    Paradise
    Lightroom Experience:
    Power User
    Primary OS:
    Mac 10.10.x Yosemite
    Lightroom Version:
    5.7
    I have no idea Hal, but it looked like something similar I think.
     
  10. b_gossweiler

    b_gossweiler Lightroom Guru Lightroom Guru

    Joined:
    Jun 19, 2009
    Messages:
    3,207
    Location:
    Zurich, Switzerland
    Lightroom Experience:
    Advanced
    Primary OS:
    Windows 7 64-bit
    The issue was noticed with images from a HP PhotoSmart R607, but you never know ....

    Beat
     
  11. RzzB

    RzzB New Member

    Joined:
    Jul 3, 2011
    Messages:
    6
    Lightroom Experience:
    Advanced
    Primary OS:
    Windows 7 64-bit
    Thanks for your interest gents.

    Lightroom version is 3.4.1 - see first post for other info.

    One thing I haven't mentioned is that these are all raw CR2 files. I only shoot in raw.

    Thanks,
    RzzB
     
  12. Kiwigeoff

    Kiwigeoff Lightroom Guru Lightroom Guru

    Joined:
    Oct 7, 2007
    Messages:
    4,627
    Location:
    Paradise
    Lightroom Experience:
    Power User
    Primary OS:
    Mac 10.10.x Yosemite
    Lightroom Version:
    5.7
    I have another example here from a forum that looks very similar to yours and funnily enough from a Canon 40D.
    I seem to recall it being something to do with the drive or connections.
    Did you move them from one drive to another when importing? Where do you store your files - on an eternal drive???
     
  13. RzzB

    RzzB New Member

    Joined:
    Jul 3, 2011
    Messages:
    6
    Lightroom Experience:
    Advanced
    Primary OS:
    Windows 7 64-bit
    Geoff,

    Many thanks for your response - could you post a link to the other example please. My cameras are a 7D and 5D but similar in respect of being Canon.

    The importing is done from a CF card using a card reader directly to my D: drive which is a 1T mirrored raid drive. I do have external backup storage in the form of Synology DS210 boxes but I never import to them.

    I have a non Raid drive in my machine - perhaps I'll create a new catalogue and import to that drive just to rule out any Lightroom/Raid problems.

    Thanks for the thought...

    Regards,
    RzzB
     
  14. Kiwigeoff

    Kiwigeoff Lightroom Guru Lightroom Guru

    Joined:
    Oct 7, 2007
    Messages:
    4,627
    Location:
    Paradise
    Lightroom Experience:
    Power User
    Primary OS:
    Mac 10.10.x Yosemite
    Lightroom Version:
    5.7
    I have the file on my computer and it looks like this:
    IMG_9058.jpg
    Interesting it was on a D drive as well. I'm trying to recall what the solution was but am too busy at the moment to search. I'd think your idea to test on your C drive would be a good one!!
     
  15. RzzB

    RzzB New Member

    Joined:
    Jul 3, 2011
    Messages:
    6
    Lightroom Experience:
    Advanced
    Primary OS:
    Windows 7 64-bit
    Hmmm... I'm now beginning to regret starting this thread :(


    I tried using the non-raid drive - and hey presto no corruption!


    However, I retried to the raid drive also and guess what - it worked ok there too.


    In the last couple of days I have transferred the same 350 images using all sorts of hardware and I have seen just one corrupt file.


    So - something has changed - and I have been racking my brain trying to work out what.


    Oh well - I think I now have to wait till I see the corruptions again and look at what is happening on my machine at the time.


    Thanks,
    RzzB
     
  16. Victoria Bampton

    Victoria Bampton Lightroom Queen / Owner Staff Member Administrator Moderator Lightroom Guru

    Joined:
    Sep 29, 2007
    Messages:
    18,303
    Location:
    Southampton, UK
    Lightroom Experience:
    Power User
    Primary OS:
    Mac 10.12.x Sierra
    Lightroom Version:
    6.7 / CC 2015.7
    Lightroom License:
    CC Subscription
    That sounds like an excellent plan RzzB. Funny how things fix themselves when you say them out loud! ;)
     
  17. Brad Snyder

    Brad Snyder Lightroom Guru Lightroom Guru

    Joined:
    Nov 20, 2007
    Messages:
    6,293
    Location:
    Port Deposit, Maryland USA
    Lightroom Experience:
    Intermediate
    Primary OS:
    Windows 7 64-bit
    Good news.

    I'd say we don't like to come across as Lightroom fanboys, but I don't think I've ever seen convincing evidence that Lr can cause raw file corruption at the import or editing stages. Lr never touches the raw files at all (with one arcane exception) and uses the OS services exclusively for file operations.

    But if you just flat out say that, it sounds like defending an Adobe who can do no wrong, which is patently false.
     
  18. Roomservice

    Roomservice New Member

    Joined:
    Jul 17, 2011
    Messages:
    11
    Lightroom Experience:
    Intermediate
    Primary OS:
    Windows 7 64-bit
    It seems this is the same problem I have:
    when importing 400 cr2 files directly from the camera into LR, one (1) file gets corrupted. It does not happen every time.
    And: when I transfer the files using Windows Explorer there are NO corrupted files.

    Lightroom 3.4.1
    Windows 7 (64bit)
    Canon 5D Mk II
    Sandisk Extreme 60MB/s 16GB

    Checked memory and hard drive.
    The corrupt file seams to appear after 6-8 GB of the 10 GB has been transfered.
    I have also used a hex comparison tool to compare the good and bad files, and there is a certain pattern: a multiple of the 6th and 14th byte are replaced... somewhere in the last 20-40% of the file content.
     
    Last edited: Jul 17, 2011
  19. Roomservice

    Roomservice New Member

    Joined:
    Jul 17, 2011
    Messages:
    11
    Lightroom Experience:
    Intermediate
    Primary OS:
    Windows 7 64-bit
    Maybe the file corrupt problem is related to this problem:

    Today I took some test shots with camera setting RAW+jpg, to another empty memory card.
    When importing them to LR it hanged. I made that import repeatedly: sometimes LR freezes, sometimes not. When it freezes you can see in windows explorer that it has lost the connection to the card. Does Lightroom cause the card to disconnect?

    But there are no problems when copying with Windows explorer.
    So RAM, hard disk, memory card, camera and usb cable seems ok (and RAM + hard disk are tested).
    But when Lightroom is involved...

    Any suggestions would be appreciated :)
     
  20. clee01l

    clee01l Lightroom Guru Staff Member Lightroom Guru

    Joined:
    Jun 20, 2009
    Messages:
    11,265
    Location:
    Bellaire, TX USA
    Lightroom Experience:
    Power User
    Primary OS:
    Mac 10.11.x El Capitan
    Lightroom Version:
    6.6 / CC 2015.6
    Lightroom License:
    CC Subscription
    Roomservice, welcome to the forum. Unless you have actually used a second USB cable, you can not eliminate 'Badd Cable" from the list of possible causes. By using two different cards, you probably have eliminated the "Bad Card" cause (Or validated that you have two bad cards). The camera needs a fresh battery to begin the transfer, Using a real card reader instead of the camear can help isolate the camera/battery as a cause.

    If you can successfully copy images to the computer and Import them into LR from the computer location, you can then remove the camera/card/cable as a source of the problem.

    I would also like to note that with flash memory, it very important to give the camera time to write its buffered image data to the card before shutting the camera off or ejecting the card from the camera. Either operation while the file is still being written can result is a corrupt image.
     
  21. Roomservice

    Roomservice New Member

    Joined:
    Jul 17, 2011
    Messages:
    11
    Lightroom Experience:
    Intermediate
    Primary OS:
    Windows 7 64-bit
    Hi clee01, and thank you for the feedback.

    The files are getting corrupt during/after the transfer (copy) to Lightroom. They are OK on the CF card.

    When copying with Windows explorer - with the same cable/card/camera - there is no problem. Neither with lost usb connections, hanging, or corrupt files. I have also tested another cable (from a canon 60D).

    There are 2 problems during file transfer: LR hangs and files gets corrupt when using LR. I believe LR hangs because the connection to the card is lost during the transfer. Maybe Lightroom has some problem with the camera windows driver? The camera driver from camera CD is installed (it's actually a MS driver from 2006).

    I will test with a card reader, even though it works fine with the camera when using Windows explorer and LR is not involved.
     
  22. Roomservice

    Roomservice New Member

    Joined:
    Jul 17, 2011
    Messages:
    11
    Lightroom Experience:
    Intermediate
    Primary OS:
    Windows 7 64-bit
    A clarification about the hanging:

    LR hangs during the initial phase of import (sometimes, not always):
    after clicking the left "Import..." button, and before clicking the right "Import" button.

    Note:
    Historically Lightroom has had problems with freezing, check out this almost identical problem from en early version:
    http://kb2.adobe.com/cps/401/kb401357.html
    That problem seem had something to do with the previews.

    Some threads about hanging issues in current versions of LR:
    Lightroom 3 freezing when using import:
    http://forums.adobe.com/thread/664110
    LR 3 freeze when trying to import (with a workaround):
    http://forums.adobe.com/message/2981433
    Horrendous import problems in Lightroom 3 64bit:
    http://forums.adobe.com/message/3022353
     
  23. clee01l

    clee01l Lightroom Guru Staff Member Lightroom Guru

    Joined:
    Jun 20, 2009
    Messages:
    11,265
    Location:
    Bellaire, TX USA
    Lightroom Experience:
    Power User
    Primary OS:
    Mac 10.11.x El Capitan
    Lightroom Version:
    6.6 / CC 2015.6
    Lightroom License:
    CC Subscription
    quoted from Brad Snyder above. If you are an exception to this with your Canon, you will be the first. When you copied these CR2 files with windows, what viewer did you use to view the files? Most viewers will only show you the embedded JPEG thumbnail in the RAW. The JPEG thumbnail in in the header section of the file and not in the compressed RAW data section that follows. The Compressed raw data could still be corrupt. What happened when you tried to import these files that you successfully copied via windows?. LR only calls standard OS File-handling APIs to do the file operations like read, copy, and write. These are the same APIs that the built in file import Windows applet uses when it imports images from a card to a HD.

    And as for USB cables, I'll leave you with this comforting thought. Remember, Canon is a camera manufacturer. All of their cables distributed with their camera likely came from a supplier that was lowest bidder.

    You mentioned the camera device driver. There is a possibility that it is corrupt. Have you checked Canon for updates? Or perhaps just re install the driver that came with your camera.
     
  24. clee01l

    clee01l Lightroom Guru Staff Member Lightroom Guru

    Joined:
    Jun 20, 2009
    Messages:
    11,265
    Location:
    Bellaire, TX USA
    Lightroom Experience:
    Power User
    Primary OS:
    Mac 10.11.x El Capitan
    Lightroom Version:
    6.6 / CC 2015.6
    Lightroom License:
    CC Subscription
    Hanging is not the same as corrupted image files, Which is your problem? I have been giving you remedies for images that get corrupted between the time the camera writes to the camera card and viewing the file that was transferred to the HD with LR.
     
  25. Roomservice

    Roomservice New Member

    Joined:
    Jul 17, 2011
    Messages:
    11
    Lightroom Experience:
    Intermediate
    Primary OS:
    Windows 7 64-bit
    Yes today I reinstalled the driver from the CD. I've also checked canons site but there are no upgrades regarding that driver.
    About the cables: I don't think any of the two cables are bad since they work fine when LR is not involved.

    I checked the raw files with two methods: added them to LR to be able to see the raw files (and not just the embedded jpg's as you pointed out). And binary compare with UltraCompare, which verified the previous result.

    Next thing to do is to test a card reader. And also test importing from hard disk to LR.

    Maybe there is no bug in LR. But if there is, it probably makes the software do something quite unexpected (as bugs often do).
     
    Last edited: Jul 18, 2011

Share This Page