This is an important blog post which I’ve meant to write for a while – a story of woe and near disaster. I hope others will see it and avoid such a situation. Please feel free to pass on or link on forums etc.
In brief – it’s about RAW files being downloaded via an old copy of Nikon Transfer (part of Nikon View NX) Corrupting RAW files from the D800!
To explain the setup/scenario which caused this for me:
On large shoots away from home (Weddings, Festivals, Collaborative shoots) I tend to shoot a lot of pictures. During most of these I shoot RAW files and as such even with a large 32Gb card it can fill up. Although I have smaller other cards sometimes these aren’t enough and so I copy files from the card (using a card reader) onto my netbook. Instead of just copying them – I’ve been using Nikon Transfer which is part of View NX. Hadn’t been doing this for any special reasons although there are advantages e.g. it can embed additional Metadata. Without realising it – I was running an old version of Transfer/View NX – not sure now but I think it was version 1.85 while the recent version was approx 2.5.
On downloading the images to the netbook local hard drive (using the Transfer application) the previews were appearing in ViewNX just fine, and I previewed one or two and they looked Ok. So after downloading the whole card I formatted it and used it again (after using some other cards).
Discovering the Issue
On returning home I copied the files over from the netbook to my desktop using the home network just using standard file copy.
Opened up Bridge to view them – no go! Bridge couldn’t see them! It said they were not in a correct format.
Went back to the netbook and viewed them in ViewNX (no bridge) and they seemed fine.
Tried to copy again – no change.
Then opened up ViewNX on the desktop – and I could see them there BUT on opening them fully – ViewNX reported them as only being 1632 x 1080! From a D800 they should be 7360 x 4912.
I couldn’t understand why ViewNX could see the images (but report them as being small) while Bridge and other applications couldn’t see anything. I did some online searching and found other people who had run into the same issue – it pointed at the fact that an old version of Transfer had copied over the RAW happy as Lary but significantly corrupted them in the process! Some people said that the files were a total loss (NOT an option for me as this was a paid one-off event!) while others did point at various possible solutions.
(Jump to Final Solution below if not interested)
Metadata swap : Partially because of comments I’d seen online and otherwise a belief that Transfer only changed the metadata of files – I attempted a fix where I downloaded a command line program which could copy Metadata from a healthy RAW file, dump it to a local file, and then paste this into one of the corrupt files. No Go 😦
In connection to this I tried checking to see what kind of damage the transfer application did – I had RAW files on cards so copied one just using file copy, and again using the erroneous Transfer application – then compared the files using a file comparison application I use sometimes with coding. Alas they were drastically different – it wasn’t just the header of the file which had changed! I find it quite shocking that Transfer would do this.
Flash card recovery : Even though I knew there was no chance of getting ALL corrupt files back since I had formatted the card and started to re-use it again – there was a chance that I could recover some of the corrupt RAW files. So I installed a current copy of the RescuePro application that comes with Sandisk Extreme cards and gave it a go. And thankfully – it managed to recover quite a lot!
The Card holds about a thousand RAW files.
I had formatted and re-used the card to about half way – and so I was able to recover about half the RAW files from the formatted but un-used card – woohoo!
I discovered two things while trying to install the Rescue Pro software:
- The code that comes with the card is not in fact the license key – it’s just a code that you have to enter on a website. They then send you the true license key shortly after. Yes I wasted quite some time trying to enter this as the license key!
- Once validated the licence key is only valid for 12 months. Therefore a previous key I had no longer worked. As such – don’t install this application as soon as you get a card – keep the Serial ID safe and only use to get a license key once you really need to us it.
Copy small (1632 x 1080) images : This wasn’t really an option which I wanted to consider. although technically “HD” they wouldn’t be remotely close to the full res. However I was having dread thoughs of having to take a screen-grab of ViewNX preview to use in Photoshop!
Contact Nikon : I contacted Nikon Support with the hope that they had some special program or technique where they could recover all the original RAW data from the corrupt file – a sort of reverse engineering the corruption, but alas it was not so. Their reply confirmed partially what I had already found online:
“I am really sorry to learn about your problem with D800 files and old View NX.
Older Nikon software released before the D800 does not have correct algorithms to recognise the new RAWfile format. Upon transferring it saves some information in an incorrect place, effectively damaging integrity of the file. The issue does not exist with the version distributed together with the camera or newer.
There is a way to recover full-resolution JPEG previews from the damaged files – they are quite highly compressed so you can see some artifacts (especially in detail-less midtones and shadows), but the quality should be sufficient for most purposes. Please note this is not a official Nikon procedure, however it worked successfully for few other customers that had the same problem. Also, as the IJFR application is not our product, we cannot take any responsibility should it cause any problems.
1. Please download and install the freeware IJFR utility: http://rawworkflow.squarespace.com/instant-jpeg-from-raw-utility/
2. Create a copy of all your affected images in a single folder (just in case)
3. Select all files, right-click and choose “Instant JPEG from RAW”
4. Select “Native Size only”, “Add suffix” and “JPEGs in folder(s) next to Raws” and click Extract
5. A separate folder will be created, containing the extracted previews.
Also, please uninstall Nikon Transfer (if it appears as a separate application) and your copy of View NX2. Then install the latest version from the link below:”
The link they gave was to Nikon View NX 2.5, but I have an updated one below. Also note above app link is currently dead but I list more below.
For the images.
So the solution was two-fold
- For the files that were still on the formatted memory card but not written over – RescuePro recoverd the original RAW files.
- For those I could not recover: So it transpires that in addition to a small thumbnail image (as is saved in some jpeg files) the RAW files contained a full resolution preview! Alas this was heavily compressed but thanks to the D800 36Mpx this was minimised for web images. While jpeg files for the D800 usually come out at 5-20Mb these files were 1.5-3Mb. Of course I would loose the editing advantage of RAW files – but a full res jpeg is better than nothing!
I found a number of applications which could do this extraction:
http://rawworkflow.squarespace.com/instant-jpeg-from-raw-utility/ (As suggested by Nikon – Site currently down)
For the Future:
Even future-proofing isn’t straight forward! It seems that if you have an old version (pre 2) of ViewNX and Transfer installed then just updating View NX will not fix the issue! It seems that it doesn’t always replace the old Nikon Transfer – just installs the new one alongside. So if you have shortcuts just pointing to Transfer (as I did) then you could still run into this issue! So my suggestion is to totally un-install Nikon View related software (+ Transfer) and then do an install of the newest version from scratch.
Currently that is : Nikon View NX 2.6.0
And of course – I will now.
But I am also considering not using Transfer – why would it need to change the RAW file at all other than maybe adding data to the metadata. really not too sure about that.
So that’s it – hopefully this article goes through the internet ether and is noticed by people with modern Nikon Cameras who may still be running the older versions of Nikon Transfer. Or failing that – those with corrupt RAW files who haven’t found the solution.