on photo storage: wherein the author yells at (i)cloud
Backing up photo assets independently and redundantly from iCloud and the iPhone camera roll is surprisingly annoying as I ranted about previously1. At some point recently, accessing your photos from icloud.com was updated. Previously, the only way to download a copy gave you a compressed JPEG file, with something like 60% compression, and a 6-8X downsize in resolution (I found this applies to raw files too, a 48MP .dng gets converted to a 6MP JPEG with ~60% compression). Hilariously bad.
But it's now possible to download the original file directly. Its hidden under a menu as 'other download options' which gives a pop-up to download 'original files with edits'. This should be the default/only option but what do I know. There's a selection limit of 1,000 items at a time (because of course there's another hurdle). To solve that I made an iOS automation where you can 'program' some iOS tasks (just call it a loop not a 'repeat'). I 'programmed' a thing to go through my entire library, create an indexed list of assets, then dump every 950 items into a new album. Now using icloud.com, I can ctrl-a within each album and get a zip file of all the original files. Hallelujah!
All to say, I can now copy the actual image files to a new device without any loss of metadata or quality. The 'photosync' app I was using for samba is a bit unreliable at running on a regular schedule, it also didnt play nice with tailscale or vpn's and a few other things. My current backup method is to have;
- images captured on the phone, active files
- images on icloud, active copy
- most recent zip from icloud, stored on SSD, cold backup 1
- copy of all zips, stored on HDD, cold backup 2.
- copy of all zips, stored on external HDD, kept in firesafe, cold backup 3.
Step 2 is in case my phone is destroyed, lost, or stolen. I treat it as just a point of access for the originals, and with zero confidence it will be available there long term. This method pretty well follows the 3-2-1 rule, though its technically 4-4-1 but whatever, more is more.
The last thing to complain about is 'Shared Albums'. These baffle me and my wife. You can create a shared album and invite other iOS users to upload photos for you all to share access, kinda. Its weird in that the storage space it takes up appears to be free. It doesnt count towards the uploader's data quota (which is paid) nor anyone that accesses it. So where is it? Why isn't the user charged for this? How far could I exploit this to get free cloud storage? The other weird thing is that everything placed in a shared album is automatically compressed. Not just when downloading a copy (from anywhere I tried), but also when viewing on any device. Its compressed in the same way I mentioned before, dng's and heic's are converted to a JPEG with up to 8x resolution reduction, and ~60% quality. So many problems with this. I now have no way to access the original full res files from the other users! I would have to setup a separate method to transfer their files to me without going through icloud. Even if the original uploader downloads back their own file, it is compressed. That original data is gone.2
So why would apple do this? By compressing, apple saves on storage and trades it for compute time to save cost. But why bother? To add to a shared album it has to be in icloud anyway, so now theres the original file and a compressed copy in each user's storage, and another compressed copy in the shared album on unpaid storage. So the data is stored on apple's side up to 1 + 0.6 x 2 = 2.2 times. This makes no sense from a cost perspective. It would make waaaaay more sense to make a shared album act as a collection of direct links to the original assets in each user's icloud on the back end. Maybe they don't do this for sEcUrItY, but surely this can be solved with public-private keys, basic file permissions, or some other accepted secure file access. It seems that google drive has this figured out just fine, and in their case it only counts towards a user's storage if you are the owner of the shared folder, having read access doesn't count to your quota. That actually makes sense. You're sharing links to get access to the other files and that's it. I would even be willing to have the entire shared folder count against my own quota, and with each shared user also getting dinged for the storage cost (provided each person actually gets a redundant copy not the same one).
So basically, if you have an apple shared album, the original files uploaded by other users and yourself are irreversibly downsized and compressed, and that data is gone forever. Downloading a copy of your own original assets can be done without loss of data, but requires jumping through some hoops. All of this is maybe nitpicky, but it's the principle that I have no choice in how my data is stored and accessed. I probably woulnd't care too much if a stereotypical iphone photo of 'person standing in front of thing' got compressed down to a JPEG of 2000 pixels on the long edge. But if I'm taking a 48MP raw image for artistic reasons, I want all the control of that data. Is that so much to ask for?
Apple 'live' photos that were split during exporting can in fact be recombined. Apparently there is a non-standard apple specific metadata field that links the two assets. If you forcefully reinsert that field, create a new unique id for both the video and the image, upon re-importing to the camera roll they will show as a single 'live' photo together like nothing ever happened. I made some reallllly jank code using swift *shudder* to interact with the macOS photos app (which i got access from quickEMU, also a pain to get working because sEcUrItY). The copy of my library from immich was sorted into folders by month so the pairing logic wasn't too bad. Some false positives came up where the filenames werent similar enough and the date/time was very similar. Or if all the metadata was stripped previously. It's hard to know how many were incorrectly matched without going through and playing all of them. But it seems like 90+% are correct. Although there is a caveat (of course there is why would it be that easy). There will now be 3 items in the camera roll for each split 'live' photo. The split photo, the split video, and the new livephoto pair that was just imported. Duplicate detection will only flag the split photo with the livephoto as a duplicate, the ~3 second video will live on. There's definitely a way to smartly filter out all these short videos by cross referencing the fixed livephoto metadata like date/name/filesize/EXIF. But ehhhh, thats alot of work and theres a risk of falsely flagging and deleting a truly unique short video.↩
This handling of raw files is really unacceptable. Despite what many photographers will tell you, there is no way to view a raw file correctly/accurately/truly/purely as it came from the sensor. A raw file is a bucket of 1's and 0's with a signature that says 'Im this camera sensor and I stored this binary data using this ISO standard.' It is raw, it is uncooked and unprocessed. To view the 'pure image' or whatever you can only read the 1's and 0's. To view the latent image, your software has to interpret the binary data into colors, how it does that is up to the conversion/interpretation method chosen by the software. The user can then adjust that interpretation with edits. After that, the raw can be exported as a fixed format like a JPEG, which is a bucket containing RGB values instead of binary. (Aside to an aside, even viewing the jpeg is not universally accurate, you have to view it through a monitor or physical print which probably doesnt have the some color range as the device you edited on. Add another layer and ask if your red is the same as my red, or if the color range of your eyes is the same as mine. By icloud converting the raw data to an interpreted image AND compressing it, they remove all control of the image at every step, and irreversibly remove the original information.)↩