Caffe can create, read and write images from an LMDB database. This is much faster than using the files from filesystem during training. However, I noticed that by default Caffe would not set write permissions for my group for the lmdb directory.
A quick investigation into src/caffe/util/db_lmdb.cpp
showed that this was because new lmdb directory was being created with
0744 permissions. I changed this to 0774 in
the code and recompiled Caffe.
Now comes the mystery: Caffe still created lmdb directories where my group did not have write permission!
Further investigation showed that the default umask in Linux is
0022. This umask does not allow the group write permissions
to be set.
Ok, so then I set the umask at the shell to 0002 and
tried to create the lmdb again. Found that lmdb directories still did
not have group write permissions!
I was creating the lmdb databases from a script that ran the
convert_imageset binary. Now the binary was run by using
the exec command of the Bash shell. This replaces the
currently running process with the convert_imageset binary.
However, unlike a fork, the new process is not passed the umask of the
parent process by the OS! This was the reason for the problem.
Once I knew this, the solution was easy. I modified the
tools/convert_imageset.cpp code and added this code:
#include <sys/types.h>
#include <sys/stat.h>
// In the main function ...
umask(0002);
// ....This solved the problem perfectly! :)