Can't you just run the program inside a VM of some sort and encrypt the whole image that way? I guess during runtime the contents in RAM won't be encrypted, but when the image gets saved back to disk, it should be.
e.g. in Parallels, they have built-in AES-based password VM encryption: https://kb.parallels.com/8832
Or in VirtualBox: https://docs.oracle.com/en/virtualization/virtualbox/6.0/adm...
You would have to be able to overwrite a syscall in userspace (which would be a massive security hole) or have some kinda kernel driver (but you said disk encryption doesn't work for some odd reason, otherwise that's your solution right there).
I guess, as you stated, you might be able to write a fuse application to do it, but I don't see the benefit over just an encrypted disk.
FHE is interesting: https://en.m.wikipedia.org/wiki/Homomorphic_encryption
Maybe something like ext4 transparent encryption?
I assume that in this scenario we don’t want to change the behaviour (program code, source code) of the application itself.
Why is it important that the syscalls (write et al) encrypts? (Even if they do so only under the hood.) Is it not sufficient that the application writes to a place that is encrypted?
It’s probably not impossible to modify the syscalls, but I think it would be vastly easier to let the outside system handle the encryption.
“I was only able to find FUSE filesystems that can transparently encrypt files on top of a file system”
I believe the reason that you mostly found such solutions is that they are much more “cost-effective” than implementing a system that operates on the syscall level.
If you need to ensure that the application writes only to the specific mountpoint you can probably use some kind of “jail” to achieve that. Here’s a decent starting point on that: https://blog.mnus.de/2020/05/sandboxing-soldatserver-with-bu... (assuming that the application runs on Linux)