When I'm logged in to the VPS, I can do
with this command, the server 1.2.3.4 authenticates me through my
public key, and the VPS acts as the SSH agent proxy. So, if that server has my public key in .ssh/authorized_keys, I'm easily in, and no security breach on the VPS would affect my security.
Unless the attacker is on the jumpbox as root as then they can also forward in the same way, but this should not happen ofcourse ;)
For this reason, I tend to use a key per device and keep all authorized key files in svn, which makes it easy to identify which nodes are possibly compromised or at least quickly remove access. You could use the forward trick and keep the private key etc on your local device.
Remote syslogging is for this reason a good idea. Jumphosts in general should also solely run an sshd and nothing else. But that is what you are aiming for, any access is then easily noticed.
On the latest DENOG meeting, there was an interesting report that offline RAM chips still hold traces of your data for few hours, so be careful with that too :)
Cold boot attacks are quite old by now ;) there is a reason there are TPMs in quite some hardware, if possible use those or other dedicated crypto storage.
As an alternative, there was somewhere a project that modifies rsync in a way that it can work with encrypted data on
the remote site.
Check duplicity for this purpose.
To start with defining who you think your adversaries are, that is the most important step in something like this.
Greet,
Jeroen