If you got this error, congratulations, you have the solution here. This is quite complicated problem as below
nvidia-container-cli: mount error: failed to add device rules: unable to generate new device filter program from existing programs: unable to create new device filters program: load program: invalid argument: last insn is not an exit or jmp processed 0 insns (limit 1000000)Turns out the solution is just run this either in your metadata startup script or inside the Google Container Optimized OS VM.
sysctl -w net.core.bpf_jit_harden=1 If you want more
bash -c "echo net.core.bpf_jit_harden=1 > /etc/sysctl.d/91-nvidia-docker.conf"
sysctl --system
systemctl restart docker