You will have to create a separate racoon configuration file with your settings and add an include directive in Apple's default /etc/racoon/nf file, to make sure the racoon client reads your modified configuration and executes it. The Apple Mac's Network Preferences GUI does not provide sufficient settings to allow you to configure a connection with the Netvanta.Even if you modify its nf file by setting 'mode_cfg off ', this client setting appears to have been hard coded by Apple and will still ask for MODE_CONFIG information from the router. The Apple Mac client asks Netvanta for MODE_CONFIG data.Even if Phase 1 completes, IPSec Phase 2 always fails. Attempting to connect without XAUTH is a hit and miss affair for IKE Phase 1.

The native Apple Mac 'Cisco IPSec' VPN client requires XAUTH.Other recent OSX versions should work too. I tried this on an Apple MacBook Pro running OSX El Capitan v.10.11.6.

Apple are using their own fork of racoon to manage IKE negotiation, but it will not work out of the box by just using the Network Preferences VPN GUI, without creating a separate configuration file for it.

After various attempts I managed to get Apple Mac's native VPN client to connect to a Netvanta 3120.

Mac ipsec vpn client free