Thank you for pointing me in the right direction. Issue solved.
Using ESXi Shell in ESXi 5.x and 6.x (2004746)
About the ESXi 5.x and 6.x firewall (2005284)
esxcli network firewall ruleset rule list -r "vSphereClient" esxcli network firewall ruleset allowedip remove -r "vSphereClient" -i "123.123.123.123" esxcli network firewall ruleset allowedip add -r "vSphereClient" -i "123.123.123.124"
Extra info: Altaro - Managing the ESXi firewall