Problem is not caused by VoLTE setting in the phone, (if not enabled in the Operator's network, there is no APN for IMS, which is required for VoLTE).
The problem you are experiencing is with the Operator's configuration of CSFB (Circuit Switched Fallback), which does the RRC Release from LTE, with Redirection to 3G for the CS (Circuit Switched) voice call to be established.
There are many parameters in the Operators network that need to be optimised for CSFB to work as quickly as a 3G CS voice call.
As an example today in Movistar España, CSFB added an additional 0.75 second to call setup time, with a total call setup time of 4.8 Secs, from Extended Service Request message (LTE layer 3 message) for call establishment to Alerting (UMTS NAS message).
These times were captured using full subscription version of Network Signal Guru, on MiMix2, with 8.7.26 firmware, VoLTE enabled but not functioning (Movistar España has not opened VoLTE service yet).