I personally have not used them, I have not had this use case for a vast number of years.
but it seems to be a relatively stable solution. Serial over Ethernet - Serial port over Network - Serial to Ethernet Connector
if you need the machine that is hosting the modem to be able to move across the cluster you will have to implement this or a similar solution. you can however, if the client is happy for the lost of DRS and vMotion capability for the particular machine just host the modem in the local hosts comport and directly map it.
i think this will depend on what interface your modem having and the VM should have supported drivers as well.