Can't get gdb to attach between two Ubuntu VMs in Fusion 13.0.0 / Apple Silicon

[stevepate]

I have two VMs:

VM-1 - Ubuntu server 22.10 - Linux kernel with kgdb compiled in - 192.168.156.131

VM-2 - Ubuntu server 22.10 - where I am running gdb from - 192.168.156.132

For VM-1, I have the following in the .vmx file

    debugStub.listen.guest64 = "TRUE"

and when I boot the VM I can see the following in vmware.log:

    vmx Debug stub: VMware Fusion is listening for debug connection on port 8864.

    vmx Debug stub:     (gdb) target remote localhost:8864

Good so far. But when I try to connect from VM-2 with either of the following:

    (gdb) target remote localhost:8864

    (gdb) target remote 192.168.156.131:8864

I get "Connection timed out".

I don't see any message in vmware.log for VM-1

[Technogeezer]

You might want to check what "localhost" means in this context.

To the gdb setup, "localhost" means the localhost IP address (127.0.0.1) on your Mac. From what I read, the GDB debugger stub is listening on IP address 127.0.0.1 (localhost) on your Mac host at port 8864.

To your VM-2, using localhost as the hostname to connect to will resolve to 127.0.0.1 and the request will be directed back to the VM itself. Using the IP address of f VM -1 192.168.156.101 will try to route the gdb remote request to that IP. In both cases, that's not where the debugger stub is running - it's running in the host.

I'm not sure if the debugger stub can be reached from the guest by using the hostIP of the Mac and port 8864. 

Update: I found something in https://ddeville.me/2015/08/using-the-vmware-fusion-gdb-stub-for-kernel-debugging-with-lldb/ that implies that there is a vmx configuration option that allows the stub to be accessible on the VM's IP address

# If you are debugging a 32-bit machine use `guest32`
debugStub.listen.guest64.remote = "TRUE"

Not sure if this works on Apple Silicon but it might be worth a try...

[stevepate]

Thanks for the suggestions.

On my Mac, I ran the following:

    $ sudo lsof -i -P | grep LISTEN | grep :8864

    vmware-vm 1031    root   58u  IPv6 0xa4c2462e5f132953      0t0    TCP localhost:8864 (LISTEN)

    vmware-vm 1031    root   62u  IPv4 0xa4c2464184589c03      0t0    TCP localhost:8864 (LISTEN)

so indeed, Fusion is listening on localhost:8864 on my Mac. Both of these entries correspond to VM-1 which has the debugStub enabled. To confirm, if I shut this VM down, lsof doesn't report anything.

I've tried various combinations of:

    debugStub.listen.guest64 = "TRUE"

    debugStub.listen.guest64.remote = "TRUE"

with various combinations of "target remote ..." and still nothing.

My Mac IP address is 192.168.0.3 so I even tried "target remote 192.168.0.3:8864" but that doesn't work either.

There is no gdb version for Apple Silicon so running from another VM is my only choice.

Having said that, I noticed that lldb is there so I gave that a go. It gets "so far" as you can see below. I don't have symbols on my Mac for this Linux kernel (yet) and I don't know lldb so will have to play. But hitting 'c' to continue and then 'q' to quit lldb results in the VM being shutdown. Not very useful!

I'd really like to find a solution for gdb from the other Linux VM.

    $ lldb

    (lldb) gdb-remote 8864

    Process 1 stopped

    * thread #1, stop reason = signal SIGTRAP

        frame #0: 0xffffffe009248b9c

    error: memory read failed for 0xffffffe009248a00

    Target 0: (No executable module.) stopped.

    (lldb)  

    error: Process 1 is currently being debugged, kill the process before connecting.

    (lldb) bt

    * thread #1, stop reason = signal SIGTRAP

      * frame #0: 0xffffffe009248b9c

        frame #1: 0xffffffe00925a12c

        frame #2: 0xffffffe00812e8ec

        frame #3: 0xffffffe00812ebd4

        frame #4: 0xffffffe0092493a8

        frame #5: 0xffffffe009aa1208

        frame #6: 0xffffffe009aa1c8c

        frame #7: 0xffffffe009aa03e0

[stevepate]

One more piece of information ... if I try to install vmtools for this VM (Ubuntu 22.10) Fusion tells me that "VMware Tools is not available for this guest". Their website says that they support "Ubuntu Linux 8.04 through 12.04"

If I'm using:

    debugStub.listen.guest64.remote = "TRUE"

then I'm assuming that the gdb stub should be running inside the guest somehow. Otherwise how do they intercept any requests to this IP:port? Perhaps vmtools is a must have? 

I have a lot invested in 22.10 (I'm working on a Linux filesystem book and have a kernel-based filesystem for 5.19.7) - perhaps I'm paying the price for not working on older, stable versions 😞

[Technogeezer]

VMware Tools do not exist for ARM Linux operating systems. They exist only for x86 Linux distributions and maintained only for really old Linux distros that haven't picked up open-vm-tools. Most of the ARM architecture distros have open-vm-tools. I don't know if there's any hooks in open-vm-tools for the gdb stub.

Ubuntu 22.10 isn't that bad a release if you got it installed on Apple Silicon (there are hoops you have to jump through because of Ubuntu kernel bugs in the released installers).

Unfortunately "paying the price for not working on older, stable versions" is unavoidable when running on arm64. Those "older stable versions" that work fine on x64 don't work well  on arm64, if at all. That's because of things like a port of both open-vm-tools and VMware SVGA (vmwgfx) drivers to arm64, (both provided to the open source community by VMware) and outright bugs in arm64 kernels prior to 5.19.

As an example. Ubuntu 18.04 simply won't run on Fusion 13. Its device discovery doesn't understand the ARM SystemReady virtual platform provided by Fusion. Ubuntu 20.04, 22.04 and 22.10 work better (each release got better than its predecessors). They may take a bit to install at first, but the kernel updates they install are pretty solid and current. I anticipate that 23.04 will install out of the box without a lot of hassle (given what I've seen in the daily builds).

Red Hat releases aren't much better. RHEL 8/CentOS 8/Rocky 8/Oracle 8 for arm64 won't even run on Apple Silicon. That's fixed in RHEL 9/etc. and later. 

[stevepate]

Well that's a depressing read 😞 but thanks again.

I actually got much further with UTM. Installing Ubuntu 22.10 was very easy (but a somewhat different experience to Fusion). Connecting a gdb client VM was easy. I can read source line by line etc etc. But ... no breakpoints. Sigh!

I've been having issues with gdb/kgdb for close to 20 years. My old colleague at VERITAS worked on getting patches into the Linux kernel many years ago. Obviously there aren't enough of us using this technology for it to have high enough priority. 

I really don't want to have to buy another laptop. I'll keep exploring ...

[stevepate]

I've gone back to UTM for now and using gdb fine sans breakpoints. But that does tell me that it's not an issue with VMtools as open-vm-tools is not running in my VMs.

I would love to get this fixed in Fusion. It would be great if someone knows how to do this.

[stevepate]

Ah! My UTM VMS do have open-vm-tools running inside 

[stevepate]

Obviously not a lot of people are running gdb/kgdb on the Linux kernel 😞 

I solved my problem in UTM by running 2 VMs using x86_64 emulation. It's slow (kernel compilation takes 17 hours compared to 1 hour native) but it does work.

It would be nice if this all worked on Fusion.

[ColoradoMarmot]

Given the lack of resources to get ARM native code running properly on current versions, I wouldn't hold out hope.  We've been waiting months for updates since the release of 13 (which really was a second tech preview given all the limitations).