ISCSI Vmkernel ports in different VLAN

[prasadmenon]

Hi,

I have ESX 4.1 with below N/W configuration:

1) vSwitch0-->Service Console and VM Network ( 172.15.14.x)----------vmnic0

2) vSwitch1--> Service Console2 and ISCSI-vmk0 ( 172.15.15.x)---------vmnic4

3) vSwitch2--> Service Console3 and ISCSI-vmk1 ( 172.17.5.x)-----------vmnic5

I am able to access all 3 service consoles. Ping/ssh is working fine for all three.

Even I am able to ping vmk1.  But vmk0 is not accessible.

While configuring IP settings for vmk0, I gave 172.15.15.1 as default gateway and it accepted. At this point of time, vmk0 is accessible. Ping is working fine from remote machines and storage.

But while configuring IP settings for vmk1, when I gave 172.17.5.1 as default gateway, it accepted  and changed the default gatway IP for vmk0 also. Then vmk0 is not accessible and ping stops working from storage and all remote machines. But vmk1 is accessible.

If I again to try to change default gateway of vmk0 to 172.15.15.1, it again override for vmk1 also. And then vmk1 becomes accessible.

So at any point of time, it keeps only one gateway for vmkernel ports.

Is there anything I am missing to make this functional. Do I have enable some routing within esx box?

Once both vmkernel starts working, then I have to do multipathing. Above scenerio is some specific requirment, so cannot be changed.

[root@esxserver1 ~]# esxcfg-vswitch -l
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         128         3           128               1500    vmnic0

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VM Network            0        0           vmnic0
  Service Console       0        1           vmnic0

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch1         128         4           128               1500    vmnic4

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  Service Console 2     0        1           vmnic4
  ISCSI-1               0        1           vmnic4

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch2         128         4           128               1500    vmnic5

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  Service Console 3     0        1           vmnic5
  ISCSI-2               0        1           vmnic5

[root@esxserver1 ~]# esxcfg-route -l
VMkernel Routes:
Network          Netmask          Gateway          Interface
172.15.15.0      255.255.255.0    Local Subnet     vmk0
172.17.5.0       255.255.255.0    Local Subnet     vmk1
default          0.0.0.0          172.15.15.1      vmk0
[root@esxserver1 ~]#

[root@esxserver1 ~]# netstat -ran
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
172.15.15.0     0.0.0.0         255.255.255.0   U         0 0          0 vswif1
172.15.14.0     0.0.0.0         255.255.255.0   U         0 0          0 vswif0
172.17.5.0      0.0.0.0         255.255.255.0   U         0 0          0 vswif2
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 vswif2
0.0.0.0         172.15.14.1     0.0.0.0         UG        0 0          0 vswif0

Any response is highly appreciated.

Regards

[AndreTheGiant]

There can be only one default gateway for service console (and only one for vmkernel).

Router between more network must be handle by external router.

Andre

[prasadmenon]

Thanks for the quick response.

Yes, in my case, default gateway for all 3 service console is one i.e. 172.15.14.1 and it is working fine for all 3 SCs.

But for vmkernel also, it is only one for both vmkernel ports, but it works fine only for one subnet.

So, does my config on ESX is fine? If yes, do you suggest any specific thing to be done on external router? Sorry I am not very great in networking.

Regards

[AndreTheGiant]

Is correct... can only work for one subnet.

Consider that networks may be better if they are isolated (like FT net, vMotion net, iSCSI net, ...)

[prasadmenon]

Two vmkernel ports in seperate vlan(vmk0 in vlan1 and vmk1 in vlan2). Does esx support this? I hope I am not doing something which is not supported.

Can someone confirm this please?

Regards

[AndreTheGiant]

Each vmkernel can be tagged on a different VLAN.

This is supported and works well.

Andre

[prasadmenon]

Thanks Andre for the prompt responses. Actually I am trying to configure ISCSI multipathing using two different vmkernel ports in different VLAN. I have done port binding on both ports successfully, but ESX does not see two paths for one lun which should usually happen. When I connect both the vmkernel ports in same VLAN, then multipathing works fine and esx can see two paths for the same lun.

You have mentioned about vlan tagging on vmkernel ports. How do we do that?

BTW; looking at the below config, where should my storage should be i.e. VLAN1, VLAN2 or VLAN3. Sorry for too many questions.

1) vSwitch0-->Service Console and VM Network ( 172.15.14.x)----------vmnic0 ( VLAN1)  ----> All VMs are here.

2) vSwitch1--> Service Console2 and ISCSI-vmk0 ( 172.15.15.x)---------vmnic4 (VLAN2)

3) vSwitch2--> Service Console3 and ISCSI-vmk1 ( 172.17.5.x)-----------vmnic5 (VLAN3)

Thank you again.

[AndreTheGiant]

Portgroup VLAN tagging is simple a VLAN ID that you put in the portgroup propertiers (and also a single vmkernel internface is a portgroup).

But this number MUST match physical switches VLAN IDs.

Andre

[prasadmenon]

Nope. Even after putting actual VLAN ID in portgroup properties, it does not work. Any other clue?

Infact after putting VLAN ID, it does not even detect single path.

BTW; are these VLAN ID given to PG are persistent across reboot?

regards