Right, I think I've got It.
I like the look of Ken's diagram (cheers).
2x 1Gb NIC's for COS/vMotion/HA - 90% no bandwidth as they should only be used if theres a problem.
2x 1Gb for the VLAN - probably handling small request 60-70% of the time but I do forsee occasions when the full 2Gb bandwidth will be used.
1x 1Gb to the SAN.
Mmm, having wrote that I now wonder if 2x NICs to the VLAN are needed. As the data/apps/files are on the SAN we have a 1Gb bootleneck. The link to the SAN needs to be the same bandwidth the link between the VM's vSwitch and the pSwitch.
Now the reason for wanting 2Gb from the pSwitch to the VM's is that I'm sure one of the biggest problems of our current file server is bandwidth, I'm not sure it's 1gb NIC is enough.
Basically at the start of every lesson a good 300 students log on to the network. Plus on top of that 150 staff. This is an average. We can peek at around 6/700 users on the network at once.
Now typical usage from the file server may include - 2/300 students opening/saving their powerpoint presentations containing 20+ pages of pointless bloated graphics and pictures. A couple of teacher playing back a video they downloaded for a lesson. Maybe some teachers preparing for a lesson downloading video into their home drive. Not to forget 10-20 art students opening RAW 6-10MP images. Plus another 50-100 users maybe opening/closing smaller more sensible files.
Now this would typically occur at the start/end of every lesson. The joys of being a large school.
So we need the bandwidth to coupe with the peeks without throttling the processors so the VMs themselves don't start grinding to a halt.
(anyone reading this, beware - I'm thinking and typing at the same time...)
But we also need to factor in the fact that we are load balancing across two host servers, so splitting file services down into smaller VM's and speeding the load across the two servers means that less bandwidth is needed on each individual server...
So we are back to 6 NICs (only a different configuration). 2 for the management - largley unused, 2 for the VM's and 2 for the SAN. But at peek load - which could happen a couple of times a day - we'd have 2Gbps coming from the SAN and out across the VM's.
I'll stop here because I'm running scenarios through my head that seem if more outlandish. Cut to the chase, the SAN will have at most two 1Gb ethernet channels, which will be shared by two hosts, so more than 1Gb from host to SAN is wasted or redundant. So only 1Gb is needed from vSwitch to pSwitch has it'd never get the bandwidth from the SAN. A second NIC from vSwitch to pSwitch is only needed for redundancy or load balancing (incoming requests on one, outgoing on the other?).
But will 1Gb be man enough if we are moving 7 over used servers into 2 physical servers?
Man I thought finding the right SAN was a challange....
