This is a very subjective question and is very dependent upon what you are trying to accomplish.
In addition you are asking some rather low level questions which leads me to believe you have not worked in this arena before, which is ok we all start at zero and level up from there.
Given those two observations, I would suggest you start small and work your way up.
[1] Start first with an iSCSI initiator(client) and an iSCSI target(server), both on different hosts. They can be a straight through cable or can go through a switch, at this point it does not matter. Play aorund with that for a while. Add more targets, create a mdadm raid out of your iSCSI targets on your clinet(don't worry if all of your iSCSI targets come off of one spindle for now). Then start playing around with LVM on your client. Create multiple PVs, add them to a VG and then create an LV. Expand your LV. Create a snapshot mirror of your LV.
Really dig into LVM, it will be the key to everything else you do.
Next add a second iSCSI target server. I would also suggest at this point getting a decent switch that supports LACP and some management. The Procurve 1800 series switches are a good bottom end, so are the Cisco SG-300's. Switch management and features will become more important later, but investing in them now is a good idea. At this point with two iSCSI target servers you will want to lather rinse repeat what you did when you had one server. If you really want to have some more fun, add a second initiator, and have it mount the same iSCSI targets. What happens when you try to have two systems write to the same EXT3 volume? Convert your volume to something like GFS (there's more involved but that is the objective at this point). Now what happens when two hosts write to the same volume?
Now let's add two nic's to all of our clients. Now you will need to learn about network bonding[2]. What are the different modes, how are they different. Get some bandwidth measurement tools running on each end of your connection, flood the link with the various bonding modes. What happens when you mix the modes so that each end is mismatched? What happens when you use a mode compatible with LACP and your switch is configured to do LACP[3]? Lather rinse repeat what you have done with regards to storage above.
Remove your interfaces from the bonded interface. Assign each interface on the target server a unique IP. Make sure your target server will share it's iSCSI targets through each interface. Now mount your iSCSI targets using the two different IP addresses. What happens when you write to each one? Now configure multipath [4], and play with that for a while. Down one of your target server's NIC's (remove the cable ifdown the device etc) what do you see in the logs? How does performance improve/decrease?
Now add a second network switch. You can either double your network interfaces or you can split them across the switches. What kind of bonding modes should you be using? Should you have a link between the two switches? Do you need to have STP enabled?
Now you're near the end of your learning and you have approached the level of master apprentice. You don't know it all, but you have a much better foundation of understanding than most people. From here you will have a better idea of how to architect a storage infrastructure. The technology will change if you use fiber channel or ATAoE, but the main concepts will be the same.
Useful web resources:
[1] http://www.cyberciti.biz/tips/rhel-centos-fedora-linux-iscsi-howto.html
[2] http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding
[3] http://en.wikipedia.org/wiki/Link_aggregation
[4] http://sources.redhat.com/lvm2/wiki/MultipathUsageGuide
I'm going to open this up for everyone to be able to edit.
