iSCSI (Internet Small Computer Systems Interface) is a protocol that allows SCSI commands to be transmitted over a TCP/IP network. It enables the creation of SANs (Storage Area Networks) by allowing servers (called initiators) to access storage devices (called targets) shared over the network as if they were locally attached.
Among various storage solutions, iSCSI plays a critical role in highly available virtualization environments. When a hypervisor host fails and its running VMs go down, another host (or node) in the cluster can quickly take over and restart those VMs. This is possible because the VM volumes are stored in centralized iSCSI storage, ensuring seamless access across all cluster nodes.
Important Concepts and Nomenclatures
- TPGs = Target Portal Groups
- A group of one or more network portals associated with an iSCSI target.
- Portal = Network Endpoint
- Composed of a binding IP and port. E.g. 192.168.1.100:3260
- LUNs = Logical Units
- A unique storage resource identifier for a partition or volume. E.g. lun0, lun1, etc.
- IQN = iSCSI Qualified Name
- Used by both initiators and targets. Can be freely chosen as long as it follows the format: iqn.yyyy-mm.reverse_domain_name:unique_id
TARGET / SERVER
targetcli is the tool used to manage shared volumes. Changes take effect immediately with no need to reload or run additional commands.
When exiting, it writes the configuration to disk in JSON format for persistence, backup, or source control. The location may vary by distribution, but it is typically found at /etc/target/saveconfig.json or /etc/rtslib-fb-target/saveconfig.json. At runtime, configuration is also held under /sys/kernel/config/target/.
Installation on Ubuntu
sudo apt install targetcli-fb -y sudo systemctl enable target --now sudo mkdir -p /iscsi_disks sudo targetcli
Walkthrough
/> ls /> cd backstores/fileio /> create disk01 /iscsi_disks/disk01.img 1G
/> cd /iscsi /> set discovery_auth userid=username /> set discovery_auth password=secret /> set discovery_auth enable=1
/> create /> delete iqn.2003-01.org.linux-iscsi.u24.x8664:sn.0dcd420db0df /> create iqn.2003-01.lan.srv:disk01 /> cd /iscsi/iqn.2003-01.lan.srv:disk01/tpg1
/> portals/ create /> portals/ delete 0.0.0.0 3260 /> portals/ create 192.168.10.10 3260 /> luns/ create /backstores/fileio/disk01
/> get attribute authentication /> set attribute authentication=1
/> acls/ create iqn.2003-01.lan.client /> cd acls/iqn.2003-01.lan.client/ /> set auth userid=username /> set auth password=secret /> set auth mutual_userid=username /> set auth mutual_password=secret
INITIATOR / CLIENT
On Linux hosts that use volumes or disks, iscsiadm is the tool used to discover and connect to storage resources.
Installation and Configuration on Ubuntu
sudo apt install open-iscsi -y sudo iscsiadm -m discovery -t sendtargets -p 192.168.10.10 sudo nano /etc/iscsi/initiatorname.iscsi
Edit or paste the IQN.
InitiatorName=iqn.2003-01.lan.client
Rebooting after changing the IQN is optional but recommended.
#sudo reboot sudo nano /etc/iscsi/iscsid.conf
Uncomment the relevant lines based on your authentication configuration on the target side. Remember to replace the values.
# ************* # CHAP Settings # ************* ## For Discovery Auth discovery.sendtargets.auth.authmethod = CHAP discovery.sendtargets.auth.username = username discovery.sendtargets.auth.password = secret ## For 1-way Auth node.session.auth.authmethod = CHAP node.session.auth.username = username node.session.auth.password = secret ## For Mutual Auth discovery.sendtargets.auth.username_in = username discovery.sendtargets.auth.password_in = secret
Test it out
sudo iscsiadm -m node --targetname iqn.2003-01.lan.srv:disk01 --portal 192.168.10.10:3260 --login sudo iscsiadm -m session sudo iscsiadm -m node -T iqn.2003-01.lan.srv:disk01 -p 192.168.10.10:3260 --logout sudo iscsiadm -m node -o delete -T iqn.2003-01.lan.srv:disk01 -p 192.168.10.10:3260
BONUS
As an alternative to iSCSI, Network Block Device (NBD) offers a simpler and more versatile solution for accomplishing essentially the same goals [Link]. It is a routable TCP-based protocol that runs on port 10809 by default.
When would you choose NBD over iSCSI or other solutions?
- NBD has a simpler protocol than iSCSI, making it easier to use in software development. See
libnbd[Link]. - It is fully integrated into QEMU (client and server), with native TLS encryption [Link].
- NFS does not allow the client to define the filesystem because it is not a block device.
- Fibre Channel over Ethernet is not routable, and ATA over Ethernet has not been actively developed for a long time.
- In summary, the right choice depends on your requirements.
sudo apt install nbd-client -y sudo nbd-client 10.0.0.10 -N data_drive_1 /dev/nbd0
On the server side, use nbdkit, a multithreaded NBD server implementation [Link].
nbdkit file file=disk0.img nbdkit file -N data_drive_1 file=disk1.img nbdkit file -N data_drive_2 file=/dev/sdb nbdkit qemu disk=disk.qcow2 nbdkit memory size=1G -i 10.0.0.10
Or to export multiple block devices at once:
nbdkit --exportname data_drive_1 file disk1.img --exportname data_drive_2 file disk2.img
Ideally, run it as a service managed by systemd.
/etc/systemd/system/nbdkit-disk0.service
[Unit] Description=NBD export disk0 [Service] ExecStart=/usr/bin/nbdkit file file=/srv/nbd/disk0.img User=nbdkit Group=nbdkit [Install] WantedBy=multi-user.target
sudo systemctl daemon-reload sudo systemctl enable --now nbdkit-data1.service
Make sure the nbdkit user exists and has read/write permission on the image file /srv/nbd/disk0.img.