Home / LDOM / How to Configure the I/O Domain on Oracle VM for SPARC ?

How to Configure the I/O Domain on Oracle VM for SPARC ?

If you allow any of the guest domain or control domain to access the any I/O (PCI) cards directly from the hardware ,usually that domain is called I/O domain or root domain.By default, two virtual I/O services that you need to configure on primary domain to provide the I/O to the guest domains.One is virtual disk (vds) for storage and another one is virtual switch(vsw) for  network communication.But in I/O domain,you can directly access the real physical LUN or physical network port.But the typical guest domain will be using the virtual I/O services.For an example, SAN LUN’s will be mapped to control domain and from there , you can map the physical LUNS to guest domains as virtual services.

Oracle VM for SPARC Tutorial:

IO Domain Example
IO Domain Example

In the above image, control domain aka primary domain and SOL10 Guest domain also called I/O domains since its directly accessing the PCI card(HBA). Here we will see how we can configure the guest domain as I/O domain.

1. Login to control domain and list the complete hardware details of the server .

-bash-3.2# ldm list-devices -a
CORE
    ID      %FREE   CPUSET
    0       0       (0, 1, 2, 3)
    1       0       (4, 5, 6, 7)
    2       0       (8, 9, 10, 11)
    3       0       (12, 13, 14, 15)
    4       0       (16, 17, 18, 19)
    5       0       (20, 21, 22, 23)
    6       0       (24, 25, 26, 27)
    7       0       (28, 29, 30, 31)

VCPU
    PID     %FREE   PM
    0       0       no
    1       0       no
    2       0       no
    3       0       no
    4       0       no
    5       0       no
    6       0       no
    7       0       no
    8       0       no
    9       0       no
    10      0       no
    11      0       no
    12      0       no
    13      0       no
    14      0       no
    15      0       no
    16      0       no
    17      0       no
    18      0       no
    19      0       no
    20      0       no
    21      0       no
    22      0       no
    23      0       no
    24      0       no
    25      0       no
    26      0       no
    27      0       no
    28      0       no
    29      0       no
    30      0       no
    31      0       no

MAU
    ID      CPUSET                                  BOUND
    0       (0, 1, 2, 3)                            primary
    1       (4, 5, 6, 7)
    2       (8, 9, 10, 11)
    3       (12, 13, 14, 15)
    4       (16, 17, 18, 19)
    5       (20, 21, 22, 23)
    6       (24, 25, 26, 27)
    7       (28, 29, 30, 31)

MEMORY
    PA                   SIZE            BOUND
    0x0                  512K            _sys_
    0x80000              1536K           _sys_
    0x200000             62M             _sys_
    0x4000000            64M             _sys_
    0x8000000            2G              primary
    0x88000000           512M            ldom1
    0xa8000000           512M            ldom3
    0xc8000000           896M            ldom2

IO
    DEVICE           PSEUDONYM        BOUND   OPTIONS
    pci@780          bus_a            yes
    pci@7c0          bus_b            yes

-bash-3.2#

In the bottom of the above command output shows that IO (pci@780 and pci7c0) have already bind to some of the domain.

2.To find where the PCI IO card has been bound , you need to use “ldm list-bindings” command.

-bash-3.2# ldm list-bindings
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -ndcv-  SP      8     2G       0.6%  4h 58m

UUID
    0066bb91-df76-c6ca-a92b-f18faf256978

MAC
    00:14:4f:9d:ca:6a

HOSTID
    0x849dca6a

CONTROL
    failure-policy=ignore

DEPENDENCY
    master=

CORE
    CID    CPUSET
    0      (0, 1, 2, 3)
    1      (4, 5, 6, 7)

VCPU
    VID    PID    CID    UTIL STRAND
    0      0      0      0.2%   100%
    1      1      0      0.2%   100%
    2      2      0      0.1%   100%
    3      3      0      0.1%   100%
    4      4      1      0.1%   100%
    5      5      1      0.4%   100%
    6      6      1      0.1%   100%
    7      7      1      0.1%   100%

MAU
    ID     CPUSET
    0      (0, 1, 2, 3)

MEMORY
    RA               PA               SIZE
    0x408000000      0x8000000        2G

VARIABLES
    keyboard-layout=US-English

IO
    DEVICE           PSEUDONYM        OPTIONS
    pci@780          bus_a
    pci@7c0          bus_b

The above outputs confirms both the PCI card has been assigned to primary domain.So our primary domain is also called I/O domain or root domain.

3.We will remove one the PCI card from primary and we will assign to one of the guest domain. Before removing,please make sure that card is not used in primary .If its in use, please un-configure it .Let me confirm which card can be removed .(pci@780 and pci@7c0)

pci@780 is used for internal disks. We are good to move the pci@7c0 to one of the guest domains since there is no device is currently using it from primary domain.

AVAILABLE DISK SELECTIONS:
       0. c0t0d0 
          /pci@780/pci@0/pci@9/scsi@0/sd@0,0
       1. c0t1d0 
          /pci@780/pci@0/pci@9/scsi@0/sd@1,0
-bash-3.2# dladm show-dev
vsw0            link: up        speed: 1000  Mbps       duplex: full
e1000g0         link: up        speed: 1000  Mbps       duplex: full
e1000g1         link: unknown   speed: 0     Mbps       duplex: half
e1000g2         link: unknown   speed: 0     Mbps       duplex: half
e1000g3         link: unknown   speed: 0     Mbps       duplex: half
-bash-3.2# cat /etc/path_to_inst |grep e1000g
"/pci@780/pci@0/pci@1/network@0" 0 "e1000g"
"/pci@780/pci@0/pci@1/network@0,1" 1 "e1000g"
"/pci@7c0/pci@0/pci@2/network@0" 2 "e1000g"
"/pci@7c0/pci@0/pci@2/network@0,1" 3 "e1000g"
-bash-3.2#

4.List the configured guest domain.

-bash-3.2# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      8     2G       0.5%  4h 51m
ldom1            bound      ------  5000    8     512M
ldom2            bound      ------  5001    8     896M
ldom3            active     -n----  5002    8     512M     0.0%  3h 20m
-bash-3.2#

5. Remove or unbind the  pci@7c0 to from primary domain.You can see couple of physical network adapters has been removed from primary.You may need to reboot the server to release the pci cards sometimes.

-bash-3.2# ldm remove-io pci@7c0
-bash-3.2# dladm show-dev
vsw0            link: up        speed: 1000  Mbps       duplex: full
e1000g0         link: up        speed: 1000  Mbps       duplex: full
e1000g1         link: unknown   speed: 0     Mbps       duplex: half
-bash-3.2#
-bash-3.2# ldm list-bindings
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      8     2G       0.2%  3h 57m

IO
    DEVICE           PSEUDONYM        OPTIONS
    pci@780          bus_a

6.Now add the pci card  pci7c0 to ldom1 .sometimes you may need to reboot the system to see the new pci cards.

# ldm add-io pci@7c0 ldom1
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.
-bash-3.2#

7.You can list the binding using “ldm list-bindings” command to confirm the changes.sometimes you need to reboot the system to see the devices  in operating system level.

ldm list-bindings ldom1
    NAME             SERVICE                     PORT
    SP

------------------------------------------------------------------------------
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
ldom1            bound      ------  5000    8     512M

UUID
    613e0c0c-3554-610d-e36f-9441124ce387

MAC
    00:14:4f:fb:11:83

HOSTID
    0x84fb1183

CONTROL
    failure-policy=ignore

DEPENDENCY
    master=

CORE
    CID    CPUSET
    2      (8, 9, 10, 11)
    3      (12, 13, 14, 15)

VCPU
    VID    PID    CID    UTIL STRAND
    0      8      2             100%
    1      9      2             100%
    2      10     2             100%
    3      11     2             100%
    4      12     3             100%
    5      13     3             100%
    6      14     3             100%
    7      15     3             100%

MEMORY
    RA               PA               SIZE
    0x8000000        0x88000000       512M

VARIABLES
    auto-boot?=false
    boot-device=vdisk1:a disk net
    keyboard-layout=US-English

IO
    DEVICE           PSEUDONYM        OPTIONS
    pci@7c0          bus_b

8. Login to ldom1 and see the pci card devices .

-bash-3.2# dladm show-dev
vsw0            link: up        speed: 1000  Mbps       duplex: full
e1000g2         link: up        speed: 1000  Mbps       duplex: full
e1000g3         link: up        speed: 1000  Mbps       duplex: full
-bash-3.2#

Now ldom1 is also called as I/O domain since its accessing the network port directly without hypervisor. The same way if your system have multiple HBA-cards , you can assign one of the cards to guest domain and SAN team can directly allocate tothe luns to guest domain.(ohhh sorry now its called I/O domain)

You have successfully removed the pci card from primary domain and assigned to the guest domain. So the guest domain also become I/O domain now. These I/O domain will be used on special application requirements. Otherwise we will configure only the guest domains.

VMTURBO-CLOUD-CAPACITY

One comment

  1. Can we unbind only the HBA which is completely free and not being used for any other purpose such as for the SAN storage. Is there a way we can still use an HBA or any card both in the primary domain and at the same time in other IO domains ? The same HBA card cannot be shared among different IO doamins and the primary domain?

    Thanks.

Leave a Reply

Your email address will not be published. Required fields are marked *