Create VM (VirtualBox)

VirtualBox установлен:

C:\Program Files\Oracle\VirtualBox\VirtualBox.exe
C:\Users\angor\VirtualBox VMs


Дистрибутив OEL и VBoxGuestAdditions

C:\distrib\oel\V921569-01.iso
C:\distrib\oel\VBoxGuestAdditions_5.2.6.iso


cd C:\Program Files\Oracle\VirtualBox\

1. Создание VM на Windows Host Operating System

-- Создаём новый 50G виртуальный hard disk image с именем 12cRAC1.vdi:
VBoxManage createhd --filename "C:\Users\angor\VirtualBox VMs\12cRAC1.vdi" --size 51200 --format VDI --variant Fixed

-- Создаём новую Virtual Machine с именем 12cRAC1:
VBoxManage createvm --name 12cRAC1 --ostype "Oracle_64" --register

-- Добавим SATA controller:
VBoxManage storagectl 12cRAC1 --name "SATA" --add sata --controller IntelAHCI

-- Подключим виртуальный диск:
VBoxManage storageattach 12cRAC1 --storagectl "SATA" --port 0 --device 0 --type hdd --medium "C:\Users\angor\VirtualBox VMs\12cRAC1.vdi"

-- Добавим IDE controller:
VBoxManage storagectl 12cRAC1 --name "IDE" --add ide

-- Подключим DVD drive с ISO файлом дистрибутива Oracle Linux 7.3:
VBoxManage storageattach 12cRAC1 --storagectl "IDE" --port 0 --device 0 --type dvddrive --medium C:\distrib\oel\V921569-01.iso

-- Подключим DVD drive с ISO файлом дистрибутива VBoxGuestAdditions:
VBoxManage storageattach 12cRAC1 --storagectl "IDE" --port 1 --device 0 --type dvddrive --medium C:\distrib\oel\VBoxGuestAdditions_5.2.8.iso

-- Системные настройки:
VBoxManage modifyvm 12cRAC1 --ioapic on
VBoxManage modifyvm 12cRAC1 --boot1 dvd --boot2 disk --boot3 none --boot4 none
VBoxManage modifyvm 12cRAC1 --cpus 2 --memory 4096 --vram 128
VBoxManage hostonlyif create
VBoxManage modifyvm 12cRAC1 --nic1 nat --nic2 hostonly --nic3 intnet

2. Установка  Oracle Linux 7.3:

Server with GUI

Выбираем следующее:

Hardware Monitoring Utilities
Java Platform
KDE
Large Systems Performance
Network file system client
Performance Tools
Compatibility Libraries
Development Tools


Сетевые настройки такие:

hostname: rac12c01
enp0s3 (eth0): DHCP (Connect Automatically)

enp0s8 (eth1): IP= 192.168.10.11,
               Subnet=255.255.255.0,
               Gateway=192.168.10.1,
               DNS=192.168.10.1,
               Search=localdomain (Connect Automatically)

enp0s9 (eth2): IP= 192.168.20.11,
               Subnet=255.255.255.0,
               Gateway=(blank),
               DNS=(blank),
               Search=(blank) (Connect Automatically)



Сеть можно будет настроить позже с помощью  NetworkManager:


Для rac12c01:

root@rac12c01# nmcli con del enp0s3
root@rac12c01# nmcli con add con-name enp0s3 ifname enp0s3 type ethernet

root@rac12c01# nmcli con del enp0s8
root@rac12c01# nmcli con add con-name enp0s8 ifname enp0s8 type ethernet ip4 192.168.10.11/24 gw4 192.168.10.1
root@rac12c01# nmcli con mod enp0s8 ipv4.dns 192.168.10.1
root@rac12c01# nmcli con mod enp0s8 ipv4.dns-search localdomain
root@rac12c01# nmcli con mod enp0s8 autoconnect true

root@rac12c01# nmcli con del enp0s9
root@rac12c01# nmcli con add con-name enp0s9 ifname enp0s9 type ethernet ip4 192.168.20.11/24
root@rac12c01# nmcli con mod enp0s9 autoconnect true

root@rac12c01# nmcli con mod enp0s3 ipv6.method "ignore"
root@rac12c01# nmcli con mod enp0s8 ipv6.method "ignore"
root@rac12c01# nmcli con mod enp0s9 ipv6.method "ignore"


Для rac12c02:

root@rac12c02# nmcli con del enp0s3
root@rac12c02# nmcli con add con-name enp0s3 ifname enp0s3 type ethernet

root@rac12c02# nmcli con del enp0s8
root@rac12c02# nmcli con add con-name enp0s8 ifname enp0s8 type ethernet ip4 192.168.10.12/24 gw4 192.168.10.1
root@rac12c02# nmcli con mod enp0s8 ipv4.dns 192.168.10.1
root@rac12c02# nmcli con mod enp0s8 ipv4.dns-search localdomain
root@rac12c02# nmcli con mod enp0s8 autoconnect true

root@rac12c02# nmcli con del enp0s9
root@rac12c02# nmcli con add con-name enp0s9 ifname enp0s9 type ethernet ip4 192.168.20.12/24
root@rac12c02# nmcli con mod enp0s9 autoconnect true

root@rac12c02# nmcli con mod enp0s3 ipv6.method "ignore"
root@rac12c02# nmcli con mod enp0s8 ipv6.method "ignore"
root@rac12c02# nmcli con mod enp0s9 ipv6.method "ignore"

Или так

Для rac12c01:
root@rac12c01# nmcli con del enp0s3
root@rac12c01# nmcli con del enp0s8
root@rac12c01# nmcli con del enp0s9
root@rac12c01# nmcli con add con-name enp0s3 ifname enp0s3 type ethernet
root@rac12c01# nmcli con add con-name enp0s8 ifname enp0s8 type ethernet
root@rac12c01# nmcli con add con-name enp0s9 ifname enp0s9 type ethernet
root@rac12c01# nmcli con mod enp0s8 ipv4.method manual ipv4.address 192.168.10.11/24 ipv4.gateway 192.168.10.1 ipv4.dns 192.168.10.1 ipv4.dns-search localdomain autoconnect true
root@rac12c01# nmcli con mod enp0s9 ipv4.method manual ipv4.address 192.168.20.11/24 autoconnect true


Для rac12c02:

root@rac12c02# nmcli con del enp0s3
root@rac12c02# nmcli con del enp0s8
root@rac12c02# nmcli con del enp0s9
root@rac12c02# nmcli con add con-name enp0s3 ifname enp0s3 type ethernet
root@rac12c02# nmcli con add con-name enp0s8 ifname enp0s8 type ethernet
root@rac12c02# nmcli con add con-name enp0s9 ifname enp0s9 type ethernet
root@rac12c02# nmcli con mod enp0s8 ipv4.method manual ipv4.address 192.168.10.12/24 ipv4.gateway 192.168.10.1 ipv4.dns 192.168.10.1 ipv4.dns-search localdomain autoconnect true
root@rac12c02# nmcli con mod enp0s9 ipv4.method manual ipv4.address 192.168.20.12/24 autoconnect true


# nmcli con reload



[root@rac12c01 ~]# cd /etc/sysconfig/network-scripts/
[root@rac12c01 network-scripts]#

[root@rac12c01 network-scripts]# ls -l |grep ifcfg
-rw-r--r--  1 root root    58 Mar 25  2018 ifcfg-enp0s3
-rw-r--r--. 1 root root   370 Mar  3  2018 ifcfg-enp0s8
-rw-r--r--. 1 root root   312 Mar  3  2018 ifcfg-enp0s9
-rw-r--r--. 1 root root   254 May  3  2017 ifcfg-lo
[root@rac12c01 network-scripts]#

[root@rac12c01 network-scripts]# more ifcfg-enp0s3
BOOTPROTO=dhcp
DEVICE=enp0s3
ONBOOT=yes
NM_CONTROLLED=no
[root@rac12c01 network-scripts]#


[root@rac12c01 network-scripts]# more ifcfg-enp0s8
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp0s8
UUID=c0fecc6f-b5dc-4118-b96e-f5f6c7e23be1
DEVICE=enp0s8
ONBOOT=yes
IPADDR=192.168.10.11
PREFIX=24
GATEWAY=192.168.10.1
DNS1=192.168.10.1
DOMAIN=localdomain
[root@rac12c01 network-scripts]#


[root@rac12c01 network-scripts]# more ifcfg-enp0s9
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp0s9
UUID=ad583180-d0fb-4fb9-b8da-40974e4fd937
DEVICE=enp0s9
ONBOOT=yes
IPADDR=192.168.20.11
PREFIX=24
[root@rac12c01 network-scripts]#


[root@rac12c01 network-scripts]# more ifcfg-lo
DEVICE=lo
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback
[root@rac12c01 network-scripts]#




[root@rac12c01 network-scripts]# ifconfig -a
enp0s3: flags=4163  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::a00:27ff:fe73:f560  prefixlen 64  scopeid 0x20
        ether 08:00:27:73:f5:60  txqueuelen 1000  (Ethernet)
        RX packets 353265  bytes 466625001 (445.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 40284  bytes 2524665 (2.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8: flags=4163  mtu 1500
        inet 192.168.10.11  netmask 255.255.255.0  broadcast 192.168.10.255
        inet6 fe80::a00:27ff:fea4:ebc7  prefixlen 64  scopeid 0x20
        ether 08:00:27:a4:eb:c7  txqueuelen 1000  (Ethernet)
        RX packets 17  bytes 3516 (3.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 138  bytes 21256 (20.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8:1: flags=4163  mtu 1500
        inet 192.168.10.10  netmask 255.255.255.0  broadcast 192.168.10.255
        ether 08:00:27:a4:eb:c7  txqueuelen 1000  (Ethernet)

enp0s8:2: flags=4163  mtu 1500
        inet 192.168.10.101  netmask 255.255.255.0  broadcast 192.168.10.255
        ether 08:00:27:a4:eb:c7  txqueuelen 1000  (Ethernet)

enp0s8:3: flags=4163  mtu 1500
        inet 192.168.10.102  netmask 255.255.255.0  broadcast 192.168.10.255
        ether 08:00:27:a4:eb:c7  txqueuelen 1000  (Ethernet)

enp0s9: flags=4163  mtu 1500
        inet 192.168.20.11  netmask 255.255.255.0  broadcast 192.168.20.255
        inet6 fe80::a00:27ff:fe6b:7e0a  prefixlen 64  scopeid 0x20
        ether 08:00:27:6b:7e:0a  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 307  bytes 53396 (52.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s9:1: flags=4163  mtu 1500
        inet 169.254.98.147  netmask 255.255.0.0  broadcast 169.254.255.255
        ether 08:00:27:6b:7e:0a  txqueuelen 1000  (Ethernet)

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 20397  bytes 46215763 (44.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 20397  bytes 46215763 (44.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:09:73:ff  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0-nic: flags=4098  mtu 1500
        ether 52:54:00:09:73:ff  txqueuelen 500  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@rac12c01 network-scripts]#



[root@rac12c01 network-scripts]# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:73:f5:60 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 85429sec preferred_lft 85429sec
    inet6 fe80::a00:27ff:fe73:f560/64 scope link
       valid_lft forever preferred_lft forever
3: enp0s8: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:a4:eb:c7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.11/24 brd 192.168.10.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet 192.168.10.10/24 brd 192.168.10.255 scope global secondary enp0s8:1
       valid_lft forever preferred_lft forever
    inet 192.168.10.101/24 brd 192.168.10.255 scope global secondary enp0s8:2
       valid_lft forever preferred_lft forever
    inet 192.168.10.102/24 brd 192.168.10.255 scope global secondary enp0s8:3
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fea4:ebc7/64 scope link
       valid_lft forever preferred_lft forever
4: enp0s9: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:6b:7e:0a brd ff:ff:ff:ff:ff:ff
    inet 192.168.20.11/24 brd 192.168.20.255 scope global enp0s9
       valid_lft forever preferred_lft forever
    inet 169.254.98.147/16 brd 169.254.255.255 scope global enp0s9:1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe6b:7e0a/64 scope link
       valid_lft forever preferred_lft forever
5: virbr0: mtu 1500 qdisc noqueue state DOWN
    link/ether 52:54:00:09:73:ff brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
6: virbr0-nic: mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:09:73:ff brd ff:ff:ff:ff:ff:ff
[root@rac12c01 network-scripts]# 



[root@rac12c01 network-scripts]# more /etc/hosts
127.0.0.1 localhost.localdomain localhost

# Public (eth1)
192.168.10.11 rac12c01.localdomain rac12c01
192.168.10.12 rac12c02.localdomain rac12c02
# Private (eth2)
192.168.20.11 rac12c01-priv.localdomain rac12c01-priv
192.168.20.12 rac12c02-priv.localdomain rac12c02-priv
# Virtual
192.168.10.101 rac12c01-vip.localdomain rac12c01-vip
192.168.10.102 rac12c02-vip.localdomain rac12c02-vip
# SCAN
192.168.10.10 scan12c.localdomain scan12c
[root@rac12c01 network-scripts]#


[root@rac12c01 network-scripts]# more /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
nameserver 192.168.1.1
[root@rac12c01 network-scripts]#

Переконфигурация сбойного узла в кластере.

Имеется двухнодовый кластер (узлы host01 и host02)

Необходимо переконфигурировать узел host02

1. Удалить сбойный узел host02 из кластера.

Удалить ПО RDBMS из inventory работающего узла host01.
(Оставить только host01)

oracle@host01# ./runInstaller -updateNodeList ORACLE_HOME=$RDBMS_HOME "CLUSTER_NODES={host01}" LOCAL_NODE=host01


Удалить ПО GI из inventory работающего узла host01.
(Оставить только host01)

oracle@host01# ./runInstaller -updateNodeList ORACLE_HOME=$GRID_HOME "CLUSTER_NODES={host01}" CRS=TRUE -silent


Удалить сбойный узел host02 из OCR (реестра кластера).

root@host01# crsctl delete node -n host02


Удалить VIP на хосте host02.

root@host01# srvctl config vip -node host02
VIP exists: network number 1, hosting node host02
VIP Name: host02-vip
VIP IPv4 Address: 192.168.1.101
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:

root@host01# srvctl stop vip -vip host02-vip -force
root@host01# srvctl remove vip -vip host02-vip
Please confirm that you intend to remove the VIPs host02-vip (y/[n]) y


Удалить экземпляр orcl2.

root@host01# srvctl remove instance -db orcl -instance orcl2
Remove instance from the database orcl? (y/[n]) y

На данный момент плохой узел host02 больше не является частью кластера.

2. Переходим на host02 и очистим остатки некоторых файлов конфигурации GI и RDBMS на этом узле

oracle@host02$ rm -rf /u01/app/oracle/product/12.1.0.2/
root@host02# rm -rf /u01/grid/crs12.1.0.2/*
root@host02# rm /var/tmp/.oracle/*
oracle@host02$ rm -rf /tmp/*
root@host02# rm /etc/oracle/ocr*
root@host02# rm /etc/oracle/olr*
root@host02# rm -rf /pkg/oracle/app/oraInventory
root@host02# rm -rf /etc/oracle/scls_scr

3. Добавляем узел host02 обратно в кластер

oracle@host01$ cd $GRID_HOME/addnode

oracle@host01$ ./addnode.sh -ignoreSysPrereqs -ignorePrereq -silent "CLUSTER_NEW_NODES={host02}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={host02-vip}"

Это приведет к клонированию  GI_HOME с host01 на host02.
В конце будет предложено запустить root.sh на host02.
Запуск этого скрипта приведет к подключению GI к OCR и Voting disks и запуску clusterware stackvting.

Но вначале запустим.

root@host02# cd $GRID_HOME/crs/install
root@host02# ./rootcrs.sh -verbose -deconfig -force

root@host02# cd $GRID_HOME
root@host02# ./root.sh


host02 теперь входит в состав кластера, а GI работает.
Проверяем

root@host02# crs_stat -t
root@host02# olsnodes -n


Также проверяем VIP.

root@host02# srvctl status vip -vip host02-vip
VIP host02-vip is enabled
VIP host02-vip is running on node: host02


Запустим клонирование RDBMS_HOME с host01 на host02.

oracle@host01]$ cd $RDBMS_HOME/addnode
oracle@host01]$ ./addnode.sh "CLUSTER_NEW_NODES={host02}"

Это запустит OUI. Проходим мастер, чтобы завершить процесс клонирования.

4. Добавим экземпляр обратно на узел host02.

oracle@host01$ srvctl add instance -db orcl -instance orcl2 -node host02
Если все пойдет хорошо, экземпляр запустится.

Проверяем:

oracle@host01$ srvctl start instance -db orcl -instance orcl2

oracle@host01$ srvctl status database -d orcl
Instance orcl1 is running on node host01
Instance orcl2 is running on node host02

SQL> select inst_id,status from gv$instance;

INST_ID STATUS
---------- ------------
 1 OPEN
 2 OPEN


Остается только перенастроить и запустить любые необходимые службы.

srvctl modify service -db orcl -service my_svc -modifyconfig -preferred "orcl1,orcl2"
srvctl start service -db orcl -service my_svc -node host02
srvctl status service -db orcl

Все работает.