Oracle Multitenant plugin
the file names in the multitenant manifest or in the full transportable database import command must match all files exactly for the operation to complete

There are some difficulties when OMF/ASM is used, files are copied, and a physical standby database is in place. There have been improvements made to the multitenant plugin operation on both the primary and standby environments, however at this time additional work must still be done on the standby database when a full transportable database import operation is performed.

RMAN has been enhanced so that, when copying files between databases it recognizes the GUID and acts accordingly when writing the files.

* If the clone/auxiliary instance being connected to for clone operations is a CDB root, the GUID of the RMAN target database is used to determine the directory structure to write the datafiles. Connect to the CDB root as the RMAN clone/auxiliary instance when the source database should be a 12c non-CDB or PDB that is going to be migrated and plugged into a remote CDB as a brand new PDB. This will ensure that the files copied by RMAN will be written to the GUID directory of source database for the migration.

* If the clone/auxiliary instance being connected to for clone operations is a PDB, the GUID of the auxiliary PDB will be used to determine the directory structure to write the datafiles. Connect to the destination PDB as the RMAN clone auxiliary instance when the source database is a 12c non-CDB or PDB that requires a cross platform full transportable database import and the data and files will be imported into an existing PDB. This will ensure the files copied by RMAN will be written to the GUID directory of the PDB target database for the migration.

Posted in ASM

If you saw the first FlexASM blog you know we installed and configured FlexASM and a CDB plus a couple of PDBs. Also, this was Policy Managed with a cardinality of 2.  Now let's see what the configuration looks like, and we can break it down using the wonderful crsctl and srvctl tools

First let's ensure we are really running in FlexASM mode:

[oracle@rac02 ~]$ asmcmd showclustermode
ASM cluster : Flex mode enabled


[oracle@rac02 ~]$ srvctl status   serverpool -serverpool naboo
Server pool name: naboo
Active servers count: 2



[oracle@rac01 trace]$ crsctl get node role status -all
Node 'rac01' active role is 'hub'
Node 'rac03' active role is 'hub'
Node 'rac02' active role is 'hub'
Node 'rac04' active role is 'hub'



[oracle@rac01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
               ONLINE  ONLINE       rac03                    STABLE
               ONLINE  ONLINE       rac04                    STABLE  


You notice that we have 4 ASM listeners one on each node in the Cluster.  You'll see the process as the following on each node:


[oracle@rac01 ~]$ ps -ef |grep -i asmnet

ooracle    6646     1  0 12:19 ?        00:00:00 /u01/app/12.1.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit



ora.CRSDATA.DATAVOL1.advm
               ONLINE  ONLINE       rac01                    Volume device /dev/a
                                                             sm/datavol1-194 is o
                                                             nline,STABLE
               ONLINE  ONLINE       rac02                    Volume device /dev/a
                                                             sm/datavol1-194 is o
                                                             nline,STABLE
               ONLINE  OFFLINE      rac03                    Unable to connect to
                                                             ASM,STABLE
               ONLINE  ONLINE       rac04                    Volume device /dev/a
                                                             sm/datavol1-194 is o
                                                             nline,STABLE
The datavol1 ADVM resource runs on all the nodes where indicated it should run.  In this case we are seeing that RAC03 is having some issues.
Let's look into that a little later.  But I like the fact crsctl tells something is amiss here on node3
 
ora.CRSDATA.dg
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
               ONLINE  ONLINE       rac03                    STABLE
               OFFLINE OFFLINE      rac04                    STABLE


ora.FRA.dg
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
               ONLINE  ONLINE       rac03                    STABLE
               OFFLINE OFFLINE      rac04                    STABLE


The crsdata and fra disk groups resource is started on all nodes except node 4



ora.LISTENER.lsnr
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
               ONLINE  ONLINE       rac03                    STABLE
               ONLINE  ONLINE       rac04                    STABLE


We all know, as in 11gR2, that this is the Node listener.


ora.PDBDATA.dg
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
               ONLINE  ONLINE       rac03                    STABLE
               OFFLINE OFFLINE      rac04                    STABLE


The pdbdata disk groups resource is started on all nodes except node 4



ora.crsdata.datavol1.acfs
               ONLINE  ONLINE       rac01                    mounted on /u02/app/
                                                             oracle/acfsmounts,ST
                                                             ABLE
               ONLINE  ONLINE       rac02                    mounted on /u02/app/
                                                             oracle/acfsmounts,ST
                                                             ABLE
               ONLINE  OFFLINE      rac03                    (2) volume /u02/app/
                                                             oracle/acfsmounts of
                                                             fline,STABLE
               ONLINE  ONLINE       rac04                    mounted on /u02/app/
                                                             oracle/acfsmounts,ST
                                                             ABLE


ACFS filesystem resource for datavol1 is started on all nodes except node3.
But I think the following has something to do w/ it :-).  Need to debug this a bit later.  I even tried:
[oracle@rac03 ~]$ asmcmd volenable --all
ASMCMD-9470: ASM proxy instance unavailable
ASMCMD-9471: cannot enable or disable volumes



ora.net1.network
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
               ONLINE  ONLINE       rac03                    STABLE
               ONLINE  ONLINE       rac04                    STABLE
ora.ons
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
               ONLINE  ONLINE       rac03                    STABLE
               ONLINE  ONLINE       rac04                    STABLE


The Network (in my case I only have only Net1) and ONS are same as in previous versions


ora.proxy_advm
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
               ONLINE  OFFLINE      rac03                    STABLE
               ONLINE  ONLINE       rac04                    STABLE


Yep, since proxy_advm is not started on node3, the filesystems won't come online….but again, i'll look at that later
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac02                    STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac03                    STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac04                    STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rac01                    169.254.90.36 172.16
                                                             .11.10,STABLE
ora.asm
      1        ONLINE  ONLINE       rac03                    STABLE
      2        ONLINE  ONLINE       rac01                    STABLE
      3        ONLINE  ONLINE       rac02                    STABLE


Since we have the cardinality of 3 ASM instance we have 3 ASM resources active


ora.cvu
      1        ONLINE  ONLINE       rac01                    STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rac01                    Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       rac01                    STABLE
ora.rac01.vip
      1        ONLINE  ONLINE       rac01                    STABLE
ora.rac02.vip
      1        ONLINE  ONLINE       rac02                    STABLE
ora.rac03.vip
      1        ONLINE  ONLINE       rac03                    STABLE
ora.rac04.vip
      1        ONLINE  ONLINE       rac04                    STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac02                    STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac03                    STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac04                    STABLE
ora.tatooine.db
      1        ONLINE  ONLINE       rac01                    Open,STABLE
      2        ONLINE  ONLINE       rac02                    Open,STABLE


As we stated above, I specified a Policy Managed database with cardinality of 2, so I have 2 database instances running
--------------------------------------------------------------------------------

Here's some other important supporting info on FlexASm:


[oracle@rac02 ~]$ srvctl config asm -detail
ASM home: /u01/app/12.1.0/grid
Password file: +CRSDATA/orapwASM
ASM listener: LISTENER
ASM is enabled.
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM


[oracle@rac02 ~]$ srvctl status  filesystem
ACFS file system /u02/app/oracle/acfsmounts is mounted on nodes rac01,rac02,rac04


ANd here's what the Database has to say about FlexASM

NOTE: ASMB registering with ASM instance as client 0x10001 (reg:1377584805)
NOTE: ASMB connected to ASM instance +ASM1 (Flex mode; client id 0x10001)
NOTE: ASMB rebuilding ASM server state
NOTE: ASMB rebuilt 2 (of 2) groups
SUCCESS: ASMB reconnected & completed ASM server state


So for the interesting part:
If you notice that ASM is not running node 4:
[oracle@rac02 ~]$ srvctl status  asm -v

ASM is running on rac01,rac02,rac03
[oracle@rac02 ~]$ srvctl status  asm -detail
ASM is running on rac01,rac02,rac03



So, how does a client (ocrdump, rman, asmcmd, etc..) connect to if ASM if there is no ASM on that node.  Well let's test this using asmcmd on node4.  You notice that a pipe is created, a connect string is generated and passed to ASMCMD to connect remotely to ASM2 on node2!!!!


22-Sep-13 12:54 ASMCMD Foreground (PID = 14106):  Pipe /tmp/pipe_14106 has been found.
22-Sep-13 12:54 ASMCMD Background (PID = 14117):  Successfully opened the pipe /tmp/pipe_14106
22-Sep-13 12:54 ASMCMD Foreground (PID = 14106):  Successfully opened the pipe /tmp/pipe_14106 in read mode
NOTE: Executing kfod /u01/app/12.1.0/grid/bin/kfod op=getclstype..
22-Sep-13 12:54 Printing the connection string
contype = 
driver = 
instanceName = <>
usr = 
ServiceName = <+ASM>
23-Sep-13 16:23 Successfully connected to ASM instance +ASM2
23-Sep-13 16:23 NOTE: Querying ASM instance to get list of disks
22-Sep-13 12:54 Registered Daemon process.
22-Sep-13 12:54 ASMCMD Foreground (PID = 14106):  Closed pipe /tmp/pipe_14106.