-------
To address the growing need for use of more memory on 32-Bit Windows platforms,
and explain how AWE is implemented by Oracle on Windows.
SCOPE & APPLICATION
-------------------
Oracle DBA's running on the Microsoft Windows platform.
Oracle Support Analysts, Field Engineers troubleshooting problems
related to AWE and/or memory issues on Windows.
AWE Memory implementation on Windows 2000
------------------------------------------
A common question on the Windows NT/Windows 2000 platform revolves around
how to take advantage of systems with more than 4 GB of RAM. As discussed
in Metalink Note 46001.1 and Note 46053.1, the 32-Bit process address
space for any process on Windows equates to a total of 4GB of addressable
RAM. Of this, by default, 2GB is reserved for the process itself, and 2GB
for the kernel. On systems running either Windows 2000 Advanced Server,
or Windows NT 4.0 Enterprise Edition, this ratio can be changed by adding
the /3GB switch to the boot.ini, allowing a process to address 3GB and
reserving 1GB for the kernel. However, the total addressable memory for
a single process is still only 4GB.
See also Note 1036312.6 : Utilizing Up to 3GB Virtual Memory on Windows NT Server 4.0
__________________________________________________________________
What can be done to address memory beyond 4GB?:
===============================================
The answer is to take advantage of Physical Address Extensions (PAE), or
Address Windowing Extensions (AWE)(These two terms are used interchangeably,
so the rest of this document will refer to this simply as AWE).
AWE support is available if you are running on a machine with more than 4GB
of physical RAM which is running any of the below Windows operating systems:
* Windows 2000 Datacenter Server
* Windows 2000 Advanced Server
* Windows 2003 Data Center Edition (32-Bit)
* Windows 2003 Enterprise Edition (32-Bit)
On the above operating systems, AWE support is built into the OS. No
special drivers are needed to take advantage of the additional memory.
AWE CANNOT be used on the following Operating Systems:
* Windows 2000 Server (Standard)
* Windows 2000 Professional
* Windows XP Home Edition
* Windows XP Professional
* Windows 2003 Standard Edition
* Windows 2003 Web Edition
NOTE Also that on 64-Bit Windows operating systems, there is no need for AWE
implementation support, because the directly addressable memory for a single
process on 64-Bit Windows is 8 Terabytes.
__________________________________________________________________
Oracle versions that can use AWE:
=================================
Oracle can take advantage of AWE in the following RDBMS releases:
* Oracle 8.1.6.x
* Oracle 8.1.7.x
* Oracle 9.2.x
* Oracle 10.1.x
Oracle does NOT implement AWE support in release 9.0.1.x
AWE support is available on both the Enterprise Edition of Oracle and
the Standard Edition of Oracle. However, on Standard Edition of 9.2.0.1,
you may receive the following error if trying to start the database with
USE_INDIRECT_DATA_BUFFERS=TRUE:
ORA-439 - FEATURE NOT ENABLED: VERY LARGE MEMORY
In Standard Edition 9.2.0.2 and 9.2.0.3, you will not receive the above errors,
but VLM functionality is still not enabled. Refer to BUG#2945011 for more detail.
This BUG is fixed in 9.2.0.3 Patch 2, and will be fixed in 9.2.0.4 as well.
__________________________________________________________________
Enabling support at the OS level:
==================================
AWE can be enabled at the OS by adding the /PAE switch to the boot.ini
as such:
multi(0)disk(0)rdisk(0)partition(1)\WINNT="Microsoft Windows 2000 Advanced Server" /PAE
It IS possible to have BOTH the /PAE and /3GB switch in place on the same
machine, as such:
multi(0)disk(0)rdisk(0)partition(1)\WINNT="Microsoft Windows 2000 Advanced Server" /3GB /PAE
However, be aware that if BOTH switches are in place, the server will only
be able to recognize up to 16GB of RAM. If you are working with a server
with more than 16GB of RAM, you will need to choose between the two.
It is important to note that once either or both of these switches are in
place in the boot.ini, ALL processes running can take advantage of these
switches. Thus, in a case where multiple Oracle instances are running on
the same server, ALL instances can take advantage of the additional memory
afforded by these switches, up to the amount of physical memory on the box.
Operating System Privileges Needed at the OS Level:
====================================================
In order to take advantage of the additional memory afforded through PAE,
the operating system user account which is used to start the OracleService
must be granted the 'Lock Pages in Memory' system privilege at the operating system
level. By default, the OracleService
The LocalSystem account has the privilege to Lock Pages in Memory granted to
it by default.
However, if you change the OracleService
LocalSystem, you may see the following errors when attempting to start the
database with USE_INDIRECT_DATA_BUFFERS set to TRUE :
SQL> startup pfile=c:\temp\initscott.ora
ORA-27102: out of memory
OSD-00010: Message 10 not found; product=RDBMS; facility=SOSD
O/S-Error: (OS 1300) Not all privileges referenced are assigned to the caller.
To rectify this, you must grant the 'Lock pages in memory' privilege to the user
that the OracleService
Start -> Programs -> Administrative Tools -> Local Security Policy
(on a Domain Controller, click on 'Domain Security Policy' instead of 'Local Security Policy')
Double-click on the 'Lock Pages in memory' policy.
Add the appropriate user and click 'Ok'.
Restart the OracleService
__________________________________________________________________
Understanding the Oracle implementation of AWE support:
=======================================================
What the PAE switch allows you to do from the Oracle perspective is to
increase the amount of memory that can be used for the Oracle Database
Block Buffer Cache. It is important to note that this additional memory
can ONLY be used by Oracle in the form of an increased value for
DB_BLOCK_BUFFERS.
There is still confusion on the old style of VLM versus AWE on Windows 2000.
With VLM on Windows NT 4.0, there was the concept of pointers pointing to
the extended memory area, but that is no longer the case on Windows 2000.
Instead, the windowing technology as described in these articles is being
used. For more information on AWE/PAE implementation on the Windows
platform, refer to Microsoft's website.
As mentioned previously, with AWE enabled, this allows the process(es)
(in this case ORACLE.EXE) to use memory above and beyond the 4GB
mark defined by a 32-Bit Process Address space. The physical location of
these blocks does not matter. However, the database blocks must still be
accessed from within a ‘window’, which exists (logically) in that regular
3GB process address space.
The size of this window is defined by a registry setting in the HOME key for
Oracle (HKLM\Software\Oracle\Homex) called AWE_WINDOW_MEMORY. By default,
this value is 1GB, so if this value is not set in the registry,
AWE_WINDOW_MEMORY will be 1GB.
It is important to realize that any database blocks accessed by Oracle
(or any user/background thread within Oracle.exe) must first be mapped into
the 'window' defined by AWE_WINDOW_MEMORY. In this scenario, it does not
matter where the blocks are physically located - there is no need to be
concerned with where the blocks are physically residing. The window will be
drawn around the block (i.e. the block will be mapped) wherever it is located
in memory. If the block is in memory but has not been mapped into the
‘window’, then it may be necessary to unmapped another block that IS in the
window, in order to accommodate the new block. While this mapping and
unmapping of blocks does add some cost, it is still faster than incurring
an I/O operation to read the block from disk. This will be discussed
further down in the section on troubleshooting.
Note:
Keep in mind that if there are multiple instances on a machine with
the /PAE switch enabled, ALL instances can take advantage of the additional
memory. However, AWE_WINDOW_MEMORY cannot be set on a per-instance basis,
so all databases that are running out of the HOMEx key where
AWE_WINDOW_MEMORY is set will inherit the same value.
__________________________________________________________________
Enabling AWE Support at the Database/Instance Level:
====================================================
To enable the AWE implementation on Oracle, you must set the following
parameter in the init file (or spfile) used to start the instance:
USE_INDIRECT_DATA_BUFFERS=TRUE
Note again that the buffer cache MUST be defined using the parameter
DB_BLOCK_BUFFERS, no matter what version of the RDBMS you are running.
The 9.2 feature allowing for Multiple block sizes in a database will be
disabled if you set USE_INDIRECT_DATA_BUFFERS=TRUE, and you cannot specify
the DB_CACHE_SIZE parameter to define the size of the buffer cache.
On 9.2, if you attempt to startup a database with this combination of
parameters:
USE_INDIRECT_DATA_BUFFERS=TRUE
DB_CACHE_SIZE=xxxxx (Any number)
The startup will fail with the following error:
SQL> startup
ORA-00385: cannot enable Very Large Memory with new buffer cache
parameters
You must change DB_CACHE_SIZE to use DB_BLOCK_BUFFERS instead, as was the
syntax under Oracle8i and earlier.
__________________________________________________________________
AWE_WINDOW_MEMORY Within the 3GB Process Address Space:
=======================================================
If you are using /PAE and the /3GB switch together, the address space for
ORACLE.EXE will be 3GB. The value for AWE_WINDOW_MEMORY must come from the
normal address space used by the ORACLE.EXE process. Memory that comes
from that 3GB address space addressable by the oracle.exe process includes
the following:
·The Value for AWE_WINDOW_MEMORY
·The rest of the SGA (shared_pool, large_pool, java_pool, log_buffers, etc)
·Overhead for Oracle.exe and DLL’s (65-100M depends on version & options)
·Stack space for all threads (Defaults to 1MB/thread, unless orastack
is used)
·PGA and UGA memory for all user sessions
Therefore, the value for AWE_WINDOW_MEMORY should be tuned such that mapping
and unmapping operations are avoided as much as possible, while still
allowing enough memory within the 3GB address space for the rest of the
process memory that MUST fit within the 3GB (i.e. overhead, remaining SGA
components and all user connection memory (stack + uga + pga) noted above).
The total size of the buffer cache can then be set to the amount of
physical memory remaining above the 4GB barrier, plus AWE_WINDOW_MEMORY.
On a machine with 12GB of RAM, using the default value of 1GB for
AWE_WINDOW_MEMORY, your total buffer cache could theoretically be as high
as 9GB:
(Total RAM - 4GB + AWE_WINDOW_MEMORY) = 12GB - 4GB + 1GB = 9GB
In reality, your maximum buffer cache size will be somewhat less than
this, allowing for some overhead and additional processes running on the
system.
Attempting to startup the database with a buffer cache larger than the
maximum value as calculated above may result in the following errors:
ORA-27102 out of memory
OSD-00034 Message 34 not found; Product=RDBMS;facility =SOSD
O/S Error: (OS 8) Not enough storage is available to process this command
(Note - If you are on Release 9.2, another possible cause for these errors
is noted further down, in the troubleshooting section)
As mentioned above, the buffer cache must be specified using
DB_BLOCK_BUFFERS rather than DB_CACHE_SIZE, so assuming an 8K block
size (8192), to get a 9GB buffer cache, you would set the following init
parameters:
DB_BLOCK_BUFFERS = 1179648
DB_BLOCK_SIZE = 8192
__________________________________________________________________
Troubleshooting AWE_WINDOW_MEMORY implementation:
=========================
=========================
Minimum Value Required for AWE_WINDOW_MEMORY in 9.2 and Above:
==============================================================
Here are key points to understand when using AWE_WINDOW_MEMORY:
1. Under Oracle 8.1.7 we do NOT enforce a minimum value for
AWE_WINDOW_MEMORY to be able to start the database.
2. This was changed under Oracle9i Release 2, such that we DO
enforce a minimum value for AWE_WINDOW_MEMORY. This change was
done to help improve performance by enforcing a larger window size.
3. You can alter the minimum required value for AWE_WINDOW_MEMORY
under 9.2 by changing/setting the value of the parameter
_DB_BLOCK_LRU_LATCHES. Under 8.1.7, this parameter was named
DB_BLOCK_LRU_LATCHES. However, under 9.x, this parameter was
changed to be a hidden parameter.
The minimum value for AWE_WINDOW_MEMORY starting with 9.2 is calculated as such:
MIN(AWE_WINDOW_MEMORY)=(4096 * DB_BLOCK_SIZE * _DB_BLOCK_LRU_LATCHES)/8
Starting with 9.2, to calculate the value for _DB_BLOCK_LRU_LATCHES, we need
this formula:
_DB_BLOCK_LRU_LATCHES = (Max buffer pools * SETS_PER_POOL)
Max Buffer Pools is a constant = 8
SETS_PER_POOL is variable, and depends on whether or not VLM is enabled.
SETS_PER_POOL = 2* CPU_COUNT (if VLM is enabled)
SETS_PER_POOL= CPU Count /2 (If VLM is NOT enabled)
/* Recall that VLM is enabled by setting USE_INDIRECT_DATA_BUFFERS=TRUE
So, as you can see, the value for _DB_BLOCK_LRU_LATCHES in 9.2 and above is
dependent on the number of CPU's in the box, and therefore
MIN(AWE_WINDOW_MEMORY) is dependent on the # of CPU's as well as the
DB_BLOCK_SIZE. The larger the Block Size, and the more CPU's in a system,
the higher the value for MIN(AWE_WINDOW_MEMORY). Here are a couple of
example configurations and caclulations showing MIN(AWE_WINDOW_MEMORY).
Example #1:
----------------
# of CPU's = 8
DB_BLOCK_SIZE = 8192
Total RAM = 8GB
SETS_PER_POOL = 2 * CPU_COUNT = 16
_DB_BLOCK_LRU_LATCHES = (max buffer Pools * sets_per_pool) = 8*16 = 128
MIN(AWE_WINDOW_MEMORY) =(4096*DB_BLOCK_SIZE*_DB_BLOCK_LRU_LATCHES)/8 =
( 4096 * 8192 * 128) / 8 = 536870912 bytes = 512 MB
Example #2:
---------------
# of CPU's = 16
DB_BLOCK_SIZE = 8192
Total RAM = 16 GB
SETS_PER_POOL = 2 * CPU_COUNT = 32
_DB_BLOCK_LRU_LATCHES = (max buffer Pools * sets_per_pool) = 8*32 = 256
MIN(AWE_WINDOW_MEMORY) =(4096*DB_BLOCK_SIZE*_DB_BLOCK_LRU_LATCHES)/8 =
( 4096 * 8192 * 256) / 8 = 1073741824 bytes = 1024 MB
These values above are the minimum values required for AWE_WINDOW_MEMORY
to be set to, UNLESS you explicitly set _DB_BLOCK_LRU_LATCHES to a lower
value. If AWE_WINDOW_MEMORY is not set to the minimum value, you will
receive the following errors:
ORA-27102 out of memory
OSD-00034 Message 34 not found; Product=RDBMS;facility =SOSD
O/S Error: (OS 8) Not enough storage is available to process this command
If you receive these errors when trying to start the database under 9.2 or 10g,
this may be because the AWE_WINDOW_MEMORY value in the registry is set
too low for the calculated minimum value. If you cannot increase the
value for AWE_WINDOW_MEMORY, then you can explicitly set
_DB_BLOCK_LRU_LATCHES to a value lower than the calculated value, and
retry the startup.
_DB_BLOCK_LRU_LATCHES must be at least 8 (Equal to the maximum number of
buffer pools)
Note #1 - Recall from the earlier section that these errors may also occur if
you are trying to start up with a buffer cache that is too large for the
physical memory available.
Note #2 - The same errors above have also been observed with a buffer
cache that is too small. When USE_INDIRECT_DATA_BUFFERS is set to TRUE
the value for DB_BLOCK_BUFFERS should equate to a buffer cache that is
AT LEAST equal to AWE_WINDOW_MEMORY. In most cases, the total buffer
cache size will be greater than AWE_WINDOW_MEMORY. If you attempt to
start up with a buffer cache that is too small (i.e. < AWE_WINDOW_MEMORY)
that may also result in the ORA-27102 error.
Note#3 - It has been observed on some systems that you may need to add a few
additional meg to AWE_WINDOW_MEMORY to calculate for overhead. Therefore, if
you go through the above calculations, and the instance still does not start,
try adding an additional 10 Meg or so to the calculated value.
Note#4 - Also, keep in mind that when calculating the # of CPU's in the system,
you have to take hyperthreading into account. On a hyperthreaded system, the OS
will think that you have double the # of CPU's in the system over what you actually
have, and this is the number that must be used in the calculations.
How to calculate the maximum used memory
=========================================
With respect to awe_window_memory the following maximum amount of memory can be used
from physical memory:
The SGA size is composed from:
((db_block_buffers * block size) + (shared_pool_size + large_pool_size +
java_pool_size + log_buffers) + 1MB
The size of SGA + Oracle's overhead must not exceed the available virtual memory.
The size of buffer cache depends on the available virual memory and can be calculated with
buffer cache = db_block_buffer * db_block_size
CPU Spins Possible When Using AWE Implementation:
=================================================
Use caution when setting _DB_BLOCK_LRU_LATCHES or AWE_WINDOW_MEMORY too low.
If we are unable to map a requested buffer into the window because all of
the space defined by AWE_WINDOW_MEMORY is in use with buffers already
actively being accessed, then we spin and wait, checking every so often
until an existing buffer in the window can be unmapped, and a new buffer can
be mapped in.
This spin will consume CPU cycles until enough buffers can be
Mapped/Unmapped to satisfy the request. In some cases, there may be so
many buffers needing to be mapped into the window, that DBWR will consume
100% of cycles on all CPUs, effectively locking up the machine. This is
normal behavior under some circumstances, and is simply an indication that
AWE_WINDOW_MEMORY is too small.
Monitoring Mapping Operations in 9.2 and later releases:
========================================================
Starting with 9.2, we have added additional statistics which can be
measured in v$sesstat (sesssion-level stats) and v$sysstat (system-wide
stats):
STATISTIC# NAME
---------- ------------------------------
154 number of map operations
155 number of map misses
This query below will give you system-wide information on map
operations and map misses:
SQL> select * from v$sysstat where statistic# in (154, 155);
If the # of Map misses is relatively high, or particularly of the # of map
misses increases consistently over time, this may be an indication that the
value for AWE_WINDOW_MEMORY is set too low.
Dynamic Memory Management/Automatic Memory Management with AWE Enabled
=============================================================
Oracle10g introduces the concept of Automatic Memory Management,
whereby the Oracle RDBMS will dynamically adjust SGA parameters
such as SHARED_POOL_SIZE, JAVA_POOL_SIZE, DB_CACHE_SIZE, etc.
This is enabled by setting the parameter SGA_TARGET to a non-zero value.
However, in order for this to work properly, you must use DB_CACHE_SIZE
for the buffer cache. When setting USE_INDIRECT_DATA_BUFFERS, you cannot
set DB_CACHE_SIZE, as noted above. Therefore, SGA_TARGET should not be set
when using AWE - these two features are mutally exclusive.
When setting USE_INDIRECT_DATA_BUFFERS=TRUE on Oracle10g, you should also
set SGA_TARGET to 0.
Diagnosing Spins Associated With AWE in 8.1.x:
==============================================
The above stats are not available in 8.1.7, so if you are encountering
problems with CPU spins, with AWE_WINDOW_MEMORY enabled, it is more
difficult to diagnose.
You can start by identifying and monitoring the thread associated with
DBWR via the following query:
SQL> select b.name, p.spid from v$process p, v$bgprocess b
where p.addr=b.paddr;
NAME SPID
----- ---------
PMON 1900
DBW0 1956
LGWR 572
CKPT 1908
SMON 1808
RECO 920
SNP0 1784
SNP1 1892
SNP2 1896
SNP3 1844
10 rows selected.
As you can see, DBWR has an SPID of 1956, which will equate to the
Thread ID of that thread within the Oracle executable. This thread can
then be monitored using Performance Monitor and/or the PSLIST utility,
which is available as a free download from http://www.sysinternals.com
If your monitoring shows that DBWR is consuming excessive CPU, you can
attempt to get an errorstack from that thread using oradebug:
SQL> oradebug setospid 1956
Oracle pid: 3, Windows thread id: 1956, image: ORACLE.EXE
SQL> oradebug unlimit
Statement processed.
SQL> oradebug dump errorstack 3
Statement processed.
This should dump the errorstack to the DBWR trace file, found in BDUMP.
If the errorstack contains the function SKGMMAP, this is an indication
that DBWR is working to map/unmap database block buffers.
Note: In 8.1.7 of the RDBMS, you cannot use DBWR_IO_SLAVES in combination with
USE_INDIRECT_DATA_BUFFERS, due to BUG#3042660/BUG#2215894. You must leave
DBWR_IO_SLAVES at its default value - otherwise, buffers are not unmapped
and eventually a spin of the process will result.
This problem is resolved in 9.2.0.1 - the fix is NOT backported to 8.1.7
KNOWN ISSUES
--------------------
BUG#2461474 - SHOW SGA DOES NOT SHOW CORRECT # OF DB BUFFERS WITH AWE
BUG#1412485 - LONG SHUTDOWN TIME WITH AWE_WINDOW_MEMORY: FIXED IN 8.1.7.1
BUG#1406194 - AWE_WINDOW_MEMORY NOT RELEASED WHEN DB SHUTDOWN: FIXED IN 8.1.7.1
BUG#2520796 - ORA-439 TRYING TO ENABLE VLM IN STANDARD EDITION OF ORACLE - FIXED IN 9.2.0.4
BUG#2945011 - VLM DOES NOT WORK ON STANDARD EDITION ORACLE 9.2.0.2 ON WINDOWS - FIXED IN 9.2.0.4
BUG#3120033 - ORA-600[KCBVMAP] may occur with AWE, or DBWR may crash with ORA-471 on 9.2.0.4 - FIXED 9.2.0.4 PATCH 2
BUG#3042660 / BUG#2215894 - IO SLAVES DON'T UNMAP BUFFERS ON LINUX IN VLM MODE (APPLIES TO WINDOWS AS WELL)
No comments:
Post a Comment