Checklist for Slow Performance of Export Data

 

 

Modified24-JUN-2010     TypeBULLETIN     StatusPUBLISHED

 

In this Document
  Purpose   

Scope and Application  

 Checklist for Slow Performance of Export Data Pump (expdp) and Import DataPump (impdp)    

  INTRODUCTION  

    PARAMETERS    

  CHECK ACTIVITY OF DATA PUMP   

   KNOWN DEFECTS OVERVIEW   

   DEFECT DETAILS   References

Applies to:


Oracle

Server - Enterprise Edition - Version: 10.1.0.2 to 11.1.0.7 - Release: 10.1 to 11.1


Oracle Server - Standard Edition - Version: 10.1.0.2 to 11.1.0.7   [Release: 10.1 to 11.1]


Oracle Server - Personal Edition - Version: 10.1.0.2 to 11.1.0.7   [Release: 10.1 to 11.1]


Enterprise Manager for RDBMS - Version: 10.1.0.2 to 11.1.0.7   [Release: 10.1 to 11.1]


Oracle Server - Enterprise Edition - Version: 10.1.0.2 to 11.1.0.7   [Release: 10.1 to 11.1]


Information in this document applies to any platform.


Oracle Server Enterprise Edition - Version: 10.1.0.2 to 11.1.0.6

Purpose

This document provides information about possible causes of performance related problems when using export DataPump and Import DataPump to transfer data from an Oracle database.

Scope and Application

The article is intended for users of the Oracle10g and Oracle11g database who use the Export Data Pump utility to export data from an Oracle source database and the Import Data Pump utility to import into an Oracle target database. This document is only applicable to the new clients Export Data Pump (expdp) and Import Data Pump (impdp) and does not apply to the original export (exp) and import (imp) clients. For Oracle10g and higher, we recommend the usage the Data Pump to transfer data between Oracle databases.

Checklist for Slow Performance of Export Data Pump (expdp) and Import DataPump (impdp)

INTRODUCTION

Starting with release 10g (10.1.0), Oracle introduced the new Oracle Data Pump technology, which enables very high-speed movement of data and metadata from one database to another. This technology is the basis for Oracle's new data movement utilities, Data Pump Export and Data Pump Import.

Under certain circumstances, a performance problem may be seen when unloading or loading data with the Data Pump clients. This document will provide details about setup and configuration settings that may have an impact on the performance of the Data Pump clients; will provide details how to check what Data Pump is doing at a specific moment; and will discuss some known defects that have an impact on the performance.

PARAMETERS

In this section, the Data Pump parameters are listed that may have an impact on the performance of an Export DataPump or import DataPump job. There are also some generic database parameters (init.ora / spfile) listed that are known to have a possible impact of the Data Pump jobs.
If you experience and need to resolve a Data Pump performance issue, and one or more of following parameters are used for the job, then first check the remarks below and see whether this performance problem reproduces if the parameter is not used, or used differently.

  1. Data Pump parameter: PARALLEL
    ...
    For details, see also:
    Note:365459.1 "Parallel Capabilities of Oracle Data Pump"
    .
  2. Data Pump parameter: DUMPFILE
    ...
    .
  3. Export Data Pump parameter: ESTIMATE
    ...
    For details about the Export Data Pump parameter ESTIMATE, see also:
    Note.786165.1"Understanding the ESTIMATE and ESTIMATE_ONLY parameter in Export DataPump"
    .
  4. Export Data Pump parameters: FLASHBACK_SCN and FLASHBACK_TIME
    ...
    .
  5. Import Data Pump parameter: TABLE_EXISTS_ACTION
    ...
    .
  6. Import Data Pump parameters: REMAP_SCHEMA or REMAP_TABLESPACE
    ...
    For details related to this issue, see also the section "Defects Details" below, and:
    Note:429846.1"Slow Data Pump with REMAP_SCHEMA and REMAP_TABLESPACE parameters"
    .
  7. Database parameter: CURSOR_SHARING
    ...
    For details related to this issue, see also the section "Defects Details" below, and:
    Note:94036.1"Init.ora Parameter "CURSOR_SHARING" Reference Note"
    Note:421441.1"Datapump Import With dblink Going Slow With cursor_sharing Set to 'force'"

CHECK ACTIVITY OF DATA PUMP

...

KNOWN DEFECTS OVERVIEW

Below an overview of known performance related defects in the various Oracle10g and Orace11g releases. See the next section after the overview for details about these defects and possible workarounds.

Note 1: besides a Data Pump specific defect, there may also be a defect in a different area such as an optimizer related defect, which also has an impact on the performance during a Data Pump job. Only defects with highest impact have been listed below.

Note 2: Defects that have an impact on the performance of Export Data Pump, will also have an impact on import Data Pump when import is done with the NETWORK_LINK parameter specified. Those defects are listed only once at the Export Data Pump section.


Export DataPump (expdp):
10.1.0.1.0  to  10.1.0.3.0
 - Bug 3447032 - Import Data Pump is slow when importing statistics
 - Bug:4513695- Poor performance for SELECT with ROWNUM=1 with literal replacement
 - Bug 5095025 - Export Data Pump runs out of memory (ORA-4030) when exporting many schema's 
 -  Bug:5464834- Export Data Pump runs out of memory (ORA-4030) when many tables are involved
 - Bug:5590185- Consistent Export Data Pump is slow when exporting row data
 - Bug:5928639- Export Data Pump of table can be very slow if CURSOR_SHARING is not EXACT
 - Bug 5929373 - Export Data Pump of a table can be very slow if database has many user tables

10.1.0.4.0  to  10.1.0.5.0  and  10.2.0.1.0  to 10.2.0.3.0
 - Bug:4513695- Poor performance for SELECT with ROWNUM=1 with literal replacement 
 - Bug 5095025 - Export Data Pump runs out of memory (ORA-4030) when exporting many schema's
 - Bug:5464834- Export Data Pump runs out of memory (ORA-4030) when many tables are involved
 - Bug:5590185- Consistent Export Data Pump is slow when exporting row data
 - Bug:5928639- Export Data Pump of table can be very slow if CURSOR_SHARING is not EXACT
 - Bug 5929373 - Export Data Pump of a table can be very slow if database has many user tables
- Bug 5573425- Slow Datapump with wrong results due to subquery unnesting and complex view

10.2.0.4.0
- Bug 7413726- Poor EXPDP performance when db COMPATIBLE=10.2.0.3 or 10.2.0.4 (duplicate of  Bug 7710931)
- Bug 7710931- DataPump export is extremely slow when extracting schema
- Bug 6460304- (affects earlier versions as well) Expdp domain index dump using RULE Optimizer and slow 
 
11.1.0.6.0
- Bug 7585314- OCSSD.BIN consumes much too much CPU while running Datapump

11.1.0.7.0
- Bug 8363441- Very Expensive Sql Statement During Datapump Import With Many Subpartitions



Import DataPump (impdp):

10.1.0.1.0  to  10.1.0.3.0
 - Bug 3447032 - Import Data Pump is slow when importing statistics
 - Bug:5292551- Import Data Pump runs out of memory (ORA-04030) and can be very slow on certain tables 
 - Bug 5555463 - Import Data Pump can be slow when importing small LOBs in External Table mode

10.1.0.4.0
 - Bug:5292551- Import Data Pump runs out of memory (ORA-04030) and can be very slow on certain tables
 - Bug 5555463 - Import Data Pump can be slow when importing small LOBs in External Table mode

10.1.0.5.0
 - Bug 3508675 - Import Data Pump is slow when importing TABLE_DATA
 - Bug:5292551- Import Data Pump runs out of memory (ORA-04030) and can be very slow on certain tables
 - Bug 5555463 - Import Data Pump can be slow when importing small LOBs in External Table mode

10.2.0.1.0  to  10.2.0.3.0
 - Bug:5071931- Import Data Pump with REMAP_SCHEMA and REMAP_TABLESPACE is slow
 - Bug:5292551- Import Data Pump runs out of memory (ORA-04030) and can be very slow on certain tables 
 - Bug 6989875-Transportable Tablespace Import Spins Using CPU
 - Bug 5555463 - Import Data Pump can be slow when importing small LOBs in External Table mode

10.2.0.4.0  - Bug 7439689- (affects earlier versions as well) Impdp workeer process spinning on MERGE statement

11.1.0.6.0
 - Bug 7585314- OCSSD.BIN consumes much too much CPU while running Datapump

11.1.0.7.0
- Bug 8363441- Very Expensive Sql Statement During Datapump Import With Many Subpartitions


 

DEFECT DETAILS

  1. Bug 3447032 - Import Data Pump is slow when importing statistics
    -Defect: Bug 3447032 "DBMS_STATS.SET_COLUMN_STATS can be slow (affects IMPORT)" (not a public bug)
    -Symptoms: an Import (original client) or Import Data Pump job may show long wait times when importing INDEX_STATISTICS or TABLE_STATISTICS
    -Releases: 10.1.0.3.0 and lower
    -Fixed in: 10.1.0.4.0 and higher; for some platforms a fix on top of 10.1.0.3.0 is available withPatch:3447032 -Patched files: exuazo.o  kustat.xsl
    -Workaround: exclude import of statistics (EXCLUDE=statistics) and manually create the statistics after the import completes
    -Cause: issue how column statistics are set on tables with (many) sub-partitions
    -Trace: SQL trace shows references to DBMS_STATS package
    -Remarks: the fix for this bug has to be applied at both sites (source and target database) and any Export or Export Data Pump dumpfile has to be regenerated to get improved performance upon import.
    .
  2. Bug 3508675 - Import Data Pump is slow when importing TABLE_DATA
    -Defect:  Bug 3508675 "APPSST10G: BAD PLAN WHEN QUERYING ALL_POLICIES WHEN IMPORTING TABLE_DATA" (not a public bug)
    -Symptoms:  an impdp job may show high CPU usage and a slow down during the import phase of: TABLE_DATA
    -Releases:  10.1.0.5.0
    -Fixed in:  10.2.0.1.0 and higher; generic fix available for 10.1.0.5.0 withPatch:3508675 -Patched files:  prvtbpdi.plb
    -Workaround:  none
    -Cause:  introduced with fix for Bug 3369744 ALL_SYNONYMS view does not show synonym for a synonym (not a public bug)
    -Trace:  SQL trace and AWR trace show high CPU usage and execution time for query:
    SELECT count(*) FROM ALL_POLICIES WHERE enable = :y and ins = :y2 and object_name = :tname and object_owner = :sname
    -Remarks:  may show up during impdp job of Oracle Applications database (apps) or any other target database where many tables are imported.
    .
  3. Bug 4513695 - Export Data Pump of large table can be very slow when CURSOR_SHARING=SIMILAR
    -Defect:  Bug:4513695"Poor performance for SELECT with ROWNUM=1 with literal replacement"
    -Symptoms:  an export Data Pump job of a large table (100+ Gb) can be much slower (e.g. 24+ hours) than an export with the original exp client
    -Releases:  10.1.0.x and 10.2.0.3.0 and lower
    -Fixed in:  10.2.0.4.0 and higher; for some platforms a fix on top of 10.2.0.3.0 is available withPatch:5481520 
    -Patched files:  apa.o kko.o kkofkr.o qerco.o
    -Workaround:  if possible, set CURSOR_SHARING=EXACT before starting the export Data Pump job
    -Cause:  query optimization issue in Cost Base Optimizer (CBO) when cursor_sharing is set to similar
    -Trace:  Data Pump Worker trace shows very high elapsed fetch time for: "SELECT NVL((SELECT /*+ NESTED_TABLE_GET_REFS */ :"SYS_B_0" FROM ... WHERE ROWNUM = :"SYS_B_1"), :"SYS_B_2") FROM DUAL"
    -Remarks:  a fix for this defect can only be provided as a fix forBug:5481520"Wrong results with ROWNUM and bind peeking".
    .
  4. Bug 5071931 - Import Data Pump with REMAP_SCHEMA and REMAP_TABLESPACE is slow
    -Defect: Bug:5071931"DATAPUMP IMPORT WITH REMAP TABLESPACE, AND SCHEMA IS VERY SLOW"
    -Symptoms:  an impdp job with REMAP_SCHEMA and REMAP_TABLESPACE slows down during the import phase of DDL such as: TABLE, INDEX, OBJECT_GRANT
    -Releases:  10.2.0.1.0 to 10.2.0.3.0
    -Fixed in:  10.2.0.4.0 and higher; a generic fix available for 10.2.0.3.0 withPatch:5071931and for some platforms a fix on top of lower releases is also available with the same number
    -Patched files:  prvtmeti.plb
    -Workaround:  if not required, do not use the REMAP_% parameters
    -Cause:  problem when multiple transforms are chained together
    -Trace:  Data Pump Worker trace shows high elapsed times between "DBMS_METADATA.CONVERT called" and "DBMS_METADATA.CONVERT returned"
    -Remarks:  this defect does not reproduce in Oracle10g Release 1; for details, see also:
    Note:429846.1"Slow Data Pump with REMAP_SCHEMA and REMAP_TABLESPACE parameters".
    .
  5. Bug 5095025 - Export Data Pump runs out of memory (ORA-4030) when exporting many schema's
    -Defect:  Bug 5095025 "ORA-4030 (KXS-HEAP-C,TEMPORARY MEMORY) USING EXPDP" (not a public bug)
    -Symptoms: a schema level expdp job of many schema's (like 50+) and where procedural objects are involved (like schema jobs), may fail due to running out of PGA (leaking memory) when exporting procedural objects
    -Releases:  10.1.0.x and 10.2.0.3.0 and lower
    -Fixed in:  10.2.0.4.0 and higher
    -Patched files:  (in patchset)
    -Workaround:  if possible, run multiple export Data Pump jobs so every job has fewer schema's to export
    -Cause:  query optimization (Rule Based Optimizer (RBO) instead of Cost Base Optimizer (CBO))
    -Trace:  ORA-4030 and Data Pump Worker trace may show reference to: "SELECT /*+rule*/ SYS_XMLGEN(VALUE(KU$), XMLFORMAT.createFormat2('PROCDEPOBJ_T', ..."
    -Remarks:  also related to this defect are:Bug:5464834andBug:5928639and Bug 5929373 (not a public bug). 
    .
  6. Bug 5292551 - Import Data Pump runs out of memory (ORA-04030) and can be very slow on certain tables
    -Defect: Bug:5292551"IMPDP VERY SLOW WHEN IMPORTING A TABLE WITH INITIALIZED COLUMN OF TYPE VARRAY"
    -Symptoms:  an impdp job of specific tables (like tables with Spatial data MDSYS.SDO_GEOMETRY) can be very slow when importing table data and the Data Pump worker process shows a continuous increase of memory when loading those tables
    -Releases:  10.1.0.x and 10.2.0.3.0 and lower
    -Fixed in:  10.2.0.4.0 and higher; for some platforms a fix on top of 10.2.0.3.0 is available withPatch:5292551 -Patched files:  kpudp.o
    -Workaround:  if possible, exclude those tables: EXCLUDE=TABLE:"in('TAB_NAME', ...) and import those tables separately in a second table level import Data Pump job: TABLES=owner.tab_name
    -Cause:  memory was not released, resulting in high amount of allocated memory
    -Trace:  Heapdumps show many freeable chunks 'freeable assoc with marc' or 'klcalh:ld_hds'
    -Remarks:  the impdp job may fail after running for days with errors such as ORA-4030 (out of process memory when trying to allocate xxx bytes) or or ORA-31626 (job does not exist) or internal error ORA-00600 [729], [12432], [space leak].
    .
  7. Bug 5464834 - Export Data Pump runs out of memory (ORA-4030) when many tables are involved
    -Defect:  Bug:5464834"ORA-4030 (KXS-HEAP-C,TEMPORARY MEMORY) USING EXPDP"
    -Symptoms:  a table level expdp job of many tables (like 250+) may fail due to running out of PGA (leaking memory) when exporting table data
    -Releases:  10.1.0.x and 10.2.0.3.0 and lower
    -Fixed in:  10.2.0.4.0 and higher; generic fix available for 10.1.0.4.0 and 10.2.0.3.0 withPatch:5464834 -Patched files:  catmeta.sql  prvtmeti.plb
    -Workaround:  if possible, run multiple export Data Pump jobs so every job has fewer tables to export
    -Cause:  query optimization (Rule Based Optimizer (RBO) instead of Cost Base Optimizer (CBO))
    -Trace:  ORA-4030 and Data Pump Worker trace may show reference to: "SELECT /*+rule*/ SYS_XMLGEN(VALUE(KU$), XMLFORMAT.createFormat2('TABLE_DATA_T', ..."
    -Remarks:  also related to this defect are: Bug 5095025 (not a public bug) andBug:5928639and Bug 5929373 (not a public bug).
    .
  8. Bug 5555463 - Import Data Pump can be slow when importing small LOBs (under 256K)
    -Defect:  Bug 5555463 "PERFORMANCE ISSUES FOR DATAPUMP IMPORT/EXTERNAL_TABLE MODE OF TABLES WITH LOBS" (not a public bug)
    -Symptoms:  slow performance, high CPU usage, and LOB redo generation when importing table with small LOBs (LOBs smaller than 256 kb)
    -Releases:  10.1.0.x and 10.2.0.3.0 and lower
    -Fixed in:  10.2.0.4.0 and higher
    -Patched files:  (in patchset)
    -Workaround:  none (if possible, run load in Direct Path mode: ACCESS_METHOD=DIRECT_PATH)
    -Cause:  using temporary LOBs when loading data in External Table mode
    -Trace:  (details not available) 
    -Remarks:  an impdp job of the same table data in Direct Path mode shows much faster performance.
    .
  9. Bug 5590185 - Consistent Export Data Pump is slow when exporting row data
    -Defect:  Bug:5590185"CONSISTENT EXPORT DATA PUMP JOB (FLASHBACK_TIME) HAS SLOWER PERFORMANCE"
    -Symptoms:  an expdp job of large amount of tables is slow when using FLASHBACK_TIME or FLASHBACK_SCN or when logical standby or Streams are used
    -Releases:  10.1.0.x and 10.2.0.3.0 and lower
    -Fixed in:  10.2.0.4.0 and higher; for some platforms a fix on top of 10.2.0.2.0 is available withPatch:5590185 -Patched files:  prvtbpm.plb
    -Workaround:  if not required, do not run a consistent Export Data Pump job
    -Cause:  full table scans on DataPump's Master table
    -Trace:  SQL trace shows execution time for statement:
    UPDATE "SYSTEM"."SYS_EXPORT_SCHEMA_01" SET scn = :1, flags = :2 WHERE (object_path_seqno = :3) AND (base_process_order = :4) AND (process_order > 0)
    -Remarks:  If a normal expdp job takes 1 hour, then the same job but now consistent, may take more than 8 hours.
    .
  10. Bug 5928639 - Export Data Pump can be very slow if CURSOR_SHARING is not EXACT
    -Defect:  Bug:5928639"DATAPUMP EXPORT SLOW WHEN CURSOR_SHARING is not EXACT"
    -Symptoms:  an export Data Pump job can be slow if many tables are involved and init.ora or spfile parameter CURSOR_SHARING is not set to EXACT
    -Releases:  10.1.0.x and 10.2.0.3.0 and lower
    -Fixed in:  10.2.0.4.0 and higher with fix forBug:5464834(see above)
    -Patched files:  catmeta.sql prvtmeti.plb
    -Workaround:  set spfile parameter CURSOR_SHARING=EXACT
    -Cause:  query optimization (Rule Based Optimizer (RBO) instead of Cost Base Optimizer (CBO))
    -Trace:  Worker trace file shows high waits for DBMS_METADATA.FETCH_XML_CLOB called, and SQL trace file shows reference to: "SELECT /*+rule*/ SYS_XMLGEN(VALUE(KU$), XMLFORMAT.createFormat2('TABLE_DATA_T', ..."
    -Remarks:  also related to this defect are: Bug 5095025 (not a public bug) andBug:5464834and Bug 5929373 (not a public bug).
    .
  11. Bug 5929373 - Export Data Pump of a table can be very slow if database has many user tables
    -Defect:  Bug 5929373 "APPS ST GSI - DATA PUMP TAKES LONG TIME TO EXPORT DATA" (not a public bug)
    -Symptoms:  an export Data Pump job of a small table can be slow if database has many user tables
    -Releases:  10.1.0.x and 10.2.0.3.0 and lower
    -Fixed in:  10.2.0.4.0 and higher with fix forBug:5464834(see above)
    -Patched files:  catmeta.sql prvtmeti.plb
    -Workaround:  none
    -Cause:  query optimization (Rule Based Optimizer (RBO) instead of Cost Base Optimizer (CBO))
    -Trace:  Worker trace file shows high waits for DBMS_METADATA.FETCH_XML_CLOB called, and SQL trace file shows reference to: "SELECT /*+rule*/ SYS_XMLGEN(VALUE(KU$), XMLFORMAT.createFormat2('TABLE_DATA_T', ..."
    -Remarks:  Data Pump may need more than an hour for the table, while the original export client finishes in a couple of minutes; also related to this defect are: Bug 5095025 (not a public bug) andBug:5464834andBug:5928639.

References


BUG:4513695- SELECT WITH ROWNUM=1 PERFORMANCE IS TOO LATE USING CURSOR_SHARING=SIMILAR


BUG:5071931- DATAPUMP IMPORT WITH REMAP TABLESPACE, AND SCHEMA IS VERY SLOW


BUG:5292551- IMPDP VERY SLOW WHEN IMPORTING A TABLE WITH INITIALIZED COLUMN OF TYPE VARRAY


BUG:5464834- ORA-4030 (KXS-HEAP-C,TEMPORARY MEMORY) USING EXPDP


BUG:5481520- WRONG RESULTS WITH ROWNUM AND BIND PEEKING


BUG:5590185- CONSISTENT EXPORT DATA PUMP JOB (FLASHBACK_TIME) HAS SLOWER PERFORMANCE


BUG:5928639- DATAPUMP EXPORT SLOW WHEN CURSOR_SHARING != EXACT


BUG:6989875- TRANSPORTABLE TABLESPACE IMPORT SPINS USING CPU


BUG:7413726- POOR EXPDP PERFORMANCE WHEN DB COMPATIBLE=10.2.0.3 OR 10.2.0.4


BUG:7710931- DATAPUMP EXPORT IS EXTREMELY SLOW WHEN EXTRACTING SCHEMA


BUG:7722575- DATAPUMP VIEW KU$_NTABLE_DATA_VIEW CAUSES POOR PLAN / SLOW EXPDP


NOTE:155477.1- Parameter DIRECT: Conventional Path Export Versus Direct Path Export


NOTE:223730.1- Automatic PGA Memory Management


NOTE:277905.1- Export/Import DataPump Parameter TABLES - How to Export and Import Tables Residing in Different Schemas


NOTE:286496.1- Export/Import DataPump Parameter TRACE - How to Diagnose Oracle Data Pump


NOTE:331221.1- 10g Export/Import Process for Oracle Applications Release 11i


NOTE:362205.1- 10g Release 2 Export/Import Process for Oracle Applications Release 11i


NOTE:365459.1- Parallel Capabilities of Oracle Data Pump


NOTE:421441.1- Datapump Import With dblink Going Slow With cursor_sharing Set to 'force'


NOTE:429846.1- Slow Data Pump with REMAP_SCHEMA and REMAP_TABLESPACE parameters


NOTE:762160.1- Impdp will hang when using paramters TRANSPORT_DATAFILES & REMAP_DATAFILE


NOTE:786165.1- Understanding the ESTIMATE and ESTIMATE_ONLY Parameters in Export DataPump


NOTE:94036.1- Init.ora Parameter "CURSOR_SHARING" Reference Note


BUG:8363441- VERY EXPENSIVE SQL STATEMENT DURING DATAPUMP IMPORT WITH MANY SUBPARTITIONS



Related

Products

  • Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition
  • Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Standard Edition
  • Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Personal Edition
  • Enterprise Management > Enterprise Manager Consoles, Packs, and Plugins > Managing Databases using Enterprise Manager > Enterprise Manager for RDBMS
  • Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition
  • Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Personal Edition
  • Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Standard Edition

Keywords

HIGH~CPU~USAGE; HANGING; DATAPUMP; RBO; LEAK; SPACE~LEAK

Errors

ORA-600[729]; ORA-31626; ORA-4030