1. On-premises 6.0
  2. Guides
  3. Concepts
  4. Tools
    1. Alert Editor
    2. Alias Sync Tool
    3. Analyzer
    4. Apache Parquet Merge Tool
    5. Avro Extractor Tool
    6. Business Schema Designer
    7. Business Schema Manager
    8. Cluster Management Console
    9. CMC Logs Manager
    10. CMC Monitoring
    11. CMC Patch Manager
    12. CMC Scheduler
    13. CMC Tenant Manager
    14. Command Line Interface
    15. Content Manager
    16. Dashboard Filters Manager
    17. Dashboard Manager
    18. Data Agent
    19. Data APIs for External Notebooks
    20. Data Lineage Viewer
    21. Data Manager
    22. Formula Builder
    23. Inspector Tool
    24. Join Editor
    25. Load Job Details Viewer
    26. Model Update Viewer
    27. Notebook Editor
    28. Profile Manager
    29. Public API
    30. Query Plan Viewer
    31. Scheduler
    32. Schema Designer
    33. Schema Diagram Viewer
    34. Schema Manager
    35. Schema Wizard
    36. Security Manager
    37. Table Editor
    38. Tenant Management Tool
    39. Upload Service Command Line Interface (CLI)
    40. Versioning Migration Tool
  5. References
  6. Integrations
  7. Data applications
  8. Releases

Tools → Apache Parquet Merge Tool

About the Apache Parquet Merge Tool

Over time, an Incorta table configured for incremental loads can generate hundreds of Apache Parquet files that are small in size. The Apache Parquet Merge tool merges multiple Parquet table increment files into a single table increment file that contains the merged segments. As the merged files are typically about 1GB in size, the result is improved performance for reading Parquet files from Shared Storage.

The Apache Parquet Merge tool runs as part of the Tenant Management tool (TMT).

There are three options for running the Parquet Merge Tool:

  • For a specific Tenant, merge all schema table increments
  • For a specific Tenant, merge all table increments for one or more Schemas
  • For a specific Tenant and Schema, merge one or more table increments

To start using the Parquet Merge Tool, follow these steps:

The tool creates a new version for each table. After merging the files of a table, the tool updates the table record in the metadata database with the new uncommitted version.

Important

After running the Parquet Merge Tool, you must load the respective schemas from staging.

Apache Parquet Merge Tool Input Parameters

The interactive shell script has the following parameters:

ParameterDescription
-clnmMandatory. Specify the cluster name.
-mpMandatory. Specify the Tenant name
-snOptional. Specify the Schema name(s).
-tnOptional. Specify the Table name(s).
-miOptional. Specify the minimum table increments count for the merge. The default is 100. If less than the minimum, the tool will not merge the table increments.
-mtOptional. Specify merge type. The default is append, knowing that available values are: append, readWrite.

For a specific Tenant, to migrate and merge all parquet files of a table under a schema, execute the following specifying a minimum increment that the table should have:

./tmt.sh  -clnm localCluster -mp demo -sn <SchemaName> -tn <TableName> -mi <MinIncrementsNumber> -mt readWrite

View the Apache Parquet Merge Tool Log Files

The Parquet Merge Tool generates a log file, whose name starts with merge-tool.log, in the following CMC installation directory:

/home/incorta/IncortaAnalytics/cmc/tenant

The file contains the Source Segment, Target Segment, and Source Offset. Use this log file to determine the results of the merge activities.

You can also find the .csv report file under cmc\tmt\work.