Release Notes 2023.7.x

Release Highlights

In the 2023.7.0 release, Incorta continues to enhance its platform and user experience by introducing multiple new features and significant enhancements.

• Schema managers can now create load plans with multiple schemas that can be executed in parallel or sequentially via load groups. To make load plans easier to manage, there is also a preview feature that shows the execution dependencies within a load group.
• Incorta Cloud platform now supports a multi-loader architecture where multiple loader services can manage data acquisition and ingestion in a cluster.
• Incorta has recently introduced dynamic fields. This feature enables dashboard developers to specify a range of measures, which dashboard viewers can then utilize to control the display of charts in the dashboard.
• Incorta has expanded its data delivery capabilities by including Google BigQuery and Microsoft Azure Synapse Analytics as data destinations. As a result, the integration between complex data sources and the new cloud data destinations is seamless.
• A new version of the Data Lineage Viewer now tracks and identifies both the upstream and downstream lineage of most system entities, including tables, views, variables, dashboards, and insights.
• Incorta continues to grow its native connector repository by adding three new data connectors.

The 2023.7.1, 2023.7.2, and 2023.7.3 maintenance packs include additional fixes and enhancements.

Upgrade considerations

Metadata database check

Upgrading to 2023.7.x requires upgrading the metadata database to support multi-group load plans and migrate existing schema load jobs or load plans. Before upgrading to a 2023.7.x release, contact Incorta Support to perform a database check to inspect and detect any issues with your metadata database that might cause the metadata database upgrade to fail. If issues are detected, Incorta Support will run scripts against the metadata database to delete the records causing these issues.

Public API v2

• Starting with 2023.7.0, all Incorta Public API v2 endpoints require defining the tenant’s name (case-sensitive) in the request URL.
• The response of the /schema/{schemaName}/list endpoint has been updated. Some parameters have been renamed while additional parameters are now available to display error messages, the source column of runtime business view columns, and their data types. For more details, refer to Public API v2 → List Schema Objects.

Columns with Integer data type

In this release, Incorta can write columns with Integer data type as Integer (instead of Long) in Parquet files. Loading objects with Integer columns from source (full load) creates new Parquet files with Integer data and requires a full load for all their dependent objects. You can instruct the Loader Service to create new Parquet files and migrate these columns to Integer during loading from staging. For more details, see Maintaining Integer data during data loading.

New Features

Data Management Layer

• Expanded Data Delivery capabilities
• Oracle Configure, Price, and Quote connector
• Oracle Transportation Management connector
• Autoline connector
• Data lineage enhancements

Architecture and Application Layer

• Multi-schema load plan enhancements
• Multi-loader support
• Maintaining Integer data during data loading
• Detecting duplicates during unique index calculations

Dashboards, Visualizations, and Analytics

• Null handling enhancements
• Dynamic fields applied in measures
• List of Values in presentation variables
• Hide prompts from the filter bar
• Grouping measure color palette for pie and donut visualizations
• Upload Components to Marketplace
• Limit prompt filters selection in dashboards

Cluster Management Console

• Enhanced logging framework
• Announcements banner
• Limit downloading insights enhancement

Expanded Data Delivery capabilities

Incorta now has Google BigQuery and Microsoft Azure Synapse Analytics destinations for data delivery. Our data destinations feature allows for the ingestion and enrichment of data and the transfer of full and incremental loads to other cloud analytics platforms. For more information, refer to the Incorta Data Delivery configuration documents.

Oracle Configure, Price, and Quote connector

With the addition of the Oracle CPQ REST API connector, Incorta now allows you to connect to your Oracle CPQ system seamlessly. By utilizing this connector, you can collect valuable data and gain meaningful insights that will help you optimize your opportunity-to-quote-to-order process.

For more information, refer to Connectors → Oracle CPQ.

Oracle Transportation and Global Trade Management connector

With Incorta, you can now easily connect to your Oracle CPQ system using the Oracle OTM/GTM connector, which is based on REST APIs. This connector allows you to collect data on all your transportation activities across your global supply chain and trading.

For more information, refer to Connectors → Oracle OTM/GTM.

Autoline connector

With the new Autoline connector, you can easily access your Autoline suite and directly query your data. This connector uses the cdata.jdbc.jdbcodbc.jar driver to establish a seamless connection to Autoline.

For more information, refer to Connectors → Autoline.

Data lineage enhancements

The Data Lineage Viewer now displays an entity's (columns, variables, objects…) upstream and downstream lineage. Upstream lineage lists the entities referenced in the current entity, while downstream lineage lists entities where the current entity is referenced. The tool has an enhanced diagram that shows the hierarchy of dependencies where you can track upstream and downstream lineage.

In addition to tracking columns and formula columns, you can also track the lineage of other entities like:

• Physical schema tables
• Business runtime views
• Session and global variables
• Dashboards and insights

Incorta has improved the accessibility of data lineage in the platform. You can now:

• Quickly identify lineage object types through improved color-coding and icon-coding.
• Access dashboards from within the lineage diagram.

The new version of the Data Lineage Viewer is a preview but removes a few previously available functionalities. Deprecated functionalities are:

• Displaying the lineage in a tabular view.
• Downloading the lineage details as CSV.
• Searching dependencies.

For more details, refer to Tools → Data Lineage Viewer v2.

Multi-schema load plan enhancements

The Multi-schema load plan is now a general availability (GA) feature and aims to simplify schema load scheduling and the management of load.

• You can now organize schemas into load groups, simplifying the management of complex data applications that involve multiple schemas. Schemas can be loaded sequentially by ordering them by group. Within each group, schemas will load simultaneously (as resources allow), ensuring faster data refreshing cycles and maintaining dependencies between schemas. For more details, refer to Tools → Scheduler.
• You can also preview the execution plan of load plans using a directed acyclic graph (DAG) that displays the load plan dependencies and order of execution. This feature provides a visual representation of the data loading process.

For more details, refer to Tools → Load Plan DAG Viewer.

Multi-loader support

The Incorta Cloud system can now support a multi-loader architecture, which allows multiple Loader Services to manage the acquisition and ingestion of data within the same cluster. This setup ensures that all loader services have access to the same schema model and file versions, maintaining consistency throughout the process. To determine which loader service should execute a load job, Zookeeper considers various factors, such as the number of running load jobs and the maximum number of concurrent load jobs a loader service can handle simultaneously.

Maintaining Integer data during data loading

In previous releases, Incorta wrote Integer columns in Parquet as Long. However, starting with 2023.7.0, Incorta writes Integer columns in Parquet as Integer for all newly created tables. For previously created tables, Incorta converts Integer columns written as Long in Parquet files to Integer during full load jobs, while Incorta keeps these columns unchanged during incremental loads.

As a result, after performing a full load of a table with Integer columns, it is recommended that you perform a full load of its dependent schema objects to ensure data consistency.

To migrate all Parquet files to have Integer data without fully loading your objects, the administrator can turn on the Enable Parquet Migration at Staging option in the Cluster Management Console (CMC) > Server Configurations > Tuning and perform a load from staging for all your objects with Integer columns.

Detecting duplicates during unique index calculations

In previous releases, when the Enforce Primary Key Constraint option was disabled for physical tables or MVs, and the selected key columns resulted in duplicate key values, unique index calculations would not fail, the first matching value was returned whenever a single value of the key columns was required.

Starting with this release, in such a case, the unique index calculation will fail, and the load job will finish with errors. You must either select key columns that ensure row uniqueness and perform a full load or enable the Enforce Primary Key Constraint option and load tables from staging to have the unique index correctly calculated.

Null handling enhancements

Sorting and filtering data based on physical columns now account for null values.

• When sorting data in an insight, Analyzer table, or Analyzer view in ascending order, null values will be first.
• Filters respect null values when applying filters to insights, dashboards, Analyzer tables, or Analyzer views.

Sorting and filtering based on formula columns do not account for null values for now.

Dynamic fields

A dynamic field is a group of dynamically interchangeable fields. The analyze user, building insights and dashboards, would add multiple measures to a dynamic field from the Manage Dashboard Filters within the dashboard scope.

Note: This feature is only available in interactive dashboards. Currently, sending and downloading a dashboard as PDF respects the selected dynamic field.

For more information, refer to Concepts → Dynamic field and Tools → Dashboard Filters Manager.

List of Values in presentation variables

Presentation variables now accept a user-defined list of values. This feature lets you easily filter data based on manual user input at runtime instead of pre-existing data columns.

When you create a presentation variable, you can choose to manually enter a set of defined values as the source of your presentation variable. The set of values can be a comma or line-separated list entered into the bulk edit of the filter values.

For more information, refer to Concepts → Presentation variable.

Hide prompts from the filter bar

This release introduces a new option that will hide prompts from the dashboard filter bar. The Hide from filter bar option is now available for prompts. This option can be enabled while setting up the prompt in the Dashboard Filters Manager. Incorta will apply the prompt to your dashboard regardless of enabling this option.

Note: The Clear All option clears all shown and hidden prompts.

For more information, refer to Concepts → Prompts and Tools → Dashboard Filters Manager.

Grouping color palette for pie and donut visualizations

Additional configurations are available to control the colors of donut and pie charts to align them with dashboard themes. The Format Color Palette option is now available for grouping dimensions in pie and donut visualizations.

Upload Components to Marketplace

In this release, Incorta enables uploading custom-developed components to the external components list.

For more information, refer to the Component SDK documentation.

Limit prompt filters selection in dashboards

In this release, Incorta is enabling Cluster Management Console (CMC) admin to limit the number of selections in a dashboard prompt filter that a dashboard consumer can select to apply on a dashboard via the new Max no. of selections for contains filter configuration in the CMC that you can find under Default Tenant Configuration > Incorta Labs.

Enhanced logging framework

Incorta has made several improvements to its logging framework, including the handling of file naming, file size, and archiving. In this framework, a log file is automatically zipped to incorta-<date>-<file_number>.log.zip when it reaches 1GB or at the end of the day. Incorta then retains the log files for a maximum of 6 months. To access older log files, please contact Incorta Support.

Announcements banner

Incorta can now broadcast announcements via an announcement banner to users within the Incorta platform. The banner announcement can now be set up and sent out through the CMC using the Notifications feature found in the Server Configurations section. This can be done by either the CMC cloud admin or the CMC admin. The banner will display to all tenant users. When users dismiss the notification banner, it will only appear to them again once the admin creates a new announcement.

For more information on configuring the notifications, refer to Guides → Configure Server.

Limit downloading insights enhancement

In this release, Incorta renamed the Users with “User” or “Individual Analyzer” roles can download insights option previously introduced in 2023.4.1 to be “Download insights”. Disable this option to prevent users with “User” or “Individual Analyzer” roles from downloading insights. The option is now available for configuration by the CMC cloud admin and CMC cloud admin.

Fixes and Enhancements

In addition to the new features and significant enhancements mentioned above, this release introduces some additional enhancements and fixes that help make Incorta more stable, engaging, and reliable.

Enhancements

Fixes

2023.7.1 Maintenance Pack

Engine Concurrent Service Thread Throttling

A dashboard’s filtering can slow or be unable to render due to all Engine threads being used concurrently for another dashboard. With this resolution, workers will be throttled or increased based on the number of waiting jobs. Contact Incorta Support to edit the engine.properties file that exists in the Analytics Service directory and enable this feature.

Fixes

2023.7.2 Maintenance Pack

Fixes

2023.7.3 Maintenance Pack

In this maintenance pack, Incorta is introducing multiple enhancements to the Oracle Cloud Application connector and Data Agent memory management that aim to enhance performance and stability. In addition to introducing multiple enhancements in the dashboard free-form layout and the donut visualization.

Fixes

Known issues

The following are the known issues in the 2023.7.x releases:

• In the case of dashboards that require a long time to load, saving insights may take longer than expected and the Analyzer will wait for all the dashboard insights to load.
• When filtering the load jobs list by the In Queue status, single-group load plans that are in queue won’t appear on the list; however, they will appear when filtering by the In Progress status.

For all of the known issues and workarounds in Incorta Cloud releases, refer to Known Issues in 2023 Releases.