Release Notes 2021.4.1

Release Highlights

The goal of the Incorta Cloud 2021.4.1 release is to enhance analytical capabilities, data management, security, automation, and performance. To that end, the 2021.4.1 release introduces enhancements to the Schema Manager, Schema Designer, and visualizations. It also introduces new built-in functions, and data connectors for FTP and SFTP. In addition, the release includes an enhanced process to import and synchronize domain users with the Lightweight Directory Access Protocol (LDAP), and simplifies single sign-on (SSO) configuration. The release alleviates the impact of daylight savings time on job schedules through the use of time zones versus GMT offsets.

New Features

Advanced Map visualization clustering and custom shapes

The Advanced Map visualization now supports the following:

Clustering, which allows you to group large numbers of data points together on a map to improve performance and presentation.
A new Custom Shape option that is available as a Geo Role in the Geo Attribute properties panel of the Layer tray. Support for the KML file format has been added in the File System connector to display geographical data.

For more information, refer to Visualizations → Advanced Map.

Sort By for Formula Columns in Aggregated and Pivot Table visualizations

The Sort By option is now available for Formula Columns in the Grouping Dimension tray of Aggregated and Pivot table visualizations.

Y-axes minimum and maximum values for the Combo Dual-Axis visualization

You can now set minimum and maximum values for the y-axes in the Combo Dual-Axis visualization settings. For more information, refer to Visualizations → Combo Dual-Axis.

Time Series Analytic, Conversion, and Boolean built-in functions

The following new built-in functions are now available:

Formula Builder comments

The Formula Builder now supports using multi-line comments as follows:

/* Your multi-
    line comment
    here */

Memory usage optimization in Aggregated Table visualizations

Memory usage has been optimized by having a full garbage collector (GC) in Aggregated tables when the Aggregation property of a measure pill is set to Distinct.

Scheduler support for time zones and running jobs between times

The Scheduler now supports the following:

Time zones instead of GMT offsets, which will alleviate the impact of daylight savings time (DST). You have the option to select a time zone if your scheduled job is not automatically converted to a time zone from a GMT offset.
Scheduling daily jobs between a specific start time and end time, such as 11AM - 5PM, and selecting the recurrence of the load job in minutes or hours within the specified time interval. This feature is available for a tenant user with a Schema Manager role.

For more information, refer to Tools → Scheduler.

SFTP and FTP Data Connectors

New Data Connectors are available for SFTP and FTP which will enable you to connect to data lakes that use the secure and non-secure file transfer protocols. For more information, refer to Connectors → SFTP and Connectors → FTP.

Configure Single Sign-On in the Cluster Management Console

You can now configure the Single Sign-On (SSO) settings through the CMC for each tenant. In addition to the new settings added, additional configurations in the Default Tenant Configurations are required whether you are upgrading your Incorta instance or configuring the SSO for the first time. For more information, refer to Guides → Configure SSO.

LDAP synchronization from the Security Manager

This release introduces an enhanced process to import and synchronize domain users and groups with Incorta Cloud using the LDAP protocol. As a CMC administrator or a Super User, you can access the Security Manager and synchronize domain users, groups, and their relations using a .properties configuration file that maps the LDAP attributes to user and group details in the Incorta metadata database. You can also download a template of this file to help you provide the required information. You can then upload this file to Incorta and have users and groups imported to or updated in the metadata database.

For more information, refer to Tools → Security Manager → Import and synchronize users and groups.

Azure Active Directory Authentication Support for SQLi Connections

Azure Active Directory (AD) authentication is now available when you connect to Incorta via SQLi from third party tools, such as DBVisualizer and Tableau. Here are the steps to enable Azure AD authentication:

Specify the client credentials for the Default Tenant Configuration Here are the steps to specify the required properties for the Default Tenant Configuration:

Sign in to the CMC.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Cluster Configurations.
In the panel tabs, select Default Tenant Configurations.
In the left pane, select Security.
In the right pane, in Authentication Type
- Select Azure AD.
- Enter your Azure AD Client ID.
- Enter your Azure AD Client Secret.
Select Save.

Specify the client credentials for a Tenant Configuration Here are the steps to specify the required properties for a specific tenant:

Sign in to the CMC.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Tenant.
For the given tenant, select Configure.
In the left pane, select Security.
In the right pane, in Authentication Type
- Select Azure AD.
- Enter your Azure AD Client ID.
- Enter your Azure AD Client Secret.
Select Save.

Restart the CMC, Analytics Service, and Loader Service.

For more information on how to obtain the Azure AD Client ID and Client Secret, refer to Quickstart: Register an app in the Microsoft identity platform.

Inspector Tool lineage insight and support for materialized views based on Scala and R

The Inspector tool in the Cluster Management Console (CMC) now supports the following:

A new minified lineage insight that detects dependencies between dashboards, columns, and physical schemas
A tenant that contains materialized views based on Scala and R. To inspect a tenant in the CMC, navigate to Clusters > Tenants, and for a tenant row, select ⋮(vertical ellipsis icon) > Execute inspector now.

For more information, refer to Tools → Inspector Tool and Tools → CMC Scheduler.

Enable the Spark SQL App without restarting the Analytics service

In the CMC Server Configurations for Spark Integration you can now change the Enable SQL App property without restarting the Analytics service.

Schema update enhancements

This release introduces multiple updates and enhancements to the Schema Manager and Schema Designer. These enhancements include the following:

The ability to make multiple changes and then push them in one schema update job
The ability to save changes to a draft or directly publish them into a saved version
Indicators for updates that require loading or validating data
The ability to list running load or update jobs that block other jobs
The ability to follow up with the status of synchronization processes related to an update job

Multiple changes, one update job

In previous releases, whenever you added, modified, or deleted a physical schema object, join relationship, formula column, runtime security filter, or load filter, a schema update job was triggered, which caused overhead on the Loader Service and the available resources.

In this release, as a Schema Manager user, you can make multiple changes to the physical schema and then apply all these changes in one schema update job. Alternatively, you can save these changes to a draft version of the physical schema and apply them later as appropriate.

Physical schema drafts and saved versions

In this release, you can immediately apply the changes you have made to the physical schema using the Save Changes option, or you can keep these changes in a draft version. A draft is created automatically whenever you add, modify, or delete a physical schema object, a formula column, a join relationship, a runtime security filter, or a load filter.

For more details, refer to Tools → Schema Designer → Schema Designer modes.

This release uses indicators to highlight the need for specific user actions, such as loading data after specific updates or validating updates for materialized views and physical schema tables.

Updates that require loading data

Some updates you make to the physical schema objects require loading data to ensure data consistency, either a load from source (full load) or load from staging. When you save these updates to a saved version, the Schema Designer and Schema Manager show multiple indicators of the objects that you must load. These indicators keep showing until you or another user perform the required load or a scheduled load job runs the required load for the related objects.

For more details, refer to Tools → Schema Designer → Saving changes that require data load.

Non-validated updates

As a Schema Manager user, you can update a physical schema table or a materialized view without validating the updates. Whether you save the updates to a draft or a saved version, the object Data Source property in both the Schema Designer and the Table Editor shows that the object has updates that are not validated.

Blocking jobs

When an update job or a load job is in the commit phase or the load and post-load phase, respectively, it blocks update and load jobs on dependent physical schemas. The blocked update jobs stay in the In Queue state until all blocking jobs are completed while the blocked load jobs stay in the Load phase. In this release, as a Schema Manager user, you can view a list of jobs that block an update job in the Schema Manager and the Model Update Viewer. This list shows the blocking physical schema, the type of the blocking job: load or update, and the current status of each blocking job. You can also access the details of the blocking job in the Load Job Viewer or the Model Update Viewer according to the type of the blocking job.

Sync Status on the Model Update Details page

A Sync Status section is added to the Model Update Details page. The Sync Status provides the following information for the selected model update service:

The name of the node to sync
The sync update status for the node

For more information, refer to Tools → Model Update Viewer.

Parquet file read optimization

This release introduces an enhanced mechanism for reading parquet files with duplicate rows. This leads to optimized performance, resource utilization, and memory usage when performing incremental loads and rendering dashboards.

Additional Enhancements and Fixes

Physical Schema

Fixed an issue with key columns being encrypted when you change the column type from key to dimension or measure and then change it back to key
Fixed an issue with interrupting load jobs in which the schema was stuck in the waiting queue and reloading the schema skipped creating a snapshot file
Enhanced the Schema Wizard to match the data types of non-measure Excel formula columns with the right data type
Fixed an issue in which the Performance Optimization toggle is enabled in the Table Designer after the table is created with performance optimization disabled
Fixed an issue with Preview data in the Columns section of the Table Editor in which timestamp values are displayed in epoch format
Fixed an issue with the Load Status in which the Schema Load Status shows Loading while the Table Load Status shows Extracting
Fixed an issue in which loading a table with an encrypted key generates an unknown error
Enhanced the error message resulting from loading schemas that have SQL derived tables with self-joins

Visualizations

Enhanced the Transpose setting of Aggregated Table visualizations to display the column count when Transpose is enabled, and the row count when Transpose is disabled
Fixed an issue with Listing and Aggregated table visualizations with Subtotals enabled in which the subtotal rows are duplicated when the insight is duplicated
Fixed an issue with the Pivot Table visualization in which disabling the Merge Rows setting resulted in the repetition of the column header value
Fixed an issue with the Sankey visualization in which the aggregate filter was not applied and no data was displayed
Enhanced the Line and Line Series visualizations to support the Running Total setting
Fixed an issue with the Treemap visualization in which drill downs created for measures did not appear in the drill down Go To menu
Fixed an issue where subtotals were being displayed when disabled in the insight settings, but enabled in the dimension properties. The insight level setting takes precedence over all dimension level settings.
Fixed an issue in which enabling Include Runtime Filters for a drill down to a dashboard tab causes the dashboard tab to duplicate in the drill down properties
Fixed an issue with the Time Series visualization in which only 1000 records are displayed when the Max Rows Limit is set to more than 1000

Dashboards

Enhanced the format of exported tabular data to CSV files so it would not include commas as thousands separators or currency symbols
Fixed an issue in which the schemaRefreshTime built-in function returns the start time of the last successful job instead of the latest created job when the extraction fails
Fixed an issue with sending dashboards that have multiple insights with no titles to data destinations, where the sending process used to fail

Analyzer

Fixed an issue that caused building header titles for a Pivot or an Aggregated Table to throw errors when the data set or the group array was empty
Fixed an issue with the Tables preview in the visualization canvas in which enabling Total and Subtotal while having multiple columns in the Grouping Dimension tray resulted in an error
Fixed an issue with the formula pills in which the query added in a formula pill used to disappear when moving the pill from one tray to another
Fixed an issue in which the copy button located in the column details of the Data panel did not copy the column name when selected

Data Agent

Enhanced the error messages resulting from exceptions in the DataAgent
Enhanced the Data Agent to support Data Lake connections without manually adding jars
Fixed an issue that caused the schema load status to remain stuck in the Extracting phase

Inspector Tool

Fixed an issue with the Inspector tool in which parsing fails if a session variable has an ampersand (&) in its description

Materialized Views and SQLi

Fixed an issue that caused creating a materialized view from a remote data source to fail
Fixed an issue that caused SQLi to not show data for an alias when the user did not have access to the underlying object
Fixed an issue where SQLi failed to extract large amounts of data or render long-running queries due to a problem with the SQLi shared load balancer
Fixed an issue with SQLi queries that referenced a business schema view formula column that used the lookup function when SQLi was configured to use the Engine port

Miscellaneous

Enhanced the CMC user interface to display the clusters’ nodes in a tabular format instead of a collapsible panel
Fixed an issue in which the application of a security filter generates an error when connecting to Incorta via SQLi
Fixed an issue in which a trailing space or hidden character in a value in the engine.properties file caused reading a wrong value and eventually using the default value
To configure Incorta to send emails using Microsoft Office365, you must ensure your Office365 email is not using the multi-factor authentication method

Known Issues and Workarounds

The following table illustrates the known issues and workarounds in this release:

Known Issue	Workaround
After importing a schema that has a Derived table (Analyzer table or SQL table) with encrypted columns, the loading job will succeed. However, dashboards using the Derived table will fail.	Contact Incorta Cloud Support to edit the Schema Definition and disable the Encryption flag on encrypted columns.
After importing a schema that has a Derived table with a disabled Performance Optimized flag, you cannot enable the flag.	Contact Incorta Cloud Support to edit the Schema Definition and enable the Performance Optimized flag for that Derived table, and then load the schema from Staging.
An SSO application cannot login to Incorta tenant when using upper case letter in the tenant name	Use a lower case tenant name while configuring your SSO application. For example, if your Incorta tenant is called Demo or DEMO, you should use demo (all lower case) while configuring your SSO application.