Guides → Migrate

Schema Developers, Dashboard Developers, Tenant Super Users, System Administrators, and a Cluster Management Console (CMC) Administrator migrate specified objects of a tenant, entire tenants, host files, and CMC configurations to ensure that tenant functionality is maintained between environments.

Types of Environments

Each environment represents a stage in the software development life cycle, from development to production. Migration centers around how to move content from a development stage to production. Dashboards, physical schemas, business schemas, session variables, external data sources, users and groups, tenant configurations, server configurations, and system configuration file are examples of migratable content.

For enterprise deployments of Incorta, there are typically three environments with each environment representing its own stage:

Development and the DEV environment

Schema and dashboard developers create new content in the DEV environment. The DEV environment represents a desired ideal for the business community in production. The DEV environment often has a subset copy of production data. Developers safely explore, learn, demo, develop, and unit test new content in this environment.

User Acceptance Testing and the UAT environment

Functional experts, business users, and quality assurance engineers perform tests in the UAT environment. Functional experts such as product owners, project managers and program managers, oversee the technical side of development and help ensure that specifications are met. Quality assurance engineers administer performance tests to measure optimizations. Business users know exactly what the outcome should look like. Together, these users test content in UAT to determine its acceptance for production. UAT may have a subset or copy of production data.

Production and the PROD environment

Having passed UAT, new content migrates to production in the PROD environment. Administrators and Super Users migrate content from UAT to PROD. In production, ad-hoc changes are often highly regulated or prohibited. Real-world users are dashboard consumers. Dashboard consumers personalize, filter, bookmark, and share dashboards in production.

Recommended Environment Configuration

There are ideal configurations for each of the deployment environments. When you use and maintain these configurations, you will mitigate failures during migration.

Incorta Version

You should maintain the same Incorta version across all environments. This guide will assume that the Incorta version for all of your environments is the same.

Migration Path

The ideal environment migration path is DEV → UAT → PROD. It is important to maintain consistent functionality between environments. There may be use cases where it is necessary to migrate changes from PROD back to UAT and DEV, but these cases should be minimal. An example use case is a dashboard change made directly in PROD that is migrated back to UAT and DEV. Permissible operations by environment and user role may differ by organization, and require that you establish clear change management guidelines.

A change management process should also require the following for every change:

A test case, proven test results, and user acceptance
An impact analysis, or comparison between the UAT and PROD environments

Tenant Backups

You should export a tenant backup in the DEV, UAT and PROD environments on a regular basis. You can schedule these backups to occur with a utility such as crontab. Tenant backups allow you to revert to an older, working version of the environment in the event that an issue is introduced during development or migration.

Group and User Permissions

User and group permissions are a useful way to mitigate unplanned changes in the PROD tenant and maintain data security.

In the DEV and UAT environments, group level permissions should be used to share content. Since permissions are not migrated along with content, it is simpler to manage group level permissions than individual level permissions.

In the PROD environment, primarily Administrators or Super Users should have Can Edit permissions. User accessibility use cases should be tested in the UAT environment before migrating to the PROD environment. Restrict edit controls in the PROD environment to encourage the practice of changing the Incorta tenant in DEV, and migrating the changes to UAT and PROD.

Note

If you are using Lightweight Directory Access Protocol (LDAP) or Single Sign-On (SSO), group and user integration will need to be performed on the UAT and PROD environments. Refer to Secure Login Access for LDAP and SSO configurations.

Prerequisites for Migration

Run the Inspector Tool

The Inspector tool checks the lineage references of Incorta metadata objects including tables, schemas, business schemas, business schema views, dashboards, and session variables. For the purposes of migration, the CMC administrator should run the Inspector tool before and after a migration. The Inspector Tool will identify broken references, unused objects, and other issues that may result from a migration. Refer to the Inspector Tool documentation for additional information on how to run the tool and view results.

Run the Formula Validation Tool

The Formula Validation tool identifies issues with the formula expression syntax in dashboards, business schemas, and schemas. In order to validate the formula expression syntax, the tool requires a tenant export file. The CMC Administrator can create the tenant export using the Cluster Management Console (CMC) or the System Administrator can create the tenant export using the Tenant Management Tool (TMT). For the purposes of migration, the System Administrator should run the Formula Validation tool before and after a migration. Refer to the Formula Validation Tool documentation for additional information on how to run the tool and view results.

Types of Migration

Migrate as a Developer

Developers typically migrate specific tenant objects, which is the simplest form of migration. Ideally use specified object migration in late phase development or in a continuous integration continuous deployment (CICD), or similar process. Also use it in early phase development if the DEV environment has a large number of extraneous objects not relevant to the UAT or PROD environments. The following are the tenant objects that can be independently moved from one environment to another, and the migration steps.

Migrate External Data Sources

Migrate external data sources with the Command Line Interface (CLI) tool. Use the export_datasources command to export the source environment data source to a ZIP file, and the import_datasources command to import the ZIP file to the target system. Refer to the CLI tool documentation for more information on accessing the tool and command usage.

Migrate Schemas

Migrate schemas with the Incorta Direct Data Platform™ user interface. Here are the schema migration steps in the Incorta Direct Data Platform™:

In the Navigation bar of the source environment, select Schema.
Select the checkbox to the left of the schema you would like to export.
In the Action menu that appears (⋮ vertical ellipsis icon), select Export.
In the Navigation bar of the target environment, select Schema.
In the Action bar, select select + New → Import Schema.
Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing schema checkbox if you are importing a schema with the same name as an existing schema.
The Import Results dialog will contain a status of Created if the import was successful. Select Close.

You can also migrate schemas with the Command Line Interface (CLI) tool. Use the export_schemas command to export the source environment schema to a ZIP file, and the import_schemas command to import the ZIP file to the target environment. Refer to the CLI tool documentation for more information on accessing the tool and command usage.

Migrate schemas along with their files

You can migrate schemas along with their files on the shared storage. You must have root access to the host that runs the Analytics platform in both environments or ask the System Administer who has root access to copy the files.

Here are the steps to migrate schemas along with their files:

Export the schema from the original environment using the platform's user interface or the CLI.
Import the schema to the target environment using the platform's user interface or the CLI.
Important
If the schema exists in both environments, do the following in the target environment before performing the next steps:
- Enable the Overwrite existing schema option when importing the schema.
- Delete the schema source directory from the shared storage.
- Delete the schema records from the FILES_VERSIONS and VERSION_LOCK metadata database tables.
- Delete the schema's time.log file. The path of this file is /home/incorta/IncortaAnalytics/Tenants/<tenant_name>/<schema_name>.
Copy the schema's folder that exists under the tenant's source directory in the original environment to the source directory of the tenant in the target environment.
Copy the schema's time.log file from the original environment to the target environment. The path of this file must be /home/incorta/IncortaAnalytics/Tenants/<tenant_name>/<schema_name>.
Load the schema from staging.

Notes

When migrating shared storage files from one environment to another, you must load the schema from staging after copying the source directory. Copying the ddm and source directories will not have the same result.
To migrate only one object in a physical schema, you need to copy the whole object directory that exists under the physical schema in the source directory. The path to the object directory that you need to copy is as follows: /home/incorta/IncortaAnalytics/Tenants/<tenant_name>/source/<schema_name>/<object_name>.
Both environments must run an Incorta release that supports file versioning.
If the schema exists in both environments, you must replace the schema's time.log file in the target environment with the one in the original environment, and the copied files should not have records in the FILES_VERSIONS or VERSION_LOCK metadata database tables to ensure data consistency when loading the schema incrementally.

Migrate Business Schemas

Migrate business schemas with the Incorta Direct Data Platform™ user interface. Here are the business schema migration steps in the Incorta Direct Data Platform™:

In the Navigation bar of the source environment, select Business Schema.
Select the checkbox to the left of the business schema you would like to export.
In the Action menu that appears (⋮ vertical ellipsis icon), select Export.
In the Navigation bar of the target environment, select Business Schema.
In the Action bar, select select + New → Import Business Schema.
Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing business schema checkbox if you are importing a business schema with the same name as an existing business schema.
The Import Results dialog will contain a status of Created if the import was successful. Select Close.

Migrate Schema Load Jobs

Migrate schema load jobs manually from one environment to another. You can recreate the schema load in the target environment based on the load job configuration in the source environment. Refer to the Scheduler documentation for additional information.

Migrate Schema Alerts

Migrate schema alerts manually from one environment to another. You can recreate the schema notification in the target environment based on the schema notification configuration in the source environment. Refer to the Scheduler documentation for additional information.

Migrate Dashboards

Migrate dashboards with the Incorta Direct Data Platform™ user interface. Here are the dashboard migration steps in the Incorta Direct Data Platform™:

In the Navigation bar of the source environment, select Content.
Select the dashboard you would like to migrate to open it.
In the More Options menu (⋮ vertical ellipsis icon), select Export.
In the Export Dashboard dialog, select Export.
In the Navigation bar of the target environment, select Content.
In the Action bar, select select + New → Import Folder/Dashboard.
Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing dashboard checkbox if you are importing a dashboard with the same name as an existing dashboard.
The Import Results dialog will contain a status of Created if the import was successful. Select Close.

You can also migrate dashboards with the Command Line Interface (CLI) tool. Use the export_dashboards command to export the source environment dashboard to a ZIP file, and the import_dashboards command to import the ZIP file to the target environment. Refer to the CLI tool documentation for more information on accessing the tool and command usage.

Migrate Dashboard Alerts

Migrate dashboard alerts with the Incorta Direct Data Platform™ user interface. Here are the Dashboard Alert migration steps in the Incorta Direct Data Platform™:

In the Navigation bar of the source environment, select Scheduler, and then the Data Notifications tab.
Select the checkbox to the left of the dashboard alert you would like to export.
In the Action menu that appears, select Export (Download icon).
In the Navigation bar of the target environment, select Scheduler, and then the Data Notifications tab.
In the Action bar, select select + New → Import Notifications.
Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing alerts checkbox if you are importing a data alert with the same name as an existing data alert.
The Import Results dialog will contain a status of Created if the import was successful. Select Close.

Migrate Dashboard Delivery Jobs

Migrate dashboard delivery jobs with the Incorta Direct Data Platform™ user interface when you migrate the associated dashboard. Select the Include Scheduled Jobs checkbox in the Export dialog.

Migrate Bookmarks

Migrate bookmarks with the Incorta Direct Data Platform™ user interface when you migrate the associated dashboard. Select the Include Bookmarks checkbox in the Export dialog. Both public and private bookmarks will be preserved.

Important

Including bookmarks during the export process will overwrite bookmarks in destination environment. To keep existing destination bookmarks, do not choose that option.

Migrate Global Variables

Migrate global variables manually from one environment to another. You can recreate the global variable in the target environment based on the global variable configuration in the source environment. Refer to the Schema Manager documentation for additional information.

Migrate Session Variables

Migrate internal and external session variables with the Command Line Interface (CLI) tool. The export_session_variables command can be used to export the session variables to a ZIP file from the source environment, and the import_session_variables command can be used to import the ZIP file to the target system. Refer to the CLI tool documentation for more information on accessing the tool and command usage.

Migrate Groups

Migrate groups manually from one environment to another. You can recreate the groups in the target environment based on the group configuration in the source environment. Refer to the User Profile Manager documentation for additional information.

Migrate Users

Migrate users manually from one environment to another. You can recreate the users in the target environment based on the user configuration in the source environment. Refer to the User Profile Manager documentation for additional information.

A more complex scenario involves the migration of more than one of the tenant objects listed above. It is critical to ensure that dependent objects are not broken when a change is made to a tenant object. For example, when you change a business schema column, it is important to migrate the affected dashboards. As mentioned above, the Inspector Tool will help to identify any migration issues.

Migrate as a Tenant Super User

Super Users typically migrate full tenants. Ideally use full tenant migration in early phase development for the initial creation of the UAT or PROD environments. The following are considerations when you migrate a full tenant:

All objects are migrated with the original ownership. It is recommended that the objects in PROD are owned by Administrators or Super Users, not by developers.
Data sources point to databases in DEV or UAT that are different than in PROD.
There is not an option to exclude certain objects from migration.

There are two options for migrating a full tenant:

Use the CMC
Use the TMT

The CMC option is user interface driven, while the TMT is a command line utility. The other difference is the TMT has an option to copy local data files and folders as part of the migration, while the CMC does not. Before you migrate the full tenant with either option, it is important to first create a tenant backup.

Backup the Existing Tenant in the Target Environment

The following are the steps to create a tenant backup using the CMC:

Sign in to the CMC of the source environment.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Tenants.
In the More Options menu (⋮ vertical ellipsis icon) at the end of the tenant row, select Backup.

Use the CMC to Migrate the Full Tenant

The following are the steps to migrate the full tenant using the CMC:

Sign in to the CMC of the source environment.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Tenants.
In the More Options menu (⋮ vertical ellipsis icon) on the right side of the tenant row, select Execute inspector now to review and verify the tenant object lineage.
In the More Options menu (⋮ vertical ellipsis icon) on the right side of the tenant row, select Export. The tenant will be packaged into a tenant_<TENANT_NAME>.zip folder containing:
- <TENANT_NAME>.properties file
- <TENANT_NAME>.zip file
  - dashboards folder containing XML
  - datasources folder containing XML
  - schemas folder containing XML
  - tenant.xml file
The exported ZIP file will be downloaded to your browser’s default download folder.
Sign in to the CMC of the target environment.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Tenants.
Select + → Import Tenant.
Drag and drop the tenant_<TENANT_NAME>.zip to the Click or drag a file here to upload panel in the Import a tenant to the cluster dialog.
Select Next.
If there is a <TENANT_NAME>.properties file located within tenant_<TENANT_NAME>.zip, select Yes, Use it to autofill the tenant property values. The tenant property values can be changed later. Otherwise, select No, ignore it.

Verify or enter the tenant properties:

Property	Control	Description
Name	text box	Enter the tenant name. Select Check to determine if a Tenant already exists with the name entered.
Username	text box	Enter the username for the Super User
Password	text box	Enter the password for the Super User
Email	text box	Enter the email address for the Super User
Path	text box	Enter the shared storage path for tenant related data
Pause scheduled jobs	toggle	Enable this property if the imported tenant will have all scheduled jobs paused on import

Enter and verify the tenant email properties:

Property	Control	Description
Sender’s Username Auth	toggle	Enable this property if the email requires username authentication
System Email Username	text box	Enable Sender’s Username Auth to configure this property. Enter the username for the system email.
System Email Address	text box	Enter the email address for the system email
System Email Password	text box	Enter the password for the system email address
SMTP Host	text box	Enter the Simple Mail Transfer Protocol (SMTP) host for the system email
SMTP Port	text box	Enter the SMTP port number
Share Notifications	toggle	Enable this property to share notifications

Select Create to finalize the tenant migration.
Important
You may need to enter the username and password for some of your imported connectors that are in the tenant. This affects the following connectors:
- Amazon Web Services (AWS)
- RedShift
- Apache Drill
- Apache Hive
- Cassandra (Simba)
- Custom SQL
- IBM DB2
- IBM Netezza
- Microsoft SQL Server
- Microsoft SQL Server (JTDS)
- MongoDB BI
- MySQL
- NetSuite - SuiteAnalytics
- Oracle
- Presto
- PostgreSQL
- SAP Hana
- SAP Sybase IQ
- Teradata
- Vertica
If the tenant contained data files that were not part of the migration, they should be added to the indicated directory for tenant data:
```
<CMC_INSTALLATION_PATH>/IncortaAnalytics/Tenants/<Tenant_Name>/data
```
Run the Inspector Tool following the migration to identify any lineage errors that may have occurred from moving the tenant between environments.

Use the TMT to Migrate the Full Tenant

Another option to migrate a full tenant is to use the TMT command line tool. A System Administrator with root privileges to the host running the CMC is able to run the TMT.

TMT file path

Following is the default path to the TMT tool in your CMC:

<CMC_INSTALLATION_PATH>/IncortaAnalytics/cmc/tmt

TMT Command for Tenant Export

Here are the TMT commands to export the full tenant:

cd <CMC_INSTALLATION_PATH>/IncortaAnalytics/cmc/tmt/
./tmt.sh --cluster-name <CLUSTER_NAME> --export <TENANT_EXPORT_NAME>.zip --cf

TMT Command for Tenant Import

Here is the TMT command to import the full tenant:

./tmt.sh -clnm <CLUSTER_NAME> -i <PATH_TO_TENANT_EXPORT_NAME>.zip

Refer to the TMT documentation for more information on command usage.

Migrate as a System Administrator

System Administrators typically migrate host files that contain configurations, properties, customizations, and more. This requires root access to the hosts. As necessary, copy host files manually from one environment to another, or edit the file manually in the target environment as indicated in the table below:

Host File	Migration Action
Linux Host Configurations - Ulimit	Manually edit `/etc/security/limits.conf`
CMC Node Configuration files	Manually edit all files under `<CMC_INSTALLATION_PATH>/cmc/cmcData` Manually edit `<CMC_INSTALLATION_PATH>/cmc/conf/catalina.properties`
Analytics and Loader Service Properties	Manually edit each of the following files: `<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/` `node.properties` `<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/` `services/<SERVICE_GUID>/conf/catalina.properties` `<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/` `services/<SERVICE_GUID>/incorta/service.properties` `<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/` `services/<SERVICE_GUID>/incorta/engine.properties` `<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/` `services/<SERVICE_GUID>/incorta/options.properties` `<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/` `nodeAgent/nodeAgent.cfg`
Spark Configuration	Manually edit `<SPARK_NODE_INSTALLATION_PATH>/spark/conf/` `spark-defaults.conf`
Zookeeper Configuration	Manually edit `<ZOOKEEPER_NODE_INSTALLATION_PATH>/zookeeper/conf/` `zoo.cfg`
Independent JARs	Copy the file to the appropriate `connectors` subdirectory `<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/` `extensions/connectors/`
Tomcat server XML	Manually edit `<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/` `services/<SERVICE_GUID>/conf/server.xml`
CSS Customizations	Manually edit `<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/` `runtime/webapps/ROOT/css/custom-new.css`
Certificates, SSL PCKS12 or Java Keystore Files for an External Data Source	Copy the file to the `certs` directory `/home/incorta/IncortaAnalytics/certs/`

Migrate as a CMC Administrator

CMC Administrators typically migrate CMC configurations. CMC configurations are migrated manually from one environment to another. You can replicate the CMC configurations in the target environment based on the CMC configurations in the source environment. Adjust any configuration values as necessary for the target environment. The following are the CMC configurations to consider for migration.

Important

An update of CMC configurations may require a restart of the analytics service, loader service, or all services.

Migrate General Configurations

Migrate Node Services Configurations

Migrate the On Heap Memory (GB), Off Heap Memory (GB), and CPU Utilization (%) for both the Analytic and Loader services as follows:

Sign in to the CMC.
In the Navigation bar, select Nodes.
In the Services tab, select either an Analytics or Loader Service.
Select Edit (Pen icon).
Update the node services configurations as necessary.
Select Update.

Migrate Notebook Add-on

Enable Notebook Add-on for for the Analytics service as follows:

Sign in to the CMC.
In the Navigation bar, select Nodes.
Select the Add-ons tab.
Select + → Add Notebook.

Migrate Server Cluster Configurations

Migrate the server cluster configurations as follows:

Sign in to the CMC.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Cluster Configurations.
In the panel tabs, select Server Configurations.
In the left pane, select the one of the following options to update the associated server configurations:
- Kafka Consumer Service Name
- SQL Interface
- Spark Integration
- Tuning
- UI Customizations
- Diagnostics

Migrate Default Tenant Cluster Configurations

Migrate the default tenant configurations as follows:

Sign in to the CMC.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Cluster Configurations.
In the panel tabs, select Default Tenant Configurations.
In the left pane, select one of the following options to update the associated default tenant configurations:
- Security - Authentication Type
- Regional Settings
- Email - SMTP Host, Enable SMTP SSL, Enable Notifications, Enable Internal Error Notifications
- Data Loading - Stop Loading on First Error, Pause Scheduled Jobs
- Integration - Google Maps API Key, Apple Maps Developer Team ID, Apple MapKit JS Key ID, Apple Maps API Key, Google Drive Client ID, Google Drive Client Secret, Dropbox App Key, Dropbox App Secret, Box Client ID, Box Client Secret, Mapbox API KeyTenant EFS
- Advanced - Insight Max Groups UI Default, Insight Max Groups Limit, Insight Max Values in Filter Subquery, Force Reload Columns, Sync In Background, Warmup Mode
- Tuning - Maximum Concurrent Load Jobs, Table Maximum Parallel Chunks, Evaluate Session Variables At Login
- Incorta Labs - Enable Insight 'View As' Menu, Notebook Integration
- External Visualization - Incorta Host, Incorta Port