Tools → Schema Designer → Load Jobs

About the Schema Designer Load Jobs

You can use the Schema Designer to perform load jobs of all objects in the physical schema or to load a specific object. You can also create load plans to schedule loading one or more physical schemas.

The Summary section in the Schema Designer shows the status of the last load job or the stage of the current load job. You can review the details of the current or previous load jobs in the Load Job Details Viewer.

Incorta supports different types and strategies for loading data.

Changes that require data load

Some updates you make to the physical schema objects require loading data, either load from source (full load) or load from staging, to ensure data consistency.

The following are the updates that require load from staging:

• Adding, updating, or removing the load filter
• Enabling performance optimization for non-optimized objects

The following are the updates that require full load:

• Adding a new physical schema table or materialized view
• Changing the data type of a physical schema table column or MV column
• Changing the source of a materialized view or physical schema table, whether by selecting another source file in the Data Source properties dialog or editing the query
• Adding or changing a key column (changing the column function from key to dimension or measure and vice versa) in a physical schema table or MV
• Adding a new physical schema table column (before 2024.7.x)
• Adding a new MV column (before 2024.7.x)
• Changing the object type, for example, changing a physical schema table to an Incorta Analyzer table or MV
• Removing a physical schema table column or a materialized view column that functions as a key
• Changing the encryption status of one or more columns in a physical schema table or materialized view

Schema Designer actions for load jobs

With the Schema Designer, you can perform the following actions for data load jobs

• Change the load order for physical schema objects
• Perform a full load of the physical schema data
• Perform an incremental load of the physical schema data
• Load physical schema data from staging
• Run a data purge job
• Create a load plan
• Perform a load of a physical schema object
• Review the status of a load job

Change the load order for physical schema objects

• In the Navigation bar, select Schema.
• In the list view of schemas, select a given physical schema.
• In the Schema Designer, select Settings (gear icon).
• Select Change Load Order.
• In the Choose Load Order dialog, you can do the following to manage the table groups before changing their load order:
  • For each table (or materialized view) that you want to add to a new load group, select the check box, and then select Add to New Group.
  • For each table (or materialized view) that you want to add to an existing load group, select the check box, and then select Add to Existing Group.
  • To reset table grouping and order, select Revert to Default.
• To set the order of a table group, select the up arrow (to the right of the group name) to move the group upwards, or select the down arrow (to the right of the group name) to move the group downwards. Repeat this action until you order the table groups as required.
• Select Save.

Perform a full load of the physical schema data

• In the Schema Designer, in the Action bar, select Load → Full Load.
• In the dialog, select Load.

Perform an incremental load of the physical schema data

• In the Schema Designer, in the Action bar, select Load → Incremental Load.
• In the dialog, select Load.

Load physical schema data from staging

• In the Schema Designer, in the Action bar, select Load → Staging Load.
• In the dialog, select Load.

Run a data purge job

If one or more schema objects (physical table or MV) have a data retention or exclusion set configurations you can start a data purge job to delete Parquet data that does not satisfy the data retention conditions or that matches the exclusion sets for schema objects that have configurations for these options. Other objects will be loaded from staging.

• In the Schema Designer, in the Action bar, select Load → Data Purge.
• In the dialog, select Load.

Create a load plan

You can organize schemas in a load plan into separate groups that the Loader Service handles sequentially in different load jobs. However, the Loader Service continues to handle schemas in a group concurrently in a single load job.

Here are the steps to add a load plan using the Schema Designer.

• In the Schema Designer, in the Action bar, select Load → Add Load Job.
• In the New Load Plan dialog, specify the load plan details, and then save the plan details.
• Set the schedule options to execute the load plan or select to Schedule later.

For information about the details of a load plan and the schedule options, refer to Tools → Scheduler → Create a scheduled load plan.

Perform a load of a physical schema object

You can start a full load or a load from staging of a specific physical schema object instead of loading all objects in the physical schema. This option is not available for an alias.

Starting 2024.7.x, you can also start a Data Purge job in the case of a physical table or MV that has data retention or exclusion set configurations to delete the object's Parquet data that does not satisfy the data retention condition or that matches the exclusion set.

Here are the steps to start a load job for a physical schema object:

• In the Schema Designer, in the Tables tab, for a given physical schema object, select More Options (⋮ vertical ellipsis).
• Do one of the following:
  • Select Load Table to start a full load job.
  • Select Load from Staging to start a load-from-staging job.
  • Select Data Purge to start a data purge job.
• In the dialog, select Load.

Review the status of a load job

• In the Schema Designer, in the Summary section, select the Last Load Status link.
• In the Load Job Details Viewer, review the load job status.