1. Home
  2. Previous Releases
  3. Latest Release
  4. Guides
  5. Concepts
    1. Aggregate filter
    2. Alias
    3. Apache Avro™
    4. Apache Kafka™
    5. Applied filter
    6. Base table
    7. Bookmark
    8. Business schema
    9. Business schema view
    10. Business schema view column
    11. Business schema view formula column
    12. Column
    13. Connector
    14. Dashboard filter
    15. Dashboard folder
    16. Dashboard runtime filter
    17. Data agent
    18. Data alert
    19. Data destination
    20. Date system variable
    21. Distinct filter
    22. Dynamic Field
    23. Expression
    24. External data source
    25. External session variable
    26. Filter expression
    27. Filter expression session variable
    28. Filter option
    29. Formula column
    30. Formula expression
    31. Full load
    32. Global variable
    33. Incorta Analyzer Table
    34. Incorta Analyzer Table column
    35. Incorta Analyzer Table formula column
    36. Incorta Analyzer View
    37. Incorta Analyzer View formula column
    38. Incorta SQL Table
    39. Individual filter
    40. Insight filter
    41. Internal query expression
    42. Internal session variable
    43. Join
    44. Join condition
    45. Join condition column
    46. Join filter
    47. Join filter column
    48. Level based measure
    49. Load filter
    50. Local data file
    51. Local data folder
    52. Materialized View
    53. Materialized View Column
    54. Materialized View Formula Column
    55. Measure filter
    56. Miscellaneous system variable
    57. Multi-Source table
    58. Physical schema
    59. Physical schema table
    60. Physical schema table column
    61. Physical schema table formula column
    62. Physical schema table runtime security filter
    63. PostgreSQL (Incorta SQL) View
    64. Presentation variable
    65. Prompt
    66. Query system variable
    67. Runtime business view
    68. Runtime business view formula column
    69. Runtime security filter
    70. Schema evolution
    71. Session variable
    72. Shared Storage
    73. Upgrade Readiness
    74. Variable
    75. Visualization
  6. Tools
  7. References
  8. Integrations
  9. Data applications
  10. Releases

Concepts → Schema Evolution

Overview

Before 2024.7.x, adding and removing physical columns required performing a full load of the respective physical table or materialized views (MVs). Starting 2024.7.x, Incorta has leveraged Delta Lake column mapping to support schema evolution, streamlining adding and removing columns without performing a full load of the respective objects.

Upgrade considerations

  • As a result of using Delta Lake column mapping, Incorta has modified the definition of columns in the schema metadata. After upgrading to 2024.7.x or later releases, the column definition will change with the next schema update.

  • When importing schemas from previous releases with the overwrite option, you can keep existing schema data. When the Keep Data option is selected,

    • The imported schema XML is updated with the new table and column definitions.
    • Incorta can use existing Parquet files.
    • You do not need to perform a full load of the imported schema.

Column name considerations

Column names in Incorta are case-sensitive. Renaming a column (in the extraction query, for example) or changing its letter case results in a new column and typically does not require a full load starting 2024.7.x. Reverting the name changes results in a new column as well.

Notes
  • The following actions still require a full load of the respective object:
    • Adding, changing, or removing a key column
    • Encrypting a column
    • Changing the column data type
  • You must revalidate the query of MVs referencing removed or updated columns.