As of 03/02 2020 the new version 2.0.0 is out

This major release was focused on restructuring the project, its artifacts, and configurations. These are not changes that should affect users but are breaking for the people who deploy the project.

Breaking Changes

  • #64 Standardization and Conformance are now combined in one artifact - spark-jobs.jar. This allows using a single JAR file for both jobs. The class names for the jobs remain the same. Note: This change requires updating helper scripts and setting the SPARK-JOBS-JAR environment variable in enceladus_env.sh.
  • #1090 All Enceladus modules are now under the same enceladus group ID in the absa.co.za organization. They will no longer have the enceldus- prefix and will not appear as top-level artifacts in maven.
  • #1099 Menas properties no longer have the za.co.absa.encealdus. prefix. See the application.properties.template fo details in deploying Menas.

Conformance Improvements

  • #1078 Added broadcast strategy for the mapping conformance rule to improve performance when a mapping table is small, as well as to support joining array elements of streaming sources to mapping tables.
  • #1129 The broadcasting mapping rule strategy can now be used in experimental mode. It’s only required that this setting be specified: conformance.mapping.rule.experimental.implementation=true. There are two additional options to control the broadcasting strategy. When conformance.mapping.rule.broadcast=auto, the strategy will be used for mapping tables smaller than the size defined (in megabytes) by conformance.mapping.rule.max.broadcast.size.mb.
  • #1136 Nested array transformations API were extracted into a separate project: spark-hats.
  • #1159 Fixed the broadcasting mapping rule dropping of standardization errors when trying to conform a null array.

Standardization Improvements

  • #1101 Added support for metadata in arrays of simple types (e.g. pattern can be specified for array of timestamps).

Menas Improvements

  • #1118 Update Spring Boot to version 2.2.2.
  • #1151 Fixed issue with selecting the Runs tile on the landing page not loading the Runs in the navigation.
  • #1167 Fixed bug where interrupted Runs in the database cause failure to query RunSummaries.

Other changes

  • #1145 Removed external modules fixed-width and kafka.
  • #1147 Set Copyright year to year of inception.