As of 15/05 2020 the new version 2.5.0 is out

Standardization & Conformance Improvements

  • #1294 Record ID generation: enceladus_record_id column is added in Enceladus (Standardization/Conformance) if it does not exist. By default it contains UUID for each record, but the implementation is overridable with -Denceladus.recordId.generation.strategy={uuid|stableHashId|none}
  • #1322 _INFO file validation: Added control.info.validation configuration option for _INFO file validation which can have the following values: strict will fail the job on failed validation, warning will display the error as warning, but will not fail the job and none will ignore the result of the validation. Providing an incorrect value will fail the job with a RuntimeException
  • #1317 _INFO file validation: When _INFO file validation is enabled (#1322), the error will provide a verbose error such as “Checkpoint validation failed: Missing source checkpoint”
  • #1329 Added support of specifying JavaX security properties in application.conf instead of passing them via command line as -D.... Added support for secure Kafka operation when Spark jobs are running in cluster mode (only client mode was supported until now for secure Kafka).

New SparkJobs Configuration Options