Usage > Menas Conformance Rules
Table Of Contents
- Table Of Contents
- Intro
- Casting Conformance Rule
- Coalesce Conformance Rule
- Concatenation Conformance Rule
- Drop Conformance Rule
- FillNulls Conformance Rule
- Literal Conformance Rule
- Mapping Conformance Rule
- Negation Conformance Rule
- SingleColumn Conformance Rule
- SparkSessionConf Conformance Rule
- Uppercase Conformance Rule
Intro
Conformance Rules are a way for the user to enhance the data. Every conformance rule has an output column and an option for running a control measure. Output column text field defines the name of the column into which the rule will output the result into. Control measure check will run an Atum control measure check as in previous stages as defined in controls of _INFO
file.
Every column also has one or more input columns.
Important - We never override a column. Each rule produces a new column.
Casting Conformance Rule
Casting conformance rule allows users to cast a specific column to another type. This conformance rule provides a selection of other types.
Allowed Conversions are:
From | To |
---|---|
Anything | String |
Boolean | Any Numeric |
Any Numeric Integer | Any Numeric given that it fits |
Any Floating point Numeric | Any Floating point Numeric given that it fits |
String | Any Numeric or Time, or Boolean given it is "true" or "false" |
Date | Timestamp |
Timestamp | Date |
Coalesce Conformance Rule
Coalesce conformance rule applies value for the new column from the first non-null value from the list of columns.
Concatenation Conformance Rule
Concatenation conformance rule concatenates two or more input columns together into a single column. Columns are first transformed into their string representation before the actual concatenation.
Drop Conformance Rule
Drop conformance rule removes a column from the output data.
FillNulls Conformance Rule
FillNulls conformance rule takes a column and replaces all the nulls with a literal provided by the user.
Literal Conformance Rule
Literal conformance rule adds a column with a string literal provided by the user.
Mapping Conformance Rule
To use a mapping conformance rule, the user first needs to define a Mapping Table in Menas. Mapping Tables have the same properties and rules around them as Datasets, and it is expected of them to be in a parquet format.
When defining a mapping conformance rule, users first need to pick a correct Mapping Table and correct version. Then there is a question if the data can have Null values in join conditions. This means if the join that will be executed should be null safe or not.
Then the join conditions convey the relationship between the Dataset and Mapping Table. The join condition specifies how the rows from one table will be combined with the rows of the other table. This is based on the equality of the values in the selected columns.
Last are the output columns that specify which columns from the mapping table will be written into their respective output column. The mapping table column is called the target value.
Negation Conformance Rule
Negation conformance rule negates any Numerical or Boolean value.
SingleColumn Conformance Rule
Single column conformance rule transforms column into a column of structs of previous column’s values. Input column alias here will be the name/key of the struct.
SparkSessionConf Conformance Rule
SparkSessionConf conformance rule is able to pull out a configuration value from the SparkSession based on the key provided.
Uppercase Conformance Rule
Uppercase conformance rule transforms all character letters in the column into capital letters.