Implemented capability to separate diff logs via log4j2 #315
Conversation
| ./spark-submit --properties-file cdm.properties \ | ||
| --conf spark.cdm.schema.origin.keyspaceTable="<keyspacename>.<tablename>" \ | ||
| --conf "spark.executor.extraJavaOptions='-Dlog4j.configurationFile=log4j2.properties'" \ | ||
| --conf "spark.driver.extraJavaOptions='-Dlog4j.configurationFile=log4j2.properties'" \ |
There was a problem hiding this comment.
What will be the name of the 2 log files that will get generated? Could we show that as an example? TY
There was a problem hiding this comment.
Yes, the details of the log file are in the log4j2.properties file. CDM users can change the details as needed, but if they use the file as-is, the logs will be directed 3 ways as below
- All non app related logs (i.e. logs generated by libraries & not CDM code) will continue to be written to console
- All logs generated by CDM code except for the
diffrelated logs (i.e. details ofmissing&mismatchedrows) will be recorded in./cdm_logs/cdm.logfile. - All
diffrelated logs (i.e. details ofmissing&mismatchedrows) will be recorded in./cdm_logs/cdm_diff.logfile.
Note: All log files will be rolled when they cross the set size limit (default is 10MB), there can be 100 rollovers by default.
There was a problem hiding this comment.
For context, most customers will not use this feature, but some customers who plan to run only validation & expect to have a lot of diff rows want a separate file for such records in a separate file instead of having to grep & find it.
RELEASE.md
Outdated
| @@ -1,5 +1,6 @@ | |||
| # Release Notes | |||
| ## [4.4.2] - 2024-10-TBD | |||
| ## [4.4.2] - 2024-10-03 | |||
There was a problem hiding this comment.
| ## [4.4.2] - 2024-10-03 | |
| ## [4.5.0] - 2024-10-03 |
| @@ -173,8 +173,6 @@ public List<Number> getNumberList(String propertyName) { | |||
|
|
|||
| @Override | |||
| public List<Integer> getIntegerList(String propertyName) { | |||
| List<Integer> intList = new ArrayList<>(); | |||
|
Looking into what made the build fail |
What this PR does: Implements capability to separate diff logs via log4j2
Which issue(s) this PR fixes:
Fixes #
Checklist: