Explanations of the CERT reports and other QA measures ====================================================== This document is prepared on the basis of the latest status prior to the final run. It is therefore possible that during the final run, when everything is rebuilt and tested and new versions of the tools are used, new findings may emerge that are not yet described here. But since we are completely transparent here and everything is regenerated under the respective build, this cannot be ruled out. The respective non-compliants in the CERT and zACS reports are commented below. In addition to CPPCHECK-Premium and zACS from IBM, we also use Scan-Build, TRIVY and OSV-SCANNER as a vulnerability scanner, Valgrind and a own instrumentation of our debug code as memory and file handle checker, whose report will be added here at the end. At the end you will also find a description of our test framework for unit and regression tests, how we test our graphical user interfaces and how we determine the degree of test coverage with CoCo and all other QA measurements. A special chapter is dedicated to Software Bills of Materials (SBOM), which we supply for our software. But now let's move on to the CERT reports from CPPCHECK-Premium as part of our valnuribility scans, which can be found under DOCZIP(CERTREP[.zip]). The HTML files have to be unpacked and have been created for this delivery as part of our automation. CERT: FL5 --------- This is our main project with delivered code from limes. With the new code, we aim to fulfill all CERT criteria. The CPPCHECK-Premium still has many false positives here, which lead to corresponding non-compliants. We have analyzed every non-compliance and commented on it below. * EXP36 Do not cast pointers into more strictly aligned pointer types and * EXP39 Do not access a variable through a pointer of an incompatible type and * INT36 Converting a pointer to integer or integer to pointer This is a mix of false positives, because a cast to a byte array pointer is perfectly permissible. And many issues from serialization, where we dispense with the additional copy depending on the platform, where it is not necessary because the data in the memory already fits. As CPPCHECK always runs through all code branches, the one branch where we are on a platform where the cast is permissible is flagged. Of course, the requirement is intended to prevent problems on other platforms, but where this is not necessary, it does not have to be coded so elaborately. One example: We have to deserialize an integer in 2's complement with normal byte order and are on a machine with byte addressing and which works with big endian (System Z with z/OS from IBM), then we cast the address, but on a RISC system (SPARC) where you can only address whole words, we have to copy the 4 bytes here. So we already know what we are doing here and it is tested that it works well on each platform. * EXP40 Do not modify constant objects These few reports are deliberate and entirely correct. We have two cases here: 1. In the few cases where parameter specifications are constant, they are adopted. However, since these are string arrays, the data type must be reformatted (a memory block with the strings in sequence into an array of pointers to the zero-terminated strings) for mapping, so the memory must be recreated and the data converted to the new form. This memory for the strings is allocated and filled with the parameters or their default values. The pointers for this are declared as ‘const’ because these parameters should not be changed afterwards. 2.Furthermore, we often work with ‘const’ declarations when they are constant in 99% of cases and are only reassigned in exceptional cases, and in these few cases, this information is also generated by the CERT scanner. For this one exception, we then use a cast that removes the ‘const’ so that we do not accidentally change the data in all other cases. Therefore, we preferred to leave the 'const' declaration with this information instead of simply removing the 'const' from the parameters. * EXP34 Do not dereference null pointers We always check for NULL pointers at our interfaces and at all points where they are returned as error indicators. With the new version of the scanner, all points must now be checked at once, even those where this cannot occur and even those that cannot be checked in normal loading modules on mainframe computers. Here we have the case of open functions that return an error leading to termination. If these functions end with NULL, all allocated return values (call by reference) are definitely set and, in the event of an error, also released again. So, with a return code of 0, we do not check all parameters for inequality to NULL again in our own functions. The second case where this large number of messages come from at once is all standard load module libraries, such as RACF, CCA/ICSF and others, which work exclusively via a call by reference for each parameter. These interfaces often use long as a data type, but long can be 32 bits or 64 bits wide. We only work with fixed-width data types, so we have to provide all these libraries with a wrapper, which is created when our fixed 32-bit value has to point to 64 bits in a long pointer. Then we create a local variable for each long and assign the 32-bit integer via dereferencing, make the call and then assign the 32-bit value to the long pointer. After that, the call is made and we then assign the long back to the 32-bit value that points to the transfer. If such a parameter is allowed to be a null pointer itself, this is checked beforehand and the NULL is passed through. All these parameters of the wrapper functions for 32 vs. 64-bit interoperability are now reported here as an issue. Since these functions often do not have a return value at all, a hard exit (e.g. over assert()) of the program would have to be built-in here, and all these checks would naturally reduce performance. In addition, a NULL pointer is often already excluded here by the compiler via calling convention. In general, it must be said about this attack that if I can introduce a NULL pointer into the system that does not belong there and the system does not terminate in a segmentation fault when it is dereferenced, then the system should not be used. If I can point to 0 here, then I can also point to 1, in which case all these checks are useless because the 1 is no longer recognized as an invalid pointer. If we receive pointers from outside and the system offers a function to check them (e.g. VSMLOC under z/OS), then we check them before use. If, for example, Linux never returns a NULL pointer for a malloc() and the system only crashes when it is used if there is not enough memory, then the system is the problem at that point and not the application running on it. * INT30 Ensure that unsigned integer operations do not wrap * INT31 Ensure that integer conversions do not result in lost or misinterpreted data * INT32 Ensure that operations on signed integers do not result in overflow These overflows are either intentional because we calculate in finite fields (for example with hash or various other algorithms) or the overflows cannot occur because the value ranges do not allow it. In the latter case, we save this check for the runtime for performance reasons and only ensure in the open that there can be no overflow or that there is none in principle. * MEM33 Allocate and copy structures containing a flexible array member dynamically These issues come from the code generated as part of the MF-EDZ supports, where C structures are generated, which end up with a variable long byte array and thus represent the interface to this very specific environment, where mainframe systems under Windows and Linux are emulated in such a half-baked way. We cannot change anything about these interface definitions, nor can we change the fact that these structures are summarized in a union and this then leads to these issues. * FIO30 Exclude user input from format strings This is about the v-functions (vsnprintf) where variable parameter lists are passed for a format string. Since we have wrappers for almost all system calls, which compensate for slight platform differences, for example to guarantee null termination in vsnprintf, these issues always arise at these points. We use the control of the format string here if this is provided by the compiler of the respective platform and the actual requirements would have to be transferred to the higher-level function and checked there to see whether a static literal is always used as the format string, which is not the case in our code. * ENV33 Do not call system() This is the one call that is made for local pre- and post-processing. Of course this is critical, but in this case it is a feature that is wanted and there are various protective measures via RACF or similar to prevent misuse if you activate these feature (disable it, to be compliant). * ERR30 Set errno to zero before calling a library function known to set errno, and check errno only after the function returns a value indicating failure We have some cases where we handle an error and only later output the errno for information purposes in the error trace, but we do not explicitly handle it depending on the errno, as the error leads to a termination anyway and the system message is only added to the error trace here. In these cases, you just see the last status of the errno regardless of who caused it. In such cases, the errno can also be 0 if it the reason was not a system call that set the errno. In such an error situation, we just output the errno because it might be interesting, but we do not initialize it. * ERR33 Detect and handle standard library errors Here we have the problem that you cannot simply return errors from a system call as return code in order to check them in the layer above. Since we have a wrapper function for almost every system call, these simply return the return code of the system call, which is only verified in the layer above. Another problem here is error handling, where we use snprintf to assemble an error message and where we do not check the return code here, because we are already in error handling and it only has to be ensured that the null termination is guaranteed and the message is simply cut off in the worst case. That is why we are currently living with these issues. CERT: CLEP ---------- The CLEP is our open-source project with which we read the commands via a compiler and convert them into a dynamic data structure, which is then processed. The same applies here as for the FL5 code with the corresponding notes below (Only for the issues that have not already been described for the FL5 project above). * EXP34 Do not cast pointers into more strictly aligned pointer types * INT30 Ensure that unsigned integer operations do not wrap * INT31 Ensure that integer conversions do not result in lost or misinterpreted data * INT32 Ensure that operations on signed integers do not result in overflow * FIO30 Exclude user input from format strings * ERR30 Set errno to zero before calling a library function known to set errno, and check errno only after the function returns a value indicating failure * ERR33 Detect and handle standard library errors See above (same as for the FL5 project) * ENV34 Do not store pointers returned by certain functions A copy of an environment variable only makes sense if it is also used later. If the content of the returned pointer is evaluated directly, there is no need to create a copy. Whether I make a strdup() or a strcmp() here, if the pointer is not equal to NULL, is equivalent and therefore these issues are to be evaluated as false positives. * MSC30 Do not use the rand() function for generating pseudorandom numbers This is only the fallback implementation if the library is not used with a strong function for random generation. This fallback is not used in the FL5 project. CERT: FLAM4 ----------- This is the legacy code for the old FLAM4 components. The basis here dates back to the 80s of the last century and it is not possible to make it compliant at a reasonable cost. Therefore this report is generated for information, there are no errors and the more than 50 warnings have all been checked and they are all non-critical. The code style issues are ignored here and, if necessary, automatically corrected later. The same applies here as for the FL5 code with the corresponding notes below (Only for the issues that have not already been described for the FL5 and CLEP projects above) * EXP34 Do not cast pointers into more strictly aligned pointer types * EXP36 Do not cast pointers into more strictly aligned pointer types * EXP39 Do not access a variable through a pointer of an incompatible type * INT30 Ensure that unsigned integer operations do not wrap * INT31 Ensure that unsigned integer operations do not wrap * INT32 Ensure that operations on signed integers do not result in overflow * ERR33 Detect and handle standard library errors See above (same as for the FL5 and CLEP project) CERT: FLCC ---------- Our Control Center is a QT application in C++, which accesses our parameter tables, the CLEP and other parts of the FL5 project. The transition between this C99 code and C++ leads to some findings that cannot be avoided. * dangerousTypeCast Potentially invalid type conversion in old-style C cast, clarify/fix with C++ cast This is due to the transition from our C tables and CLP code to the C++ world of QT, with which we make the same code base available. * CTR50 Guarantee that container indices and iterators are within the valid range * INT50 Do not cast to an out-of-range enumeration value All these cases are new false positives of the new version of the scanner. In an initial analysis, a switch case ensures that only valid data is accessed. However, we need to carry out more detailed analysis here. Once again, it appears that the transition from C++ to C definitions is being criticized here, but this cannot be avoided. * DCL50 Do not define a C-style variadic function Functions with printf format strings from the CLEP project are included here. As we are reusing the C99 code here, this cannot be avoided. * DCL51 Do not declare or define a reserved identifier Defines which control the build from outside via the make files have '__' as a prefix and these are set to default values if they are not specified from outside. The latter comes into play here for the C++ code, as these values are not actually needed here and this leads to these issues. Renaming all these defines would be feasible, but as this has an impact on all platforms, we have not yet tackled this. There is no naming conflict here. * EXP57 Do not cast or delete pointers to incomplete classes Incomplete classes of QT are used here (e.g. QWidget), which we use for the user interfaces. Unfortunately, we do not currently know how we can avoid this. * EXP34 Do not dereference null pointers * EXP39 Do not access a variable through a pointer of an incompatible type * INT31 Ensure that integer conversions do not result in lost or misinterpreted data * INT32 Ensure that operations on signed integers do not result in overflow * INT36 Converting a pointer to integer or integer to pointer * ERR33 Detect and handle standard library errors See above (same as for the FL5 and CLEP project) CERT: FLGT ---------- The GUI tools are simple QT application actions that can be executed via a context menu for files in the file manager. The same applies here as has already been said for the FLCC. * EXP57 Do not cast or delete pointers to incomplete classes * EXP34 Do not dereference null pointers * ERR33 Detect and handle standard library errors See above (same as for the FLCC project) CERT: GENDOKU ------------- The GENDOCU project contains the libhtml, which is also supplied. We use this tool to create our documentation. The tool is not used productively. * CTR50 Guarantee that container indices and iterators are within the valid range See above (same as for the FLCC project) Remaining CERT reports for external libraries --------------------------------------------- The remaining reports relate to all external libraries used, where we check the code with both CPPCHECK and ScanBuild and where currently only non-critical findings are included, which the maintainers do not want to correct. Critical issues are fixed by the projects and the reports here are for information purposes only and are not commented on further. zACS: FLAMSTC ------------- The zACS report (DOCTXT(IBMZACSR)) is the result of a zACS run with the zACS tool of z/OSv3.1 from SVA against our current FLAMSTC. Current analysis of the zACS report by Jörn Meyburg (SYSeins GmbH) can be found in DOCTXT(IBMZACSA). We rate both issues as false positives of the new zACS tool and our customers are welcome to open a PMR with IBM. We have incorporated the checking of the transferred pointers with VSMLOC to resolve one of the issues, but this macro is not stable and returns errors where there are none. Therefore, we are currently ignoring the return code here so that FLAMSTC does not crash at the customer's site due to VSMLOC malfunctioning. If incorrect pointers or lengths are transferred here, which is not the case for us, this leads to a 0C4 of the FLAMSTC, which cannot be used for an attack in this case either, because the process (address space) is down. CPPCHECK-Report --------------- The following is the standard CPPCHECK report for all sources including the external libraries and our test framework without any style issues. ------------------------------------------------------------------------ CSRC/IO/DRIOLOCL.c 342 readdirCalled 477 portability Non reentrant function 'readdir' called. For threadsafe applications it is recommended to use the reentrant replacement function 'readdir_r'. fuse/flucFS.c 1167 readdirCalled 477 portability Non reentrant function 'readdir' called. For threadsafe applications it is recommended to use the reentrant replacement function 'readdir_r'. 2467 readdirCalled 477 portability Non reentrant function 'readdir' called. For threadsafe applications it is recommended to use the reentrant replacement function 'readdir_r'. fuse/flucFScheck.c 223 readdirCalled 477 portability Non reentrant function 'readdir' called. For threadsafe applications it is recommended to use the reentrant replacement function 'readdir_r'. fuse/flucFSconf.c 127 readdirCalled 477 portability Non reentrant function 'readdir' called. For threadsafe applications it is recommended to use the reentrant replacement function 'readdir_r'. 156 readdirCalled 477 portability Non reentrant function 'readdir' called. For threadsafe applications it is recommended to use the reentrant replacement function 'readdir_r'. ------------------------------------------------------------------------ The issues concern the readdir() function, where the recommended solution is deprecated and must therefore also be regarded as a false positive. Scan-Build-Report ----------------- The scan build is run over the complete code including all external libraries and our test framework. Here is the corresponding report. ------------------------------------------------------------------------ Bug Group File Function/Method Line Unused code Dead assignment lib/bzip2/bzlib.c BZ2_bzCompress 450 Unused code Dead assignment lib/bzip2/compress.c sendMTFValues 503 Unused code Dead assignment lib/bzip2/compress.c generateMTFValues 225 Unused code Dead assignment lib/bzip2/compress.c sendMTFValues 518 Unused code Dead nested assignment lib/lmdb/mdb_stat.c main 171 Logic error Dereference of null pointer CSRC/FLMMAT.c pcGetElementData 263 Logic error Result of operation is garbage or undefined flam4/nucmod.c flmsort 4667 Logic error Result of operation is garbage or undefined lib/bzip2/compress.c generateMTFValues 170 Logic error Unix API lib/lz4/lz4.c LZ4_saveDict 1764 Logic error Unix API lib/lz4/lz4.c LZ4_decompress_generic 2112 Logic error Unix API lib/lz4/lz4.c LZ4_decompress_generic 2106 ------------------------------------------------------------------------ Dead assignments can be disregarded as a performance issue in this external libraries. The remaining logical errors were checked and can be considered false positives, except in the lz4 library where assert() functions are used to check for a null pointer (e.g. after failed memory allocation) before use. Valgrind-Report --------------- The Valgrind runs under x86 in a current 64-bit Arch Linux across all our tests, resulting in the following report. It checks whether there have been memory area violations and whether all allocated resources (memory, files) have been released again. ------------------------------------------------------------------------ ==837912== Memcheck, a memory error detector ==837912== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al. ==837912== Using Valgrind-3.25.0 and LibVEX; rerun with -h for copyright info ==837912== Command: /home/pope/longrun/FL5/bin/linx64d0/FTMAIN USEUP BUILD ==837912== Parent PID: 837801 ==837912== ==837912== ==837912== FILE DESCRIPTORS: 4 open (2 inherited) at exit. ==837912== Open file descriptor 2: /home/pope/memchecklog-master.txt ==837912== at 0x88AAA2B: dup2 (dup2.c:29) ==837912== by 0x624A985: FT_CLProgramTest (FTEST.c:1637) ==837912== by 0x624B88B: FTCL_ProgramTestMode (FTEST.c:1782) ==837912== by 0x624A331: FTUPCL_ProgramTestPropertyMode (FTEST.c:1564) ==837912== by 0x621592A: CHECK_LICORRUP (FTFLMACS.c:538) ==837912== by 0x400B565: FTRunTest (FTEST.c:92) ==837912== by 0x400A67A: FTTestArray_RUN (FTSUITE.c:106) ==837912== by 0x4009E98: FTRunSuite (FTSUITE.c:50) ==837912== by 0x40188F6: FTRunClass (FTCLASS.c:36) ==837912== by 0x4005900: main (FTMAIN.c:431) ==837912== ==837912== Open file descriptor 1: /home/pope/memchecklog-master.txt ==837912== at 0x88AAA2B: dup2 (dup2.c:29) ==837912== by 0x624A94C: FT_CLProgramTest (FTEST.c:1637) ==837912== by 0x624B88B: FTCL_ProgramTestMode (FTEST.c:1782) ==837912== by 0x624A331: FTUPCL_ProgramTestPropertyMode (FTEST.c:1564) ==837912== by 0x621592A: CHECK_LICORRUP (FTFLMACS.c:538) ==837912== by 0x400B565: FTRunTest (FTEST.c:92) ==837912== by 0x400A67A: FTTestArray_RUN (FTSUITE.c:106) ==837912== by 0x4009E98: FTRunSuite (FTSUITE.c:50) ==837912== by 0x40188F6: FTRunClass (FTCLASS.c:36) ==837912== by 0x4005900: main (FTMAIN.c:431) ==837912== ==837912== ==837912== HEAP SUMMARY: ==837912== in use at exit: 7,591 bytes in 67 blocks ==837912== total heap usage: 4,533,577,541 allocs, 4,533,577,474 frees, 2,875,833,794,619 bytes allocated ==837912== ==837912== LEAK SUMMARY: ==837912== definitely lost: 0 bytes in 0 blocks ==837912== indirectly lost: 0 bytes in 0 blocks ==837912== possibly lost: 0 bytes in 0 blocks ==837912== still reachable: 0 bytes in 0 blocks ==837912== suppressed: 7,591 bytes in 67 blocks ==837912== ==837912== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 55 from 55) --837912-- --837912-- used_suppression: 59 ignore_libcrypto_leak_errors /home/pope/longrun/FL5/valgrind-suppress.txt:50 suppressed: 7,591 bytes in 67 blocks ==837912== ==837912== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 55 from 55) ------------------------------------------------------------------------ The two open file descriptors come from the redirection of the output to the HTML reports of the test framework. The Valgrind here means that the two file descriptors were not closed by the last redirection, which is a false positive and only affects the test framework but not the tested code. The suppression of these 59 findings concerns the libcrypto. Uninitialized memory is deliberately used here to generate random values, which is why we have suppressed this error for these modules. MEMCHECK -------- Our self-written memory and file checker basically does the same as Valgrind does for our debug code. This is instrumented accordingly and then checked on all platforms. It checks whether we release everything we allocate and whether we only work in our and valid memory areas. No finding is permitted here. Test errors will be triggered by any issue here. Test-Framework -------------- Our test framework is a self-written C99 application in which we can easily integrate unit tests of components and functions as well as complete regression tests of our programs and carry out corresponding target/actual comparisons of the return codes and outputs. We currently have almost 1 million such asserts. Furthermore, customers can outsource their regression tests for FLAM to us here, so that it is ensured that with every new release of FLAM, its use by our customers is tested. This means that customers can install new versions of our software very quickly without increased risk, which is particularly important in the event of security problems. The test framework runs on all platforms where FLAM is available and all tests are carried out on all platforms against a new and an old version of the respective operating system and only when all systems run through without errors is a new version released. Coco ---- We determine the test coverage with CoCo from QT. Central modules of the FL5CORE must have at least 70% test coverage, for the archive over 80% and all other modules that are not generated tables or similar must be over 50%. Squish, x3270 and Phyton ------------------------ With Squish from QT, we test our user interfaces automatically in the test framework to ensure that they function properly. To automatically test the ISPF panels and line commands on mainframes, we use x3270 and Phyton scripts, which we transfer via the pre-processing and execute via the post-processing of the FLCL. Software Bill of Materials (SBOM) --------------------------------- With ':>flcl sbom' you can generate an SBOM file in SPDX-JSON format for all external libraries used, which we also deliver as FLELSBOM.json in DOCTXT. This is a minimalist version that only lists the name, license and version of the libraries. With ':>flcl sbom_cyclondx_json' you can generate the same in CyclonDX format as JSON file. Furthermore, when packaging with the SBOM tool, we generate an SBOM file (manifest.package_name.json) in the 'doc' folder also in SPDX format in JSON via our delivered components, which also contains checksums for all files in the installation package and represent our internal dependencies. Both SBOM files are validated with Trivy and we use Trivy and OSV-Scanner in-house as additional vulnerability scanners for our systems and projects. Furthermore, we use OSV server (for "ecosystem": "Red Hat") to retrieve security vulnerabilities for the external libraries we use online and to maintain the security patches (back ports or update to new version) here or report them further. Miscellaneous ------------- We run a dependency check so that we don't forget anything that is necessary for operation and we know which modules depend on which, which we also use to generate the make files. We have a header check so that we don't make any duplicate includes. We have a spell checker that finds common spelling mistakes. Then we have a character checker that does not allow multibyte characters in the sources. Then we have a check for the use of diacritical characters so that they are not used in literal output and thus lead to hyroclyphs if the local character set is not correct. All these and some more checks are done during the makefile build and lead to normal compile errors. If the build is error-free, then it is tested locally (Linux) and remotely (z/OS, Windows) (build tests). If a new code version is pushed, a merge is made between the employee's remote GIT repo and the central GIT repo via the push automation, this must be buildable and the push test must run successfully. Only then is the status transferred to the master branch of the main repo. The subsequent push automation starts the pack factory, which builds the installation package that is rolled out to all test machines every night. Here, the software is installed on the naked system, just like at the customer, and a complete test run (ALL (Build, Push, Auto)) is carried out, the results of which can be viewed in the test framework via a browser. As part of this automation, the Valgrind, the vulnerability scanner and code coverage tool also run, so that all results are available at a glance every morning. This form of continues integration allows us to publish a new version of our software in the form of a rolling release every week on Monday when everything is green.