Rebase dev

.gitignore

@@ -224,4 +224,8 @@ cython_debug/
 #  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
-#.idea/
+#.idea/
+
+# Documentation Stuff
+Gemfile.lock
+_site/

CONTRIBUTING.md

@@ -57,21 +57,21 @@ To ensure `clang-format` and **unit tests** are always run against your code on
 
 ### Adding New Functionality
 
-If you have new functionality that can be added to the package, open a GitHub pull request against the `dev` branch with your changes. In the PR, make sure to clearly document the new functionality including why it is valuable.
+If you want to continube new functionality, open a GitHub pull request against the `dev` branch with your changes. In the PR, make sure to clearly document the new functionality including why it is valuable.
 
 ### Testing
 
-The main way to test RACFu is to write **unit tests** in the [`tests`](tests) folder which contains **mocks** that mock the real **IRRSMO00** and **IRRSEQ00** RACF callable services to enable **request generation** and **response parsing** logic to be validated in a **fast** and **automated** way. The unit test suite can be run by just running `make test`. It is also recommended to do manual tests on a **z/OS system** for **new functionality** and **bug fixes** to test the real calls to **IRRSMO00** and **IRRSEQ00**.
+The main way to test RACFu is to write **unit tests** in the [`tests`](tests) directory, which contains **mocks** that mock the real **IRRSMO00** and **IRRSEQ00** RACF callable services to enable **request generation** and **response parsing** logic to be validated in a **fast** and **automated** way. The unit test suite can be run by just running `make test` in the root directory of this repository. It is also recommended to do manual testing on a **z/OS system** for **new functionality** and **bug fixes** to test the real calls to **IRRSMO00** and **IRRSEQ00**.
 
 * **Unit Tests**
 
   > :bulb: _See the [Unity Unit Testing For C](https://www.throwtheswitch.org/unity) documentation for more details on writing test cases._
 
   > :white_check_mark: _In order to facilitate development and unit testing, the real **API calls** to **IRRSMO00** and **IRRSEQ00** have been mocked in [`tests/mock`](tests/mock). Additionally, implementations of some **z/OS specific C/C++ Runtime Library functions** are provided in [`tests/zoslib`](tests/zoslib) to enable the RACFu unit test suite to more or less be run on any 64-bit POSIX system where the `clang` compiler is installed. This ensures that development and testing can be done when contributors do not have access to a z/OS system, and also enables faster iteration since contributors can just run `make test` on their workstation without needing to copy the files to a z/OS system to run the unit tests._
 
-  * Unit tests should be placed in the **subfolder** corresponding to the **RACF callable service** you are creating a test for. The main focus of these tests is to validate the **generation of requests** to and **parsing of responses** from the **IRRSMO00** and **IRRSEQ00** callable services, and more genenerally testing various code paths in the RACFu code. There are directories called `request_samples` and `response_samples` in the [`tests/irrseq00`](tests/irrseq00) and [`tests/irrsmo00`](tests/irrseq00) test folders to put request and response samples. All **raw request samples** and **raw response samples** for a given callable service should end with the `.bin` file extension. `get_raw_sample()` and `get_json_sample()` are defined in [`tests/unit_test_utilities.hpp`](tests/unit_test_utilities.hpp) to facilitate the loading of request and response samples in test cases.
+  * Unit tests should be placed in the **subdirectory** corresponding to the **RACF callable service** you are creating a test for. The main focus of these tests is to validate the **generation of requests** to and **parsing of responses** from the **IRRSMO00** and **IRRSEQ00** callable services, and more genenerally testing various other code paths in the RACFu code. There are directories called `request_samples` and `response_samples` in the [`tests/irrseq00`](tests/irrseq00) and [`tests/irrsmo00`](tests/irrseq00) test folders to put request and response samples. All **raw request samples** and **raw response samples** for a given callable service should end with the `.bin` file extension. `get_raw_sample()` and `get_json_sample()` are defined in [`tests/unit_test_utilities.hpp`](tests/unit_test_utilities.hpp) to facilitate the loading of request and response samples in test cases. Other categories of test cases and test utilities must follow the same conventions described here.
 
-    > _**Example:** A test case for verifying that RACFu can parse the result of an **extract user request** should be placed in the [`test_extract.cpp`](tests/irrseq00/test_extract.cpp) unit test module within the [`irrseq00`](tests/irrseq00) subfolder. A **JSON request** sample containing the parameters for a **profile extract request** should be created in the [`irrseq00/request_samples`](tests/irrseq00/request_samples). A **raw response** sample that contains the **mocked** result of the profile extract request and the corresponding expected **post-processed JSON response** should be created in the [`irrseq00/result_samples`](tests/irrseq00/result_samples). Request/response samples should be loaded in the unit test case using the `get_raw_sample()` and `get_json_sample()` functions defined in [`tests/unit_test_utilities.hpp`](tests/unit_test_utilities.hpp). [`irrseq00.hpp`](tests/mock/irrseq00.hpp) and [`irrsmo64.hpp`](tests/mock/irrsmo64.hpp) provide all of the necessary **global varibales** for mocking the result of requests made to `callRadmin()` and `IRRSMO64()` respectively._
+    > _**Example:** A test case for verifying that RACFu can parse the result of an **extract user request** should be placed in the [`test_irrseq00.cpp`](tests/irrseq00/test_irrseq00.cpp) unit test module within the [`irrseq00`](tests/irrseq00) subdirectory. A **JSON request** sample containing the parameters for a **profile extract request** should be created in the [`irrseq00/request_samples/user`](tests/irrseq00/request_samples/user) directory. A **raw response** sample that contains the **mocked** result of the profile extract request and the corresponding expected **post-processed JSON response** should be created in the [`irrseq00/result_samples/user`](tests/irrseq00/result_samples/user) directory. Request/response samples should be loaded in the unit test case using the `get_raw_sample()` and `get_json_sample()` functions defined in [`tests/unit_test_utilities.hpp`](tests/unit_test_utilities.hpp). [`tests/unit_test_utilities.hpp`](tests/unit_test_utilities.hpp) also provides various other utility functions for facilitating the creation of test cases that should be used when applicable. [`irrseq00.hpp`](tests/mock/irrseq00.hpp) and [`irrsmo64.hpp`](tests/mock/irrsmo64.hpp) provide all of the necessary **global varibales** for mocking the result of requests made to `callRadmin()` and `IRRSMO64()` respectively._
 
 * **Functional Verification Tests**
   > :warning: _Ensure that the `RACFU_FVT_USERID` environment variable is set to a z/OS userid that doesn't exist on the system where the functional verifification tests are being run prior to running `make fvt`._
@@ -111,10 +111,10 @@ When contributing to RACFu, think about the following:
 
 * Make any necessary updates to `pyproject.toml`.
 * Make any necessary updates to `README.md`.
-* Make any necessary updates to the gitHub pages documentation in the `gh-pages` branch _(Pull requests should be opened against the `gh-pages-dev` branch)_.
+* Make any necessary updates to the GitHub pages documentation in the `gh-pages` branch _(Pull requests should be opened against the `gh-pages-dev` branch)_.
 * Add any necessary test cases to `/tests`.
 * Ensure that you have __pre-commit Hooks__ setup to ensure that `clang-format` and **unit tests** are run against the code for every commit you make.
-* Run unit test suite by running `make test`.
+* Run unit tests by running `make test`.
 * Run functional verification tests by running `make fvt`.
 * Run `cppcheck` static code analysis cans by running `make scan`
 

Makefile

@@ -9,9 +9,11 @@ IRRSMO00_SRC	= ${PWD}/racfu/irrsmo00
 IRRSEQ00_SRC	= ${PWD}/racfu/irrseq00
 KEY_MAP			= ${PWD}/racfu/key_map
 LOGGER			= ${PWD}/racfu/logger
+PYTHON			= ${PWD}/racfu/python
 VALIDATION      = ${PWD}/racfu/validation
 EXTERNALS		= ${PWD}/externals
 TESTS			= ${PWD}/tests
+PYTHON_INC		= $(shell python3 -c "import sysconfig; print(sysconfig.get_path('include'))")
 
 ifeq ($(UNAME), OS/390)
 	AS 			= as
@@ -119,6 +121,7 @@ check:
 		--enable=all \
 		--suppress='*:*/externals/*' \
 		--suppress='*:*openxl\*' \
+		--suppress='*:*/include/python*' \
 		--output-file=artifacts/cppcheck/output.xml \
 		--checkers-report=artifacts/cppcheck/checkers_report.txt \
 		--cppcheck-build-dir=artifacts/cppcheck \
@@ -135,13 +138,15 @@ check:
 		-I $(VALIDATION) \
 		-I $(LOGGER) \
 		-I $(EXTERNALS) \
+		-I $(PYTHON_INC) \
 		$(INCZOSLIB) \
 		$(SRC)/*.cpp \
 		$(IRRSMO00_SRC)/*.cpp \
 		$(IRRSEQ00_SRC)/*.cpp \
 		$(KEY_MAP)/*.cpp \
 		$(LOGGER)/*.cpp \
-		$(VALIDATION)/*.cpp
+		$(VALIDATION)/*.cpp \
+		$(PYTHON)/*.c
 
 clean:
 	$(RM) $(ARTIFACTS) $(DIST)

README.md

@@ -10,13 +10,21 @@ While there are a number of languages that can be used to manage RACF, _(from lo
 
 ## Getting Started
 
+
+## Minimum z/OS & Language Versions
+
+All versions of **z/OS** and the **IBM Open Enterprise SDK for Python** that are fully supported by IBM are supported by RACFu.
+* [z/OS Product Lifecycle](https://www.ibm.com/support/pages/lifecycle/search/?q=5655-ZOS,%205650-ZOS)
+* [IBM Open Enterprise SDK for Python Product Lifecycle](https://www.ibm.com/support/pages/lifecycle/search?q=5655-PYT)
+
 ### Dependencies
 
-* z/OS **2.4** or higher.
 * **R_SecMgtOper (IRRSMO00)**: Security Management Operations.
-  * More details about the authorizations required for **IRRSMO00** can be found [here](https://www.ibm.com/docs/en/zos/3.1.0?topic=operations-racf-authorization).
+  * More details about the authorizations required for **IRRSMO00** can be found [here](https://www.ibm.com/docs/en/zos/latest?topic=operations-racf-authorization).
 * **R_Admin (IRRSEQ00)**: RACF Administration API.
-  * More details about the authorizations required for **IRRSEQ00** can be found [here](https://www.ibm.com/docs/en/zos/3.1.0?topic=api-racf-authorization).
+  * More details about the authorizations required for **IRRSEQ00** can be found [here](https://www.ibm.com/docs/en/zos/latest?topic=api-racf-authorization).
+* **RACF Subsystem Address Space**: This is a dependency for both **IRRSMO00** and **IRRSEQ00**.
+  * More information can be found [here](https://www.ibm.com/docs/en/zos/latest?topic=considerations-racf-subsystem).
 
 ### Installation
 
@@ -31,7 +39,7 @@ python3 -m pip install racfu
 
 ## Authors
 
-* Joe Bostian: [email protected]
-* Frank De Gilio: [email protected]
 * Leonard Carcaramo: [email protected]
 * Elijah Swift: [email protected]
+* Frank De Gilio: [email protected]
+* Joe Bostian: [email protected]

dev_tools/README.md

@@ -0,0 +1,27 @@
+## Description
+
+This is a folder where we store tools that were useful for the development of RACFu but do not hold any meaninful code. The files and their natures are explained below in sections based on the purpose of the file.
+
+## Unit Test Generation/Conversion
+
+This/these files were helpful in either generating unit tests or converting files involved in unit tests. RACFu uses a lot of binary files that are ebcdic-encoded xml data as an example, so this was beneficial for making quick adjustments to the files used in these tests.
+
+### convert_dirs.py
+
+This is a python script that contains a number of functions designed to encode/decode files to and from one ebcdic codepage to another (or ascii). These are used in the script to take a folder of xml files in ascii encoding (e.g. `request_xml`) and convert these to binary files in a similarly named folder in IBM-1047 ebcdic (e.g. `request_bin`).
+
+## Key Mapping Generation/Compilation
+
+This/these files were a part of how key mappings for both RACFu and pyRACF were generated. These scripts were helpful tools, but they alone did not generate the final mappings used in RACFu or pyRACF. These tools are, as a result, somewhat incomplete, but including them still felt like it would help if someone were to try and replicate this work.
+
+### load_seg_data.py
+
+Attempts to convert an xlsx Microsoft Excel spreadsheet with the data for the "x administration" callable service tables like in [IBM Documentation](https://www.ibm.com/docs/en/zos/3.1.0?topic=tables-user-administration). Manually copying this data to an "x_admin.xlsx" file allows you to use this script to somewhat convert that to a json file with most of the data needed to generate meaningful key mappings (but not everything). I know for a fact that many booleans were missed, repeat groups weren't properly accounted for and dataset and setropts administration tables were not copied appropriately.
+
+### map_fields.py
+
+Takes the json file generated with the `load_seg_data.py` file and offers functions that attmept to format this into the c++ key_map structure files that RACFu uses. At one point this also generated the key mappings for pyRACF, but I believe those features were pruned to establish the RACFu ones. Takes in the `x_admin.json` files generated in the previous step and generates appropriate c header files. Conceivably this file works well, and just collecting the appropriate json data would enable this to perform its function perfectly.
+
+### generate_doc_tables.py
+
+Takes both the json files generated with the `load_seg_data.py` file and data from the finalized key mapping headers generated by `map_fields.py` along with any changes made post script (as declared in this readme, we made several manual changes) and generates .md files in the intended format of the RACFu documentation for the specific admin type. Does not remove undocumented traits so requires some manual revision, but otherwise, as long as the `key_map_ADMINTYPE.hpp` files are up-to-date and the json files are as functional as they currently are, this does work for programmatic documentation generation.

dev_tools/convert_dirs.py

@@ -0,0 +1,52 @@
+import ebcdic
+import sys
+
+def convert_file(file_name, CCSID_1="ascii", CCSID_2="ascii", output_file_name = "",remove_newline=False,convert_to_bin=True):
+    CCSID_1 = determine_ccsid(CCSID_1) if not CCSID_1 == "ascii" else "UTF-8"
+    CCSID_2 = determine_ccsid(CCSID_2) if not CCSID_2 == "ascii" else "UTF-8"
+
+    if output_file_name == "":
+        output_file_name = f'{file_name.split('.')[0]}_decoded.{file_name.split('.')[1]}'
+
+    if convert_to_bin:
+        output_file_name = f"{output_file_name.split('.')[0]}.bin"
+
+    f = open(file_name, "rb")
+    file_text = f.read().decode(CCSID_1)
+    f.close()
+
+    if remove_newline:
+        file_text = file_text.replace("\n","")
+
+    f = open(output_file_name, "wb")
+    f.write(file_text.encode(CCSID_2))
+    f.close()
+    return True
+
+def convert_directory(directory_name, CCSID_1="ascii", CCSID_2="ascii", output_directory_name="", remove_newline=False):
+    if output_directory_name == "":
+        output_directory_name = f'{directory_name}_decoded'
+    admin_types = ["group_connection", "racf_options", "data_set", "group", "user", "resource", "permission"]
+    for filename in os.listdir(directory_name):
+        admin_dir = ""
+        for admin_type in admin_types:
+            if admin_type in filename:
+                if admin_type == "group" and "group_connection" in filename:
+                    continue
+                admin_dir = f"/{admin_type}"
+        convert_file(f'{directory_name}/{filename}',CCSID_1,CCSID_2,f'{output_directory_name}{admin_dir}/{filename}',remove_newline,convert_to_bin=True)
+
+def determine_ccsid(CCSID):
+    if isinstance(CCSID,int):
+        return f"cp{CCSID:03}"
+    if "cp" in CCSID:
+        return CCSID
+    return "cp"+CCSID
+
+directory = sys.argv[1]
+
+for directory in sys.argv[1:]:
+    print(f"Converting {directory}_xml to EBCDIC in {directory}_bin...")
+    convert_directory(f"{directory}_xml",CCSID_2=1047,output_directory_name=f"{directory}_bin",remove_newline=True)
+
+print("All done!")