Up until very recently almost nobody was worried about encrypting the communication going to and from the mainframe. Why would they, since all the communication was happening “inside the house” so to speak, usually with a much narrower audience than the whole infrastructure, so the window of exposure was considered to be very narrow. Also, since everything was happening “in the house”, the channels and everything was considered as secure by default. In the end, if you can’t trust your own, who can you trust?
Nowadays, everyone is coming around to a zero-trust security model which can be easily summed up to “never trust, always verify”. This usually means that the services and communication channels should be treated as though there is always a bad actor lurking around the corner. A common way to fight against such bad actors on the network is to encrypt the communication channels and simultaneously force all communication partners (both the servers and clients) to continuously authenticate themselves to one another. TLS is a very nifty way to achieve both the encrypt and authenticate portions of the network communication, since it uses digital certificates for authentication and asymmetric cryptography to encrypt the contents of communication. In a standard implementation, it is usually the server who identifies itself to the client using a digital certificate which was issued by a mutually trusted party, the public key of the server is used to establish initial encryption on the channel and then the client identifies itself usually using a token or a password or something similar.
In CROZ we have a small learning environment (1 CP and 1 zIIP) which we use for our internal training and getting familiar with various concepts and ideas on the mainframe (especially new ones). As such, this system really does not need any kind of special protection since nothing sensitive is contained there (maybe some badly written REXX scripts), so we never bothered to implement TLS, since who would want bother with eavesdropping just to see us trying out a new feature in IMS or CICS. This system is an emulated one (zPDT) which also comes with a prebuilt z/OS image with anything you can think of when you think about z/OS products.
As a fun learning experience, we have recently decided to implement TLS anywhere and everywhere on z/OS and try to tighten everything up, so that the whole environment is tight as a drum, so this blog will be describing our adventures with setting up Policy agent. An important thing to note is that a lot of these steps could be and would be easier if done via z/OSMF utilities, but since we don’t have the luxury of enough zIIPs which could offload the processing and not hog the CP. Also, if we get the chance, this might become a series of blog posts which will describe the rest of the setup.
AT-TLS introduction
Since a lot of the applications on z/OS were written well before TLS was even an idea, IBM has recognized this – those applications and services today need TLS, and it would be very hard to retrofit them with TLS support. So, they devised a way where z/OS takes the brunt of the change while the applications continue to work as though nothing has changed. It’s called Application Transparent TLS. When AT-TLS is configured, the application communicates with its partner using normal TCP stack functions while the TCP stack takes care of setting up, controlling and terminating the TLS session. In some cases, the application can choose to inspect and/or control the TLS session settings which is known as a “application-aware” or “application-controlled” session behavior.
The decision which connection and session should use TLS, and which session should be left “as-is” is decided by the AT-TLS policy which is implemented by Policy Agent that communicates with TCPIP stack.
Implementation of Policy Agent and AT-TLS
The first step is to copy and modify the sample started task procedure in EZAPAGSP member in the SEZAINST library to an appropriately named member of our PROCLIB concatenation.
One of the important decisions to make is whether to keep the configuration in MVS or in UNIX files. We decided to keep it simple and all together in TCPPARMS PDS so that it is easier to check and cross-reference the configuration parameters while working with TCPIP. For logging we chose to redirect everything to a file in /tmp directory, but for production purposes SYSLOGD is the way to go. If you don’t like dealing with OMVS files, you can always configure SYSLOGD to output important messages to the console and SYSLOG, but that’s a topic for another time and another blog.
As far as the modification of STC procedure goes, it is fairly straightforward, but you do need to consider that the PARM limit is 100 characters, so if you have a lot of environment variables, it might be prudent to put them in a separate file and point to that file via ENVAR(“_CEE_ENVFILE_S “) statement in EXEC PARM. As far as other runtime parameters are concerned, they are fairly straightforward, and if you can’t fit them inside of the EXEC PARM, you can also specify them via environment variables, which is nice. To keep things simple, we chose to specify our parameters via single in-stream environment variables. This is what we ended up with (with comments removed):
//PAGENT PROC
//PAGENT EXEC PGM=PAGENT,
// REGION=0K,
// TIME=NOLIMIT,
// PARM='ENVAR("_CEE_ENVFILE_S=DD:STDENV")/'
//STDENV DD *
_CEE_ENVFILE_COMMENT=#
PAGENT_CONFIG_FILE=DD:CONFIG
PAGENT_LOG_FILE=/tmp/pagent.log
/*
//CONFIG DD DSN=USER.Z31A.TCPPARMS(PAGENT),
// DISP=SHR
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//CEEDUMP DD SYSOUT=*,
// RECFM=FB,
// LRECL=132,
// BLKSIZE=132
That is all JCL that you need to start up Policy Agent by itself. It probably can be simplified even more by keeping the in-stream data in a separate PDS member, but this allows for quicker referencing and checking. Do keep in mind that you need to have your ESM definitions ready to assign a proper user to the started task. For that purpose, few simple RACF definitions are enough:
ADDUSER PAGENT NOPASSWORD NOPHRASE DFLTGRP(OMVSGRP) +
OMVS(UID(0) HOME('/'))
RDEFINE STARTED PAGENT.** STDATA(USER(=MEMBER))
SETROPTS RACLIST(STARTED) REFRESH
One thing that surprised me was that PAGENT requires UID 0 to automatically monitor the applications.
The next part involves reconfiguring TCPIP stack to recognize that it has policies to follow. In your TCPIP.PROFILE file you need to add the statement:
TCPCONFIG TTLS
After that, a general configuration file for Policy Agent needs to be written. This file specifies where the actual policy definitions for each image are contained. If you have a single TCPIP stack, the file can be very simple:
TcpImage TCPIP CommonTTLSConfig //'USER.Z31A.TCPPARMS(TTLSCONF)' TTLSConfig //'USER.Z31A.TCPPARMS(TTLSCONF)' FLUSH PURGE
After that, it comes to the policy definitions themselves. They must all be contained in a single file which is very troublesome if you have a large setup, and you need to make a change. When you are editing a single large file, it is very hard to keep everything logically organized and when you are making a change, it is always possible to make a mistake on something totally unrelated to the part you were trying to edit. If you could include separate files into the configuration like with TN3270 configuration, you could have a single file for Db2 setup, separate one for TN3270 TLS settings, separate one for CICS etc., and then when you need to make a change, you know that your changes related to a single file don’t accidentally spill over to something else because you mistyped or repositioned your cursor accidentally.
AT-TLS policy making
As we’ve stated previously, the policies and everything else related to actual configuration of PAGENT is probably a lot easier if you have (and like using) z/OSMF. Since we don’t have that luxury at the moment, and as it is always worth knowing how a “clickety-click wizard” works under the hood, it’s good practice to at least try to write out a single policy from scratch, just for testing at least. But first things first, not necessarily in that order.
We decided to cut our teeth by setting up Db2 DRDA connection with TLS encryption and client authentication. A bit ambitious, but not that much. When dealing with a setup from scratch, you need to plan and prepare everything and think about the questions that might not be that obvious at first:
- Do we have a PKI infrastructure that we can already use?
- How old or new are our clients that are going to connect to Db2? What kind of ciphers and signatures do they support?
- What version of z/OS are we running? What kind of ciphers and signatures does it support?
- How are we going to distribute the client certificates? How long should they be valid?
- What happens in case of certificate revocation?
We decided to implement a small RACF-based PKI, which is sufficient for a lab or a very small production environment. Some time in the future we might switch over to fully fledged z/OS PKI Services, but at this moment for a mere 10-20 certificates it would be an overkill.
When setting up z/OS RACF PKI, same as with any other PKI, you start with a self-signed Root CA certificate:
RACDCERT GENCERT +
CERTAUTH +
SUBJECTSDN(+
CN('KIT ROOT CA') +
OU(+
'MAINFRAME ODJEL',+
'ODJEL ZA INFRASTRUKTURNA RJESENJA'+
) +
O('CROZ') +
L('ZAGREB') +
SP('GRAD ZAGREB') +
C('HR')+
) +
SIZE(4096) +
NOTBEFORE(+
DATE(2024-08-30) +
TIME(00:00:00) +
) +
NOTAFTER(+
DATE(2054-08-30) +
TIME(00:00:00) +
) +
WITHLABEL('CROZMFROOTCA') +
RSA +
KEYUSAGE(CERTSIGN)
This will create a certificate that is used for signing other certificates. It will have an RSA private key of 4096 bits, and it will last for the next 30 years. Hopefully by that time we will remember to renew it. 😊 It is a shame that we can’t have larger key sizes natively generated by RACF, but based on current recommendations 4096 bits is still fine for all uses. Maybe things will change when we become “quantum-endangered” by an actual quantum computer.
Once we have this certificate, which will be used to sign other certificates, we need to distribute it to our clients since they need to import it into their trust stores. Java has its’ own trust store procedure using keytool, while for Windows, you just double click the file and go next-next-next.
Nevertheless, if our clients want to import it, we first need to export it and make it available to them via FTP or some other means:
RACDCERT +
EXPORT(+
LABEL('CROZMFROOTCA') +
) +
CERTAUTH +
DSN('MY.OWN.CA.CERT.DER') +
FORMAT(CERTDER)
After that is done, we can choose to create an intermediate CA which will then issue the certificate for Db2 and clients, or we can create and sign the actual server certificate that will represent the Db2 itself, and it will be used to identify Db2 to its’ clients. This certificate can be signed directly by Root CA, without an intermediate, and it is always a good idea to include all of the possible IP addresses and host names that clients might use to connect to Db2 as subject alternate names (SANs).
RACDCERT GENCERT +
ID(DB2USER) +
SUBJECTSDN(+
CN('DB2 DBCG') +
OU(+
'BAZE PODATAKA',+
'KIT',+
'MAINFRAME ODJEL',+
'ODJEL ZA INFRASTRUKTURNA RJESENJA'+
) +
O('CROZ D.O.O.') +
L('ZAGREB') +
SP('GRAD ZAGREB') +
C('HR')+
) +
SIZE(4096) +
SIGNWITH( +
CERTAUTH +
LABEL('CROZMFROOTCA') +
) +
NOTBEFORE(+
DATE(2024-08-30) +
TIME(00:00:00) +
) +
NOTAFTER(+
DATE(2034-08-30) +
TIME(00:00:00) +
) +
WITHLABEL('DB2_DBCG_CERT') +
RSA +
KEYUSAGE(+
HANDSHAKE +
DATAENCRYPT +
) +
ALTNAME( +
IP(10.0.0.40) +
DOMAIN('kit.lan.croz.net')+
)
This certificate should be owned by the user that is running Db2 started tasks. Next step is to create a keyring which will be used by AT-TLS. It will have all of the CA certificates that were in the chain (in our case only the Root CA) and our actual server certificate.
RACDCERT +
ADDRING(DB2_RING) +
ID(DB2USER)
RACDCERT CONNECT(+
CERTAUTH +
LABEL('CROZMFROOTCA') +
RING(DB2_RING) +
USAGE(CERTAUTH)+
) +
ID(DB2USER)
RACDCERT CONNECT(+
ID(DB2USER) +
LABEL('DB2_DBCG_CERT') +
RING(DB2_RING) +
DEFAULT +
USAGE(PERSONAL)+
) +
ID(DB2USER)
Now that we have our keyring, we can go back to our policy making… When writing the policy rules, one starts with TTLSRule which defines who is affected by that policy and what are the actions that will be undertaken for that rule. Rules can be specified in roughly three actions statements, and they are processed and honored in a hierarchical order:
- TTLSConnectionAction
- TTLSEnvironmentAction
- TTLSGroupAction
- Predefined default value
- No value is explicitly used by AT-TLS and System SSL
This order of processing allows us to generalize and not to repeat the common settings like cipher suites and signature suites that are used. The rules are fortunately very easy to write, and understandable. Following is the set of rules that are needed to implement basic Db2 DRDA TLS protection with client authentication:
TTLSRule Db2_DBCG_Server
{
LocalPortRange 4081
JobName DBCGDIST
Direction Inbound
Priority 1
TTLSGroupActionRef GEN_Group_Action
TTLSEnvironmentActionRef DB2_Environment_Action
}
TTLSEnvironmentAction DB2_Environment_Action
{
TTLSKeyRingParmsRef DB2_Keyring_Parms
HandShakeRole ServerWithClientAuth
TTLSCipherParmsRef GEN_Cipher_Parms
TTLSSignatureParmsRef GEN_Signature_Parms
TTLSEnvironmentAdvancedParmsRef DB2_Env_Advanced_Parms
#Trace 255
}
TTLSKeyRingParms DB2_Keyring_Parms
{
Keyring DB2_RING
}
TTLSEnvironmentAdvancedParms DB2_Env_Advanced_Parms
{
TLSv1.3 On
TLSv1.2 On
TLSv1.1 Off
TLSv1 Off
SSLv3 Off
SSLv2 Off
ClientAuthType SAFCheck
}
We start off with a rule that says, “all incoming connections to TCP port 4081 and where DBCGDIST is listening on that port are covered by GEN_Group_Action and DB2_Environment_Action”. You can really play around with rule filters and even specify time of day or day of week when the rule applies. Since a single connection can match multiple rules, priority defines which rule applies, meaning that the highest priority rule is honored.
After that, we move on to TTLSEnvironmentAction DB2_Environment_Action which tells which keyring will be used, what ciphers and signatures will be used and what version of TLS will be used. All those parameters are enclosed in separate structures which allows for reuse of same settings between rules and environments. For some parameters, we set the value directly such as HandShakeRole that says this environment will be for a server on z/OS side that will perform client authentication.
As far as the Db2 connection-specific parameters are concerned, we are specifying the keyring name here, and, because we do not explicitly specify the owner of the keyring, it is implied that it will be the user who is running the DBCGDIST started task. If we did not have multiple Db2 subsystems sharing the same user while having separate certificates, we could use here the virtual keyring feature of RACF, but that is much more practical in the client role of AT-TLS for CA certificate keyring. We also specified here via TTLSEnvironmentAdvancedParms DB2_Env_Advanced_Parms, that we only want TLS v1.2 and v1.3 which are still considered secure, and that we want for the client which is connecting not just to have a certificate that is trusted by our CA from the keyring, but also to have a certificate which RACF can map to a single user.
There are several parameters which reference blocks whose names begin with GEN. Those settings are shared between several different rules (like TN3270, FTP and others) and they are listed below for clarity. We allow all TLS v1.3 and v1.2 ciphers for use, and all possible signature ciphers.
TTLSGroupAction GEN_Group_Action
{
TTLSEnabled On
#Trace 255
}
TTLSCipherParms GEN_Cipher_Parms
{
V3CipherSuites TLS_AES_128_GCM_SHA256
V3CipherSuites TLS_AES_256_GCM_SHA384
V3CipherSuites TLS_CHACHA20_POLY1305_SHA256
V3CipherSuites TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
V3CipherSuites TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
V3CipherSuites TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
V3CipherSuites TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
}
TTLSSignatureParms GEN_Signature_Parms
{
ClientECurves secp224r1
ClientECurves secp256r1
ClientECurves secp384r1
ClientECurves secp521r1
ClientECurves secp192r1
ClientECurves x25519
ClientECurves x448
ClientKeyShareGroups secp256r1
ClientKeyShareGroups secp384r1
ClientKeyShareGroups secp521r1
ClientKeyShareGroups x25519
ClientKeyShareGroups x448
SignaturePairs TLS_SIGALG_SHA512_WITH_RSA
SignaturePairs TLS_SIGALG_SHA512_WITH_ECDSA
SignaturePairs TLS_SIGALG_SHA384_WITH_RSA
SignaturePairs TLS_SIGALG_SHA384_WITH_ECDSA
SignaturePairs TLS_SIGALG_SHA256_WITH_RSA
SignaturePairs TLS_SIGALG_SHA256_WITH_ECDSA
SignaturePairs TLS_SIGALG_SHA256_WITH_DSA
SignaturePairs TLS_SIGALG_SHA224_WITH_RSA
SignaturePairs TLS_SIGALG_SHA224_WITH_ECDSA
SignaturePairs TLS_SIGALG_SHA224_WITH_DSA
SignaturePairs TLS_SIGALG_SHA1_WITH_RSA
SignaturePairs TLS_SIGALG_SHA1_WITH_ECDSA
SignaturePairs TLS_SIGALG_SHA1_WITH_DSA
SignaturePairs TLS_SIGALG_SHA512_WITH_RSASSA_PSS
SignaturePairs TLS_SIGALG_SHA384_WITH_RSASSA_PSS
}
After the policy is done, and saved into the correct file, it is time to make the Policy Agent aware of the new policy. The easiest way to do that is of course to restart PAGENT if it is running, but since that can be very disruptive, there are two ways to dynamically reload the configuration. The first one is MODIFY PAGENT,UPDATE , while the second one is MODIFY PAGENT,REFRESH . Now you might think that these two are the same, but they have subtle differences between them with regards to FLUSH and PURGE parameters in Policy Agent configuration file. There is a nice table that explains how Policy Agent acts with regards to the various combinations of the three parameters on the following link in IBM Documentation. Do be careful about that. For our purposes, since no one is dependent on our Db2 connections staying up 24/7/365, we can safely recycle PAGENT without worries.
Db2 configuration
As far as Db2 is concerned, you just need to decide on which port it will use for secure communication and make sure that that same port is specified as the LocalPortRange in your TTLSRule. Change of the port is done by changing the BSDS using DSNJU003 and setting the correct SECPORT for the DDF location of your database instance.
//DSNJU003 EXEC PGM=DSNJU003 //SYSUT1 DD DSN=DSNC10.DBCG.BSDS01, // DISP=SHR //SYSUT2 DD DSN=DSNC10.DBCG.BSDS02, // DISP=SHR //SYSPRINT DD SYSOUT=* //SYSIN DD * DDF LOCATION=KIT12 DDF PORT=446 DDF SECPORT=4081 /*
It would be nice to have Db2 listen on more than one port for unsecured and one for secured communications, but we digress. Once again, do be very careful that the SECPORT value matches the value in policy configuration (LocalPortRange).
Client-side configuration
Now, when talking about clients, things can be either very easy, or unbelievably difficult, depending on the variety of your clients. If all your Db2 clients are, for example, Windows clients which are centrally managed using Active Directory, you could just generate the necessary client certificates in RACF, export them out securely with their private keys, and push them via AD to each respective client. Then depending on the programming language or environment which is used, you just provide a correct configuration to use the Windows trust store and voila. If you are using a mix and match of everything (various OS-es, various Db2 drivers), you need to consider what is the easiest way to deploy the certificates. When planning, do think about the certificate validity and how you are going to replace them once they expire, and don’t treat the deployment as a one-time thing.
We chose to test the setup of Java client connecting to Db2 using both Windows trust store and Java keystore. To that end, we first started off with a Java keystore since that is a platform-neutral implementation. It is fairly simple to set up a new keystore that will be used both as a trust store and a key store and by a single client only:
keytool -importcert -trustcacerts -file CACERT.DER -keystore testtruststore.p12 -alias croz_mf_root_ca
This command will create a new PKCS#12 keystore (if it doesn’t exist already), and import our CA certificate under the alias croz_mf_root_ca. You will be prompted to specify the protection password for the keystore if you don’t specify it via -storepass argument. When importing the client certificate, it is best that it is exported from RACF using a PKCS#12 format so that you don’t have to “recombine” the private and public key into a single package for import. PKCS#12 files are already Java keystores since Java 9, so you can use the original .p12 or .pfx file as-is as a keystore, or you can import the certificate into the keystore that we created previously. If you want to contain everything in a single file, you just need to run a command like so:
keytool -importkeystore -srckeystore PERSONAL.P12 -srcstorepass MYPKCS12PASSWORD -destkeystore testtruststore.P12 -deststorepass MYKEYSTOREPASSWORD
If you are unsure about using keytool, KeyStore Explorer developed by Wayne Grant and Kai Kramer is an excellent open-source GUI tool for managing Java keystores in all shapes and formats.
Windows implementation is much easier since you just need to double click the certificate and follow the steps in the wizard.
OK, we stored the certificates, but how do we tell Db2 driver that it should use them and where it can find them? If you are using JDBC drivers, it all can be specified in a single JDBC string:
jdbc:db2://kit.lan.croz.net:4081/KIT12:securityMechanism=18;sslConnection=true;sslTrustStoreType=Windows-ROOT-CURRENTUSER;sslTrustStoreLocation=NUL;sslKeyStoreType=Windows-MY;sslKeyStoreLocation=NUL;
The first part of the connection string is familiar to anyone who has connected to Db2 using JDBC. You specify the host name, followed by the port and location name and Bob’s your uncle. With TLS and client authentication you need to add a few more properties in a format of key=value, and delimited by a semi-colon. The last parameter also needs to have a semicolon!
Let’s explain the parameters:
- sslConnection=true – we want to use a TLS encrypted connection (predecessor of TLS was SSL)
- sslTrustStoreType=Windows-ROOT-CURRENTUSER – where can the Db2 driver find the CA certificate which it can use to see if the server is to be trusted. In our case, we want it to check the Trusted Root Certification Authorities certificate store of the current Windows user. If you want to use a Java keystore file as a trust store, omit this parameter completely.
- sslTrustStoreLocation=NUL – necessary to specify when using Windows trust store. If you are using a Java keystore as a trust store, this is where you specify its’ file name. In that case you also need to set sslTrustStorePassword property.
- sslKeyStoreType=Windows-MY-CURRENTUSER – where can the Db2 driver find the client certificate which it can use to authenticate us. In our case, we want it to check the Personal certificate store of the current Windows user. If you want to use a Java keystore file as a key store, omit this parameter completely.
- sslKeyStoreLocation=NUL – necessary to specify when using Windows key store. If you are using a Java keystore as a key store, this is where you specify its’ file name. In that case you also need to set sslKeyStorePassword property.
- securityMechanism=18 – we are using TLS_CLIENT_CERTIFICATE_SECURITY mechanism which is defined as having value of 18. Look into Table 2 at the following IBM Documentation topic for all possible values.
For clarity, this is how would an equivalent JDBC connection string look if we are using Java keystore files:
jdbc:db2://kit.lan.croz.net:4081/KIT12:securityMechanism=18;sslConnection=true;sslTrustStoreLocation=/home/testuser/keystore.p12;sslKeyStoreLocation=/home/testuser/keystore.p12;sslTrustStorePassword=MYKEYSTOREPASSWORD;sslKeyStorePassword=MYKEYSTOREPASSWORD;
Also, did you know that if you are running your Java applications on z/OS you can use similarly easy use RACF keyrings to store the certificates? Stay tuned because this might become a small blog post in the future.
What about errors?
Encryption is notoriously difficult to debug and troubleshoot sometimes, since you can’t clearly see with tools like Wireshark what exactly is going on. Policy Agent supports setting a Trace parameter in TTLSEnvironmentAction and TTLSGroupAction to allow us to diagnose errors. We had some issues that turned out to be caused by existing certificates from ADCD distribution mixing up with our own certificates and expecting the system to take our certificates as default when choosing one, if we didn’t specify exactly what certificate or keyring to use. Trace 255 parameter proved to be a lifesaver in those cases, along with Wireshark which can still see into the very beginning of the handshake, and you can see if there is something in the handshake parameters that doesn’t feel or look right.
If that combination is not enough, you do need to set up a GTF trace for GSKSRVR (which also means you need to set up GSKSRVR) to trace the System SSL, and no, setting the GSK trace environment variables in PAGENT STC unfortunately will not be enough. Tips and tricks about that fun little side-quest might come sometime later.
Falls Sie Fragen haben, sind wir nur einen Klick entfernt.
