Building a DLL with clang

[khickel]

I'm working on porting librdkafka using clang, and have run into a few challenges, this one is about building a shared library with clang.
One of the test Makefiles tries to build a shared library with the command and errors below:

clang -Wl,-bedit=no -pthread -Wno-unused-command-line-argument -L~/zopen/prod/ncurses-6.3/lib -L../../src -shared interceptor_test.o -o interceptor_test.so.1 -lm -pthread  -lncurses -lrdkafka
 IEW2456E 9207 SYMBOL test_level UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM 
          THE DESIGNATED CALL LIBRARY. NAME SPACE = 3                           
 IEW2456E 9207 SYMBOL test_curr UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM  
          THE DESIGNATED CALL LIBRARY. NAME SPACE = 3                           
 IEW2456E 9207 SYMBOL ictest UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM THE 
          DESIGNATED CALL LIBRARY. NAME SPACE = 3                               
 IEW2456E 9207 SYMBOL test_fail0 UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM 
          THE DESIGNATED CALL LIBRARY.                                          
 IEW2456E 9207 SYMBOL test_conf_get UNRESOLVED.  MEMBER COULD NOT BE INCLUDED   
          FROM THE DESIGNATED CALL LIBRARY.                                     
 IEW2665S 40FF MODULE *NULL*  IS NON-EXECUTABLE AND WAS NOT SAVED BECAUSE       
          STORENX=NEVER.                                                        
IEW5033 The binder ended with return code 12.
ibm-clang: error: linker command failed with exit code 12 (use -v to see invocation)
make: *** [../../mklove/Makefile.base:110: interceptor_test.so.1] Error 12	

re-running the clang command with -v shows the following /bin/ld command being executed, with -e, -O and -u flags that seem to be incorrect to produce a shared library:
"/bin/ld" -o interceptor_test.so.1 -b AMODE=64,LIST,DYNAM=DLL,MSGLEVEL=4,CASE=MIXED,REUS=RENT -e CELQSTRT -O CELQSTRT -u CELQMAIN -x interceptor_test.so.x -L~/zopen/prod/ncurses-6.3/lib -L../../src -L/Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/bin/../lib/s390x-ibm-zos -bedit=no interceptor_test.o -lm -lncurses -lrdkafka -S //'CEE.SCEEBND2' -S //'SYS1.CSSLIB' //'CEE.SCEELIB(CELQS001)' //'CEE.SCEELIB(CELQS003)' /Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/lib/clang/14.0.0/lib/s390x-ibm-zos/libclang_rt.builtins.a

Running that ld command separately, having removed the -e, -O and -u flags, and added the -bdyam=dll and -bncal flags finishes without error:
"/bin/ld" -o interceptor_test.so.1 -b AMODE=64,LIST,DYNAM=DLL,MSGLEVEL=4,CASE=MIXED,REUS=RENT -x interceptor_test.so.x -L~/zopen/prod/ncurses-6.3/lib -L../../src -L/Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/bin/../lib/s390x-ibm-zos -bedit=no -bdynam=dll -bncal interceptor_test.o -lm -lncurses -lrdkafka -S "//'CEE.SCEEBND2'" -S "//'SYS1.CSSLIB'" "//'CEE.SCEELIB(CELQS001)'" "//'CEE.SCEELIB(CELQS003)'" /Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/lib/clang/14.0.0/lib/s390x-ibm-zos/libclang_rt.builtins.a

I don't yet know if the .so that was produced is a good one, but the question here is how to get clang to invoke ld with what I currently believe is the correct set of flags?

Note: When I say I'm using clang, I'm actually using a script wrapper that runs "ibm-clang -m64" instead of "clang", I don't know if, how or why that would matter, but I wanted to mention it just in case it does.

Thanks,
Kelly

[IgorTodorovskiIBM]

@perry-ca is there a clang equivalent to -qDLL? Is there a clang option to avoid setting these ld options: -e CELQSTRT -O CELQSTRT -u CELQMAIN?

[perry-ca]

Yep, -shared.

However, the xlclang option is really -Wl,DLL. The -qdll option is for 32-bit non-xplink mode to generate code that works in a dll. It's kind of like the -fPIC but with more.

[khickel]

The command is already using -shared, adding -Wl,DLL does not get rid of the options that are causing the issue: -e CELQSTRT -O CELQSTRT -u CELQMAIN .

[perry-ca]

The -Wl,DLL is only for the XL compiler.

In your ld command that worked. Can you try it again without the -bdynam=dll (already specified) and the -bncal options. Why did you add -bncal? Can you also add the -v option. I want to see if ld is passing the default values to the binder for the -e/-O/-u options. Another experiment to try is adding -Wl,-bncal to the clang command. Does that work?

[khickel]

I believe that the -bdynam=dll isn't really needed because it was already handled by the -shared, but here is the example of running clang -v with that:

bash-5.2$ clang -v -Wl,-bedit=no -Wl,-bdynam=dll -pthread -Wno-unused-command-line-argument -L../../../../../prod/ncurses-6.3/lib -L../../src -shared interceptor_test.o -o interceptor_test.so.1 -lm -pthread  -lncurses -lrdkafka
IBM Open XL C/C++ 1.1 for z/OS, (5650-ZOS), clang version 14.0.0
Target: s390x-ibm-zos
Thread model: posix
InstalledDir: /usr/lpp/IBM/cnw/v1r1/openxl/bin
 "/bin/ld" -o interceptor_test.so.1 -b AMODE=64,LIST,DYNAM=DLL,MSGLEVEL=4,CASE=MIXED,REUS=RENT -e CELQSTRT -O CELQSTRT -u CELQMAIN -x interceptor_test.so.x -L../../../../../prod/ncurses-6.3/lib -L../../src -L/Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/bin/../lib/s390x-ibm-zos -bedit=no -bdynam=dll interceptor_test.o -lm -lncurses -lrdkafka -S //'CEE.SCEEBND2' -S //'SYS1.CSSLIB' //'CEE.SCEELIB(CELQS001)' //'CEE.SCEELIB(CELQS003)' /Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/lib/clang/14.0.0/lib/s390x-ibm-zos/libclang_rt.builtins.a
 IEW2456E 9207 SYMBOL test_level UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM 
          THE DESIGNATED CALL LIBRARY. NAME SPACE = 3                           
 IEW2456E 9207 SYMBOL test_curr UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM  
          THE DESIGNATED CALL LIBRARY. NAME SPACE = 3                           
 IEW2456E 9207 SYMBOL ictest UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM THE 
          DESIGNATED CALL LIBRARY. NAME SPACE = 3                               
 IEW2456E 9207 SYMBOL test_fail0 UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM 
          THE DESIGNATED CALL LIBRARY.                                          
 IEW2456E 9207 SYMBOL test_conf_get UNRESOLVED.  MEMBER COULD NOT BE INCLUDED   
          FROM THE DESIGNATED CALL LIBRARY.                                     
 IEW2665S 40FF MODULE *NULL*  IS NON-EXECUTABLE AND WAS NOT SAVED BECAUSE       
          STORENX=NEVER.                                                        
IEW5033 The binder ended with return code 12.
ibm-clang: error: linker command failed with exit code 12 (use -v to see invocation)
bash-5.2$ 

I came up with -bncal based on the errors that I was seeing and the binder doc, adding just -Wl,-bncal to the clang command reduces the errors to just the ones related to CELQSTRT and CELQMAIN, here is the example:

bash-5.2$ clang -v -Wl,-bedit=no -Wl,-bncal -pthread -Wno-unused-command-line-argument -L../../../../../prod/ncurses-6.3/lib -L../../src -shared interceptor_test.o -o interceptor_test.so.1 -lm -pthread  -lncurses -lrdkafka
IBM Open XL C/C++ 1.1 for z/OS, (5650-ZOS), clang version 14.0.0
Target: s390x-ibm-zos
Thread model: posix
InstalledDir: /usr/lpp/IBM/cnw/v1r1/openxl/bin
 "/bin/ld" -o interceptor_test.so.1 -b AMODE=64,LIST,DYNAM=DLL,MSGLEVEL=4,CASE=MIXED,REUS=RENT -e CELQSTRT -O CELQSTRT -u CELQMAIN -x interceptor_test.so.x -L../../../../../prod/ncurses-6.3/lib -L../../src -L/Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/bin/../lib/s390x-ibm-zos -bedit=no -bncal interceptor_test.o -lm -lncurses -lrdkafka -S //'CEE.SCEEBND2' -S //'SYS1.CSSLIB' //'CEE.SCEELIB(CELQS001)' //'CEE.SCEELIB(CELQS003)' /Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/lib/clang/14.0.0/lib/s390x-ibm-zos/libclang_rt.builtins.a
 IEW2454W 9203 SYMBOL CELQMAIN UNRESOLVED.  NO AUTOCALL (NCAL) SPECIFIED.       
 IEW2470E 9511 ORDERED SECTION CELQSTRT NOT FOUND IN MODULE.                    
 IEW2665S 40FF MODULE *NULL*  IS NON-EXECUTABLE AND WAS NOT SAVED BECAUSE       
          STORENX=NEVER.                                                        
IEW5033 The binder ended with return code 12.
ibm-clang: error: linker command failed with exit code 12 (use -v to see invocation)
bash-5.2$ 

[perry-ca]

Sorry, I meant -v with the ld commands. Can you add the -v to your ld command that works and try that ld command without the -bncal (with -v).

[khickel]

The working ld command with -v added:

bash-5.2$ "/bin/ld" -o interceptor_test.so.1 -b AMODE=64,LIST,DYNAM=DLL,MSGLEVEL=4,CASE=MIXED,REUS=RENT  -x interceptor_test.so.x -L../../../../../prod/ncurses-6.3/lib -L../../src -L/Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/bin/../lib/s390x-ibm-zos  -bedit=no -bdynam=dll -bncal interceptor_test.o -lm -lncurses -lrdkafka -S "//'CEE.SCEEBND2'" -S "//'SYS1.CSSLIB'" "//'CEE.SCEELIB(CELQS001)'" "//'CEE.SCEELIB(CELQS003)'" /Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/lib/clang/14.0.0/lib/s390x-ibm-zos/libclang_rt.builtins.a 
bash-5.2$ pwd
/home/mvskxh/zopen/dev/librdkafkaport/librdkafka/tests/interceptor_test
bash-5.2$ 
bash-5.2$ 
bash-5.2$ pwd
/home/mvskxh/zopen/dev/librdkafkaport/librdkafka/tests/interceptor_test
bash-5.2$ "/bin/ld" -v -o interceptor_test.so.1 -b AMODE=64,LIST,DYNAM=DLL,MSGLEVEL=4,CASE=MIXED,REUS=RENT  -x interceptor_test.so.x -L../../../../../prod/ncurses-6.3/lib -L../../src -L/Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/bin/../lib/s390x-ibm-zos  -bedit=no -bdynam=dll -bncal interceptor_test.o -lm -lncurses -lrdkafka -S "//'CEE.SCEEBND2'" -S "//'SYS1.CSSLIB'" "//'CEE.SCEELIB(CELQS001)'" "//'CEE.SCEELIB(CELQS003)'" /Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/lib/clang/14.0.0/lib/s390x-ibm-zos/libclang_rt.builtins.a 
//* ld  --------------------------------------------------------------- 
//LINKEDIT EXEC PGM=IEWBLINK, 
// PARM='TERM=YES,PRINT=NO,MSGLEVEL=4,STORENX=NEVER,AMODE=64,LIST,DYNAMc
//             =DLL,MSGLEVEL=4,CASE=MIXED,REUS=RENT,edit=no,dynam=dll,nc
//             cal' 
//SYSLIB   DD  DSN=CEE.SCEEBND2,DISP=SHR,DCB=DSORG=DIR 
//         DD  DSN=SYS1.CSSLIB,DISP=SHR,DCB=DSORG=DIR 
//LDIN0001 DD  DSN=CEE.SCEELIB(CELQS001),DISP=SHR 
//LDIN0002 DD  DSN=CEE.SCEELIB(CELQS003),DISP=SHR 
//SYSLMOD  DD  PATH='./interceptor_test.so.1',PATHOPTS=(OWRONLY,OCREAT)c
//             ,PATHMODE=(SIRWXO,SIRWXG,SIRWXU) 
//SYSDEFSD DD  PATH='./interceptor_test.so.x',PATHOPTS=(OWRONLY,OCREAT)c
//             ,PATHMODE=(SIRWXO,SIRWXG,SIRWXU) 
//SYSPRINT DD  PATH='/dev/fd1',PATHOPTS=(ORDWR,OCREAT,OAPPEND),FILEDATAc
//             =TEXT,PATHMODE=(SIROTH,SIRGRP,SIRUSR,SIWOTH,SIWGRP,SIWUSc
//             R) 
//SYSTERM  DD  PATH='/dev/fd2',PATHOPTS=(ORDWR,OCREAT,OAPPEND),FILEDATAc
//             =TEXT,PATHMODE=(SIROTH,SIRGRP,SIRUSR,SIWOTH,SIWGRP,SIWUSc
//             R) 
//SYSLIN   DD  * 
 INCLUDE  './interceptor_test.o'                                                
 LIBRARY  '/usr/lib/libm.a'                                                     
 LIBRARY  './../../../../../prod/ncurses-6.3/lib/libncurses.a'                  
 LIBRARY  './../../src/librdkafka.a'                                            
 INCLUDE  LDIN0001                                                              
 INCLUDE  LDIN0002                                                              
 LIBRARY  '/Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/lib/clang/14.0.0/lib/s39c        
 0x-ibm-zos/libclang_rt.builtins.a'                                             
/*
bash-5.2$ 

Same command without the -bncal, this .so would be linked into some code that provided these symbols, so they aren't present when building the .so:

bash-5.2$ "/bin/ld" -v -o interceptor_test.so.1 -b AMODE=64,LIST,DYNAM=DLL,MSGLEVEL=4,CASE=MIXED,REUS=RENT  -x interceptor_test.so.x -L../../../../../prod/ncurses-6.3/lib -L../../src -L/Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/bin/../lib/s390x-ibm-zos  -bedit=no -bdynam=dll interceptor_test.o -lm -lncurses -lrdkafka -S "//'CEE.SCEEBND2'" -S "//'SYS1.CSSLIB'" "//'CEE.SCEELIB(CELQS001)'" "//'CEE.SCEELIB(CELQS003)'" /Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/lib/clang/14.0.0/lib/s390x-ibm-zos/libclang_rt.builtins.a 
//* ld  --------------------------------------------------------------- 
//LINKEDIT EXEC PGM=IEWBLINK, 
// PARM='TERM=YES,PRINT=NO,MSGLEVEL=4,STORENX=NEVER,AMODE=64,LIST,DYNAMc
//             =DLL,MSGLEVEL=4,CASE=MIXED,REUS=RENT,edit=no,dynam=dll' 
//SYSLIB   DD  DSN=CEE.SCEEBND2,DISP=SHR,DCB=DSORG=DIR 
//         DD  DSN=SYS1.CSSLIB,DISP=SHR,DCB=DSORG=DIR 
//LDIN0001 DD  DSN=CEE.SCEELIB(CELQS001),DISP=SHR 
//LDIN0002 DD  DSN=CEE.SCEELIB(CELQS003),DISP=SHR 
//SYSLMOD  DD  PATH='./interceptor_test.so.1',PATHOPTS=(OWRONLY,OCREAT)c
//             ,PATHMODE=(SIRWXO,SIRWXG,SIRWXU) 
//SYSDEFSD DD  PATH='./interceptor_test.so.x',PATHOPTS=(OWRONLY,OCREAT)c
//             ,PATHMODE=(SIRWXO,SIRWXG,SIRWXU) 
//SYSPRINT DD  PATH='/dev/fd1',PATHOPTS=(ORDWR,OCREAT,OAPPEND),FILEDATAc
//             =TEXT,PATHMODE=(SIROTH,SIRGRP,SIRUSR,SIWOTH,SIWGRP,SIWUSc
//             R) 
//SYSTERM  DD  PATH='/dev/fd2',PATHOPTS=(ORDWR,OCREAT,OAPPEND),FILEDATAc
//             =TEXT,PATHMODE=(SIROTH,SIRGRP,SIRUSR,SIWOTH,SIWGRP,SIWUSc
//             R) 
//SYSLIN   DD  * 
 INCLUDE  './interceptor_test.o'                                                
 LIBRARY  '/usr/lib/libm.a'                                                     
 LIBRARY  './../../../../../prod/ncurses-6.3/lib/libncurses.a'                  
 LIBRARY  './../../src/librdkafka.a'                                            
 INCLUDE  LDIN0001                                                              
 INCLUDE  LDIN0002                                                              
 LIBRARY  '/Z25R03/usr/lpp/IBM/cnw/v1r1/openxl/lib/clang/14.0.0/lib/s39c        
 0x-ibm-zos/libclang_rt.builtins.a'                                             
/*
 IEW2456E 9207 SYMBOL test_level UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM 
          THE DESIGNATED CALL LIBRARY. NAME SPACE = 3                           
 IEW2456E 9207 SYMBOL test_curr UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM  
          THE DESIGNATED CALL LIBRARY. NAME SPACE = 3                           
 IEW2456E 9207 SYMBOL ictest UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM THE 
          DESIGNATED CALL LIBRARY. NAME SPACE = 3                               
 IEW2456E 9207 SYMBOL test_fail0 UNRESOLVED.  MEMBER COULD NOT BE INCLUDED FROM 
          THE DESIGNATED CALL LIBRARY.                                          
 IEW2456E 9207 SYMBOL test_conf_get UNRESOLVED.  MEMBER COULD NOT BE INCLUDED   
          FROM THE DESIGNATED CALL LIBRARY.                                     
 IEW2665S 40FF MODULE *NULL*  IS NON-EXECUTABLE AND WAS NOT SAVED BECAUSE       
          STORENX=NEVER.                                                        
IEW5033 The binder ended with return code 12.
bash-5.2$ 

[perry-ca]

oh, so test_level etc are not defined in the shared lib but in some other shared lib? You need to include the side for that shared lib on the link step. I think you do that you'll solve the link errors.

I will look into the -e/-O/-u options. The clang compiler is building shared libs the same way as the xlc & xlclang compilers. From what I can see so far is that binder will default to the same values these options specify. They don't control if the output is a shared lib or not. The -u might make a difference.

[khickel]

I'm not an expert on this code but I believe the point is that any executable that wants to use this shard object will define those and link in the shared object. Defining them at the time the shared object is being built would defeat the purpose of this being a shared object.
Again, that is just a guess, and I don't know the exact reasons that they did it this way.

The -e/-O/u seem to be trying to specify a startup routine and main entry point for the shared object, which in this case just don't exist, and don't need to exist.

[perry-ca]

What you are describing requires runtime linking. As far as I know we can't do that on z/OS. When a shared object is linked all of the symbols have to resolve to a symbol in a .o file or is imported from a known shared object. I haven't seen a way to defer symbol resolution until runtime. I'll check again to confirm.

The --shared option is working as it is for other shared object builds. It may be better to drop the -u option. I'll explore that too.

[khickel]

I forgot to point out that whether or not run time linking is required for this or supported on z/OS, the original point is that by changing the ld command that clang is invoking, the .so file does get built, and does work as far as I can tell.