Satellite Application Facility for Numerical Weather Prediction › Forums › RTTOV › RTTOV v13 › RTTOV v13 General Discussion › Problems when importing rttov_wrapper_f2py
- This topic has 9 replies, 2 voices, and was last updated 3 years ago by James Hocking.
-
AuthorPosts
-
September 17, 2021 at 8:38 pm #46698Jun ZhouParticipant
Hello,
I compiled the rttov v13 package with hdf5 version 1.10.4, Python 3.8.8, conda 4.10.3. I do see the rttov_wrapper_f2py.so under ./lib. I have added its path to $PYTHONPATH. But when I try to import rttov_wrapper_f2py, it gives me this error:>>>import rttov_wrapper_f2py
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
ImportError: /usr/lib64/libhdf5_fortran.so.102: undefined symbol: H5Fset_dset_no_attrs_hintThanks!
Jun
September 20, 2021 at 7:39 am #46704James HockingKeymasterHi Jun,
You must ensure that the HDF5 library that is linked at run-time is the same version as that used when compiling RTTOV. Before launching Python, you can do:
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/full/path/to/your/hdf5-1.10.4/lib
Be sure to modify the path to the HDF5 lib/ directory to be the one used when compiling RTTOV.
Best wishes,
JamesSeptember 20, 2021 at 2:47 pm #46705Jun ZhouParticipantHi James,
Thank you for your prompt reply! The full path to hdf5 lib has already been added to LD_LIBRARY_PATH. It seems the problem is not that the libhdf5_fortran.so.102 cannot be found. The problem is the symbol: H5Fset_dset_no_attrs_hint is not defined in the libhdf5_fortran.so.102. Is there any specific requirement when compiling the HDF5 library? Or specific version of HDF5?
Thanks!Jun
September 20, 2021 at 5:08 pm #46707James HockingKeymasterHi Jun,
I’ve tried using pyrttov with an HDF5 v1.10.5 build and I get a similar error to you.
The strange thing is that:
1. When using python, the dynamic linker is only looking in the HDF5 libraries installed in my miniconda environment and ignoring what I’ve set in my $LD_LIBRARY_PATH.
I can see this by doing the following:
$ export LD_DEBUG=symbols
$ python pyrttov_example.py &> ld_debug.txt
$ unset LD_DEBUGAnd then looking for the failure towards the end of ld_debug.txt. This shows where the dynamic linker is looking.
By contrast, I can run the test suite executables because they use the $LD_LIBRARY_PATH as expected.
2. The missing symbol is apparently not present in the miniconda version of the HDF5 v1.10.5 library, but it is present in my own HDF5 v1.10.5 build.
I can see this by doing (in my case) the following:
$ nm /path/to/my/build/of/hdf5-1.10.5/build/ifort17/lib/libhdf5_fortran.so | grep h5global_mp_h5t_std_i16le_
00000000002553b0 B h5global_mp_h5t_std_i16le_$ nm ~/miniconda3/lib/libhdf5_fortran.so | grep h5global_mp_h5t_std_i16le_
# No output i.e. not foundI am still looking into this and will let you know if I can figure out how to get it to look in the right lib/ directory, but I thought I’d post this update in case it helps you solve the problem.
Best wishes,
JamesSeptember 20, 2021 at 6:17 pm #46708Jun ZhouParticipantI have submitted a post. Why it shows “This post is awaiting moderation”?- This reply was modified 3 years ago by Jun Zhou.
September 20, 2021 at 6:29 pm #46710Jun ZhouParticipantHi James,
I can see the posts now.
I’m still struggling with the problem.Jun
September 21, 2021 at 9:58 am #46712James HockingKeymasterHi Jun,
I should have said that I usually test RTTOV with HDF5 v1.8.x (v1.8.8 or later) and this works without any problem with conda (i.e. including the Python wrapper). Based on what I’ve seen above, I suspect that might be because all of the necessary symbols are contained in the conda HDF5 v1.10.x libraries.
There are a couple of things you could therefore try:
1. compile RTTOV against HDF5 v1.8.x: this should work fine.
2. compile RTTOV against the HDF5 libraries in your conda installation. However, you will be limited to using a compatible version of gcc/gfortran as conda uses (which may be different to your system one) so this might be more difficult than it first seems.
I will continue to look into the HDF5 v1.10.x problems with the wrapper.
Best wishes,
JamesSeptember 21, 2021 at 10:17 am #46713James HockingKeymasterHi again Jun,
Your missing symbol “H5Fset_dset_no_attrs_hint” was introduced in HDF5 v1.10.5.
This suggests that somehow you are compiling RTTOV against v1.10.5 (which has that symbol) and then at run-time RTTOV is being linked to v1.10.4 (which does not have that symbol and hence is failing).
Alternatively perhaps there is some other mismatch between the system and conda HDF5 versions. If you can modify your conda environment to use exactly the same HDF5 version as your system version that might resolve the conflict.
Best wishes,
JamesSeptember 22, 2021 at 8:08 pm #46716Jun ZhouParticipantHi James,
I’m currently using HDF5 v1.10.4. Should I upgrade it to HDF5 v1.10.5, then compile rttov with HDF5 v1.10.5?
Thanks!Jun
September 23, 2021 at 8:17 am #46717James HockingKeymasterHi Jun,
Yes, I hope that would solve the problem.
Best wishes,
James -
AuthorPosts
- You must be logged in to reply to this topic.