Problems when importing rttov_wrapper_f2py

[Jun Zhou]

Hello,
I compiled the rttov v13 package with hdf5 version 1.10.4, Python 3.8.8, conda 4.10.3. I do see the rttov_wrapper_f2py.so under ./lib. I have added its path to $PYTHONPATH. But when I try to import rttov_wrapper_f2py, it gives me this error:

>>>import rttov_wrapper_f2py
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
ImportError: /usr/lib64/libhdf5_fortran.so.102: undefined symbol: H5Fset_dset_no_attrs_hint

Thanks!

Jun

[James Hocking]

Hi Jun,

You must ensure that the HDF5 library that is linked at run-time is the same version as that used when compiling RTTOV. Before launching Python, you can do:

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/full/path/to/your/hdf5-1.10.4/lib

Be sure to modify the path to the HDF5 lib/ directory to be the one used when compiling RTTOV.

Best wishes,
James

[Jun Zhou]

Hi James,
Thank you for your prompt reply! The full path to hdf5 lib has already been added to LD_LIBRARY_PATH. It seems the problem is not that the libhdf5_fortran.so.102 cannot be found. The problem is the symbol: H5Fset_dset_no_attrs_hint is not defined in the libhdf5_fortran.so.102. Is there any specific requirement when compiling the HDF5 library? Or specific version of HDF5?
Thanks!

Jun

[James Hocking]

Hi Jun,

I’ve tried using pyrttov with an HDF5 v1.10.5 build and I get a similar error to you.

The strange thing is that:

1. When using python, the dynamic linker is only looking in the HDF5 libraries installed in my miniconda environment and ignoring what I’ve set in my $LD_LIBRARY_PATH.

I can see this by doing the following:
$ export LD_DEBUG=symbols
$ python pyrttov_example.py &> ld_debug.txt
$ unset LD_DEBUG

And then looking for the failure towards the end of ld_debug.txt. This shows where the dynamic linker is looking.

By contrast, I can run the test suite executables because they use the $LD_LIBRARY_PATH as expected.

2. The missing symbol is apparently not present in the miniconda version of the HDF5 v1.10.5 library, but it is present in my own HDF5 v1.10.5 build.

I can see this by doing (in my case) the following:

$ nm /path/to/my/build/of/hdf5-1.10.5/build/ifort17/lib/libhdf5_fortran.so | grep h5global_mp_h5t_std_i16le_
00000000002553b0 B h5global_mp_h5t_std_i16le_

$ nm ~/miniconda3/lib/libhdf5_fortran.so | grep h5global_mp_h5t_std_i16le_
# No output i.e. not found

I am still looking into this and will let you know if I can figure out how to get it to look in the right lib/ directory, but I thought I’d post this update in case it helps you solve the problem.

Best wishes,
James

[Jun Zhou]

I have submitted a post. Why it shows “This post is awaiting moderation”?

• This reply was modified 3 years, 2 months ago by Jun Zhou.

[Jun Zhou]

Hi James,
I can see the posts now.
I’m still struggling with the problem.

Jun

[James Hocking]

Hi Jun,

I should have said that I usually test RTTOV with HDF5 v1.8.x (v1.8.8 or later) and this works without any problem with conda (i.e. including the Python wrapper). Based on what I’ve seen above, I suspect that might be because all of the necessary symbols are contained in the conda HDF5 v1.10.x libraries.

There are a couple of things you could therefore try:

1. compile RTTOV against HDF5 v1.8.x: this should work fine.

2. compile RTTOV against the HDF5 libraries in your conda installation. However, you will be limited to using a compatible version of gcc/gfortran as conda uses (which may be different to your system one) so this might be more difficult than it first seems.

I will continue to look into the HDF5 v1.10.x problems with the wrapper.

Best wishes,
James

[James Hocking]

Hi again Jun,

Your missing symbol “H5Fset_dset_no_attrs_hint” was introduced in HDF5 v1.10.5.

This suggests that somehow you are compiling RTTOV against v1.10.5 (which has that symbol) and then at run-time RTTOV is being linked to v1.10.4 (which does not have that symbol and hence is failing).

Alternatively perhaps there is some other mismatch between the system and conda HDF5 versions. If you can modify your conda environment to use exactly the same HDF5 version as your system version that might resolve the conflict.

Best wishes,
James

[Jun Zhou]

Hi James,
I’m currently using HDF5 v1.10.4. Should I upgrade it to HDF5 v1.10.5, then compile rttov with HDF5 v1.10.5?
Thanks!

Jun

[James Hocking]

Hi Jun,

Yes, I hope that would solve the problem.

Best wishes,
James

[Author]

Posts