BUG: f2py cannot find .mod files automatically for the new meson backend #27018
Description
Describe the issue:
I'm trying to compile my Fortran code using f2py. For python>=3.12, the meson backend is now the default. But meson/f2py cannot automatically find the generated .mod files in the current directory. These .mod files could be automatically found for older versions of python and distutils backend. By the way, gcc/gfortran in this machine is 4.8.5.
Thanks for any kind help and suggestion!
Best,
Jingxiang
Reproduce the code example:
A simple example is shown using two Fortran files and one Python script:
here is a.f90
module para
implicit none
integer, parameter :: n = 5
end module para
b.f90
subroutine prt_n
use para, only: n
implicit none
write(6,'(A,I2)') 'n=', n
end subroutine prt_n
Compiling commands are
gfortran a.f90 -c
f2py -c a.o b.f90 -m test
The first command would generate a.o and para.mod files in the current directory. para.mod can be automatically detected for the distutils backend, but not for the meson backend.
test.py
from test import prt_n
prt_n()Assuming that the test.xxx.so is generated (which not True for the meson backend), the result would be n= 5.
Error message:
After running f2py -c a.o b.f90 -m test, the error message is
Found ninja-1.10.2 at /public/home/jxzou/software/anaconda3/envs/py312/bin/ninja
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /public/home/jxzou/software/anaconda3/envs/py312/bin/ninja -C /public/home/jxzou/tmp/tmpgljbyueg/bbdir
ninja: Entering directory `/public/home/jxzou/tmp/tmpgljbyueg/bbdir'
[3/6] Compiling Fortran object test.cpython-312-x86_64-linux-gnu.so.p/b.f90.o
FAILED: test.cpython-312-x86_64-linux-gnu.so.p/b.f90.o
gfortran -Itest.cpython-312-x86_64-linux-gnu.so.p -I. -I.. -I/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/core/include -I/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/src -I/public/home/jxzou/software/anaconda3/envs/py312/include/python3.12 -fvisibility=hidden -D_FILE_OFFSET_BITS=64 -Wall -O3 -fPIC -Jtest.cpython-312-x86_64-linux-gnu.so.p -o test.cpython-312-x86_64-linux-gnu.so.p/b.f90.o -c ../b.f90
../b.f90:2.5:
use para, only: n
1
Fatal Error: Can't open module file 'para.mod' for reading at (1): No such file or directory
[5/6] Compiling C object test.cpython-312-x86_64-linux-gnu.so.p/3fa16a5cf62c3f6199e0fa73eb3f172c17c452f1_.._.._f2py_src_fortranobject.c.o
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/public/home/jxzou/software/anaconda3/envs/py312/bin/f2py", line 11, in <module>
sys.exit(main())
^^^^^^
File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 766, in main
run_compile()
File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 738, in run_compile
builder.compile()
File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 178, in compile
self.run_meson(self.build_dir)
File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 173, in run_meson
self._run_subprocess_command(compile_command, build_dir)
File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 167, in _run_subprocess_command
subprocess.run(command, cwd=cwd, check=True)
File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['meson', 'compile', '-C', 'bbdir']' returned non-zero exit status 1.
Python and NumPy Versions:
python>=3.12 and numpy>=1.26
Runtime Environment:
[{'numpy_version': '1.26.4',
'python': '3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, '
'15:12:24) [GCC 11.2.0]',
'uname': uname_result(system='Linux', node='mu003', release='3.10.0-862.el7.x86_64', version='#1 SMP Fri Apr 20 16:44:24 UTC 2018', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2',
'AVX512F',
'AVX512CD',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL'],
'not_found': ['AVX512_KNL', 'AVX512_KNM']}},
{'filepath': '/public/home/jxzou/software/anaconda3/envs/py312/lib/libmkl_rt.so.2',
'internal_api': 'mkl',
'num_threads': 64,
'prefix': 'libmkl_rt',
'threading_layer': 'intel',
'user_api': 'blas',
'version': '2023.1-Product'},
{'filepath': '/public/home/jxzou/software/anaconda3/envs/py312/lib/libiomp5.so',
'internal_api': 'openmp',
'num_threads': 64,
'prefix': 'libiomp',
'user_api': 'openmp',
'version': None}]
Context for the issue:
I'm a developer of the open source package MOKIT. By using f2py, we offer lots of convenient and efficient Python APIs to users in the computational chemistry field. We are grateful for your numpy f2py functionalities. MOKIT source code are mainly written using Fortran, so we heavily reply on the f2py to obtain corresponding .so files.
Currently my workaround is to change all .o filenames into .f90 ones in f2py commands, e.g.
f2py -c a.f90 b.f90 -m test
This is a tiny change for this simple example. But a somewhat big work to do in MOKIT. And this change would make test.so contain more APIs than previous. For example, if there are many subroutines in a.f90, it will be all embraced into test.so. This problem does not exist when the distutils backend is used previously. I do not know whether this is a feature or bug for the new meson backend. I'm wondering if there is any elegant solution.