Skip to content

BUG: Cannot import numpy from different interpreters in the same process #28271

@jabraham17

Description

@jabraham17

Describe the issue:

When using the Python C API, it is possible to close a Python interpreter and then restart it fresh. However, attempting to import numpy when doing this causes an unavoidable RuntimeError. It seems that something is not being properly cleaned up on interpreter exit, resulting in a new interpreter retaining the old state.

Also note there is a typo in the RuntimeError message

Reproduce the code example:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

// clang bug.c `python3-config --embed --cflags` `python3-config --embed --ldflags`

void import_numpy() {
  Py_Initialize();
  PyObject* numpy = PyImport_ImportModule("numpy");
  if (!numpy) {
    PyErr_Print();
  }
  Py_CLEAR(numpy);
  Py_Finalize();
}

int main() {
  import_numpy(); // this works
  import_numpy(); // this hits the exception
  return 0;
}

Error message:

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/numpy/__init__.py", line 114, in <module>
    from numpy.__config__ import show_config
  File ".venv/lib/python3.12/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy._core._multiarray_umath import (
  File ".venv/lib/python3.12/site-packages/numpy/_core/__init__.py", line 23, in <module>
    from . import multiarray
  File ".venv/lib/python3.12/site-packages/numpy/_core/multiarray.py", line 10, in <module>
    from . import overrides
  File ".venv/lib/python3.12/site-packages/numpy/_core/overrides.py", line 7, in <module>
    from numpy._core._multiarray_umath import (
RuntimeError: CPU dispatcher tracer already initlized

Python and NumPy Versions:

numpy 2.2.1
pythom 3.12.7

Runtime Environment:

[{'numpy_version': '2.2.1',
'python': '3.12.7 (main, Oct 1 2024, 02:05:46) [Clang 15.0.0 '
'(clang-1500.1.0.2.5)]',
'uname': uname_result(system='Darwin', node='<>', release='22.6.0', version='Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000', machine='arm64')},
{'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
'found': ['ASIMDHP'],
'not_found': ['ASIMDFHM']}},
{'architecture': 'neoversen1',
'filepath': '.venv/lib/python3.12/site-packages/numpy/.dylibs/libscipy_openblas64_.dylib',
'internal_api': 'openblas',
'num_threads': 10,
'prefix': 'libscipy_openblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.28'}]

Context for the issue:

The Chapel language team is working on Python interop support in general and creating multiple interpreters that interop with numpy is essential. This also prevents us from using other Python modules that rely on numpy. For example, replacing 'numpy' with 'pandas' in the code above hits the exact same error, because pandas uses numpy.

Metadata

Metadata

Assignees

No one assigned

    Labels

    00 - Bug57 - Close?Issues which may be closable unless discussion continued

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions