gh-135427: Fix DeprecationWarning for os.fork when run in threads with -Werror #136796

rani-pinchuk · 2025-07-19T11:41:42Z

This is a small fix as suggested in #135427 - using PyErr_WriteUnraisable() instead of PyErr_Clear().

The fix includes a test which checks that the DeprecationWarning is provided when using fork or forkpty within a thread, also when running with -Werror.

@encukou

Issue: -Werror doesn't error out on DeprecationWarning for os.fork() #135427

…ds with -Werror

python-cla-bot · 2025-07-19T11:41:46Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2025-07-19T11:41:46Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app · 2025-07-19T11:45:45Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

rani-pinchuk · 2025-07-19T11:57:25Z

I have created a NEWS entry for this fix, but actually this change has indeed a little impact on Python users, so maybe skip news tag is more appropriate here.

…he exception.

…nto fix-issue-135427

rani-pinchuk · 2025-07-19T15:52:52Z

Many tests are failing with a warning - as now the DeprecationWarning is no longer hidden (and as I unedrstand it, the tests run with the equivalent of -Werror).

Many of the failing tests do not fail where one would expect to have more than one thread running. For example, in test_uuid.testIssue8621.

When printing there the number of threads using threading.active_count() we get 1. However, when investigating further, the number of threads are actually 2 - just that one of the threads is not a python thread, and therefore not counted by threading.active_count().

Therefore, we should find a way to suppress the warning of running a fork within a thread in all the tests that fail. Petr suggests to consider using the sys.unraisablehook (see https://vstinner.github.io/sys-unraisablehook-python38.html) in all the places where the tests fail.

If anyone knows where this extra thread comes from, it will be appreciated if this is explained :-)

gpshead · 2025-07-19T17:51:33Z

If anyone knows where this extra thread comes from, it will be appreciated if this is explained :-)

That is ironically exactly why we have this warning - threads in a process can come from everywhere including external libraries outside of our control.

I do think you're on the right track in terms of how to enable this to be something -Werror captures. As a potentially disruptive behavior change for some test suites, it isn't a bug fix - but it'd make sense in 3.15. Having a news entry is indeed appropriate.

using sys.unraisablehook in these parts of the test suites is an interesting idea.

…ead. Started to fix tests.

…sed.

…line

encukou

Thank you! This is looking good, but needs a bit more polish yet.

.gitignore

Lib/test/_test_multiprocessing.py

changed, 7314 insertions(+), 3271 deletions(-)

bedevere-bot · 2025-08-11T10:54:09Z

🤖 New build scheduled with the buildbot fleet by @encukou for commit 8b699d8 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136796%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

Co-authored-by: Petr Viktorin <[email protected]>

encukou · 2025-08-12T09:52:21Z

Lib/test/_test_multiprocessing.py

-            warnings.filterwarnings('ignore',
-                                    message=".*fork.*may lead to deadlocks in the child.*",
-                                    category=DeprecationWarning)
+        with warnings_helper.ignore_fork_in_thread_deprecation_warnings():
            super().setUpClass()


Could you do this in the other instances as well? (4 in this file & one in test_concurrent_futures/util.py?)

.gitignore

encukou · 2025-08-19T08:57:25Z

@rani-pinchuk, I think this is close to the finish line. Would you like me to fix up the remaining issue?

rani-pinchuk · 2025-08-19T09:15:32Z

Hi Petr, I am on holiday, so I can fix it only in this weekend. But if you want to push it faster of course you can.

…

On Tue, Aug 19, 2025, 10:57 AM Petr Viktorin ***@***.***> wrote: *encukou* left a comment (python/cpython#136796) <#136796 (comment)> @rani-pinchuk <https://github.com/rani-pinchuk>, I think this is close to the finish line. Would you like me to fix up the remaining issue? — Reply to this email directly, view it on GitHub <#136796 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH6O62VAUIB26FXWRUIBS5D3OLRI3AVCNFSM6AAAAACB4QZFAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCOJZHA3DCMRUGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

bedevere-bot · 2025-08-22T11:20:36Z

🤖 New build scheduled with the buildbot fleet by @encukou for commit baa8446 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136796%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

bedevere-bot · 2025-08-26T09:05:29Z

🤖 New build scheduled with the buildbot fleet by @encukou for commit b08021d 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136796%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

bedevere-bot · 2025-08-26T16:08:58Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM Raspbian Linux Asan 3.x (no tier) has failed when building commit fd8f42d.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1811/builds/156) and take a look at the build logs.
Check if the failure is related to this commit (fd8f42d) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1811/builds/156

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 39, done.        
remote: Counting objects:   2% (1/36)        
remote: Counting objects:   5% (2/36)        
remote: Counting objects:   8% (3/36)        
remote: Counting objects:  11% (4/36)        
remote: Counting objects:  13% (5/36)        
remote: Counting objects:  16% (6/36)        
remote: Counting objects:  19% (7/36)        
remote: Counting objects:  22% (8/36)        
remote: Counting objects:  25% (9/36)        
remote: Counting objects:  27% (10/36)        
remote: Counting objects:  30% (11/36)        
remote: Counting objects:  33% (12/36)        
remote: Counting objects:  36% (13/36)        
remote: Counting objects:  38% (14/36)        
remote: Counting objects:  41% (15/36)        
remote: Counting objects:  44% (16/36)        
remote: Counting objects:  47% (17/36)        
remote: Counting objects:  50% (18/36)        
remote: Counting objects:  52% (19/36)        
remote: Counting objects:  55% (20/36)        
remote: Counting objects:  58% (21/36)        
remote: Counting objects:  61% (22/36)        
remote: Counting objects:  63% (23/36)        
remote: Counting objects:  66% (24/36)        
remote: Counting objects:  69% (25/36)        
remote: Counting objects:  72% (26/36)        
remote: Counting objects:  75% (27/36)        
remote: Counting objects:  77% (28/36)        
remote: Counting objects:  80% (29/36)        
remote: Counting objects:  83% (30/36)        
remote: Counting objects:  86% (31/36)        
remote: Counting objects:  88% (32/36)        
remote: Counting objects:  91% (33/36)        
remote: Counting objects:  94% (34/36)        
remote: Counting objects:  97% (35/36)        
remote: Counting objects: 100% (36/36)        
remote: Counting objects: 100% (36/36), done.        
remote: Compressing objects:   7% (1/13)        
remote: Compressing objects:  15% (2/13)        
remote: Compressing objects:  23% (3/13)        
remote: Compressing objects:  30% (4/13)        
remote: Compressing objects:  38% (5/13)        
remote: Compressing objects:  46% (6/13)        
remote: Compressing objects:  53% (7/13)        
remote: Compressing objects:  61% (8/13)        
remote: Compressing objects:  69% (9/13)        
remote: Compressing objects:  76% (10/13)        
remote: Compressing objects:  84% (11/13)        
remote: Compressing objects:  92% (12/13)        
remote: Compressing objects: 100% (13/13)        
remote: Compressing objects: 100% (13/13), done.        
remote: Total 39 (delta 23), reused 23 (delta 23), pack-reused 3 (from 2)        
From https://github.com/python/cpython
 * branch                    main       -> FETCH_HEAD
Note: switching to 'fd8f42d3d1038a812340c3ec3cbfc995a80c4e13'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at fd8f42d3d10 gh-135427: Fix DeprecationWarning for os.fork when run in threads with -Werror (GH-136796)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module

In file included from ./Include/internal/pycore_dict.h:11,
                 from Objects/typeobject.c:7:
In function ‘Py_DECREF_MORTAL’,
    inlined from ‘PyStackRef_XCLOSE’ at ./Include/internal/pycore_stackref.h:730:9,
    inlined from ‘_PyThreadState_PopCStackRef’ at ./Include/internal/pycore_stackref.h:810:5,
    inlined from ‘vectorcall_maybe’ at Objects/typeobject.c:3108:9:
./Include/internal/pycore_object.h:481:8: warning: array subscript 0 is outside array bounds of ‘PyObject[0]’ {aka ‘struct _object[]’} [-Warray-bounds]
  481 |     if (--op->ob_refcnt == 0) {
      |        ^

Timeout (0:05:00)!
Thread 0x0000007fad2ef100 [Thread-2] (most recent call first):
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 1252 in _remaining_time
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 2053 in _wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 1277 in wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 194 in _run_process
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 299 in run_tmp_files
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 363 in _runtest
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 403 in run
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1074 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1036 in _bootstrap

Thread 0x0000007fadaff100 [Thread-1] (most recent call first):
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 2042 in _wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 1277 in wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 194 in _run_process
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 299 in run_tmp_files
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 363 in _runtest
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 403 in run
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1074 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1036 in _bootstrap

Thread 0x0000007fb98276c0 [python] (most recent call first):
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/logger.py", line 47 in get_load_avg
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/logger.py", line 27 in log
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 553 in _get_result
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 610 in run
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 455 in _run_tests_mp
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 561 in _run_tests
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 598 in run_tests
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 767 in main
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 775 in main
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/__main__.py", line 2 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main
make: *** [Makefile:2494: buildbottest] Error 1

Cannot open file '/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/test-results.xml' for upload

pythongh-135427: Fix DeprecationWarning for os.fork when run in threa…

6622c55

…ds with -Werror

rani-pinchuk requested a review from gpshead as a code owner July 19, 2025 11:41

bedevere-app bot added the awaiting review label Jul 19, 2025

bedevere-app bot mentioned this pull request Jul 19, 2025

-Werror doesn't error out on DeprecationWarning for os.fork() #135427

Closed

Merge branch 'main' into fix-issue-135427

7b13bdc

📜🤖 Added by blurb_it.

bf97fe0

rani-pinchuk added 2 commits July 19, 2025 15:34

pythongh-135427: check that exception occured, and also clear later t…

3482918

…he exception.

Merge remote-tracking branch 'refs/remotes/origin/fix-issue-135427' i…

ce90d77

…nto fix-issue-135427

rani-pinchuk added 4 commits July 20, 2025 15:06

Behavior changed to raise an exception when fork is used within a thr…

1a20748

…ead. Started to fix tests.

Seems like test_multiprocessing fork deprecation warnings are suppres…

9bead0e

…sed.

Warning of deprecated fork are suppressed in more tests.

d5336dc

WIP - suppressing warnings in tests that fail in the tests running on…

ae3a5ed

…line

rani-pinchuk requested review from rhettinger, a team, 1st1, asvetlov, kumaraditya303 and willingc as code owners July 20, 2025 15:54

rhettinger removed their request for review July 21, 2025 06:06

rani-pinchuk added 3 commits July 21, 2025 16:56

WIP - continuing suppressing warnings in tests.

f768f5d

WIP - fix imports.

6308e19

WIP - more suppressing of warnings

8955136

rani-pinchuk requested a review from vsajip as a code owner July 21, 2025 15:50

WIP - fix ignore_warnings to wrap also async functions

8731cba

rani-pinchuk added 4 commits July 31, 2025 21:09

Remove Misc/mypy/_colorize.py - should be a symlink

36147a1

Remove Misc/mypy/_token.py and Misc/mypy/_pyrelp - should be a symlink

baab332

Ignore the symlinked files

244869e

Fix warning for asymc function

8b699d8

encukou reviewed Aug 11, 2025

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved

changed, 7314 insertions(+), 3271 deletions(-) Outdated Show resolved Hide resolved

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 11, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 11, 2025

rani-pinchuk and others added 3 commits August 12, 2025 08:42

Update Lib/test/_test_multiprocessing.py

1cc929b

Co-authored-by: Petr Viktorin <[email protected]>

Merge in the main branch

3d954be

Restore files in Misc

7ee8c23

encukou reviewed Aug 12, 2025

View reviewed changes

encukou added 5 commits August 21, 2025 12:32

Use the helper in more setUp methods

c0ebdf1

Merge in the main branch

4bf706d

Remove unused import

a95785d

Add required import ^^;

80172be

Remove accidentally-committed summary

baa8446

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 22, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 22, 2025

encukou added 2 commits August 25, 2025 15:01

Merge in the main branch

776e2b7

Add decorator to new tests

b08021d

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 26, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 26, 2025

encukou merged commit fd8f42d into python:main Aug 26, 2025
114 checks passed

bedevere-app bot removed the awaiting review label Aug 26, 2025

Uh oh!

gh-135427: Fix DeprecationWarning for os.fork when run in threads with -Werror #136796

gh-135427: Fix DeprecationWarning for os.fork when run in threads with -Werror #136796

Conversation

rani-pinchuk commented Jul 19, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

python-cla-bot bot commented Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app bot commented Jul 19, 2025

Uh oh!

bedevere-app bot commented Jul 19, 2025

Uh oh!

rani-pinchuk commented Jul 19, 2025

Uh oh!

rani-pinchuk commented Jul 19, 2025

Uh oh!

gpshead commented Jul 19, 2025

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Aug 11, 2025

Uh oh!

encukou Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

encukou commented Aug 19, 2025

Uh oh!

rani-pinchuk commented Aug 19, 2025 via email

Uh oh!

bedevere-bot commented Aug 22, 2025

Uh oh!

bedevere-bot commented Aug 26, 2025

Uh oh!

Uh oh!

bedevere-bot commented Aug 26, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Uh oh!

rani-pinchuk commented Jul 19, 2025 •

edited by bedevere-app bot

Loading

python-cla-bot bot commented Jul 19, 2025 •

edited

Loading