Nice start. We have this on the current master:

$XONSH_TRACE_SUBPROC=True

echo 1
# TRACE SUBPROC: (['echo', '1'],), captured=hiddenobject

_ = $(echo 1)
# TRACE SUBPROC: (['echo', '1'],), captured=stdout

_ = !(echo 1)
# TRACE SUBPROC: (['echo', '1'],), captured=object

![echo 1]
# TRACE SUBPROC: (['echo', '1'],), captured=hiddenobject

$[echo 1]
# TRACE SUBPROC: (['echo', '1'],), captured=False

And we want to switch first case to captured=False without any effect on other cases.

By the proposed changes:

if captured == "hiddenobject" and not builtins.xonsh.env.get("XONSH_CAPTURE_ALWAYS"):

we will change also ![] behavior that we should not change.

Also I think this setting should be in xonfig to analysis the future issues around freezing the terminal in the captured mode.

When running commands in xonsh they are actually transformed into ![]. It's the same thing

Yep, and I think we should have the way to switch this transformation to $[].

I suggest for example $XONSH_SUBPROC_NO_BRACES_CAPTURED variable. Default: True. But in the future it will be False.

Why? That way you can't know the return value and other info.
As I see it, since ![] denotes an "uncaptured subprocesses", xonsh shouldn't capture its stdout unless explicitly told to via XONSH_CAPTURE_ALWAYS.
![] and $[] should behave identically (except their return values), a la the docs:

Uncaptured subprocesses are denoted with the $[] and ![] operators.
They are the same as $() captured subprocesses in almost every way.
The only difference is that the subprocess’s stdout passes directly through xonsh and to the screen.
The return value of $[] is always None.

Another current issue is that when THREAD_SUBPROCS=False, !() fails to capture stdout while $() succeeds (without running the process in another thread).
As I see it, this doesn't make any sense and they should behave identically as well.

Why? That way you can't know the return value and other info.

Yep, maybe it should (have to) be ![]. I just have no success with this operator:

cd /tmp
# download interactive tool i.e. Miniconda from https://docs.conda.io/en/latest/miniconda.html
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x Miniconda3-latest-Linux-x86_64.sh
![./Miniconda3-latest-Linux-x86_64.sh ]
# press Enter and got freezed shell
$[./Miniconda3-latest-Linux-x86_64.sh ]
# working as expected

It looks like we should have another operator that can run uncaptured command but can catch return code etc.

As I understand we want to solve this case. We want to run just:

./Miniconda3-latest-Linux-x86_64.sh
# or the same case is #4243:
git log

and save the return code and timing to the history. For now these cases are failed because captured is the default.

Subproc has stdout/stderr and return code. Sometimes we want to catch both. Sometimes only return code. Sometimes nothing. Is the current operators/run_subproc functionality enough?

It looks like we should have another operator that can run uncaptured command but can catch return code etc.

That's exactly what XONSH_CAPTURE_ALWAYS is for. No need for a different operator.
This whole 'printing and also capturing at the same time' mechanism is unexpected and causes most of these problems.

I've tested the branch and it works for me!

No need for a different operator.

We lose the use case like:

# current master
r = ![ls]  # the tool that produce some output
if 'file' in r.out:  # searching in the output
   print('found file')

In this case ![ls] returns colored output but r.out has the output without colors. The case looks useful.

r = ![ls] # the tool that produce some output

So this is the case to turn on XONSH_CAPTURE_OUTPUT
But in reality if you want the output of a command you need to run $() or !()

In this case ![ls] returns colored output but r.out has the output without colors. The case looks useful.

I wonder why that is.. this is really strange! (just to be clear it also happens on main)

Codecov Report

Merging #4283 (606531a) into main (8e547d0) will decrease coverage by 0.05%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #4283      +/-   ##
==========================================
- Coverage   57.18%   57.12%   -0.06%     
==========================================
  Files         138      138              
  Lines       22151    22152       +1     
  Branches     4059     4059              
==========================================
- Hits        12666    12655      -11     
- Misses       8438     8444       +6     
- Partials     1047     1053       +6     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e547d0...606531a. Read the comment docs.

Thanks @anki-code !
@gforsyth I would also appreciate your review since this changes default behavior