As the result of compression is cached and certificates are usually small (few kb), I think it would be better to use Z_BEST_COMPRESSION.

Z_DEFAULT_COMPRESSION - is a compromise between two evils: non-efficient and slow compression.
If proven useful, a separate directive can be introduced, similar to gzip_comp_level.

A similar function ngx_ssl_early_data() has different text:

    ngx_log_error(NGX_LOG_WARN, ssl->log, 0,                                     
                  "\"ssl_early_data\" is not supported on this platform, "       
                  "ignored"); 

The same text exists in ngx_ssl_session_ticket_keys(), but the example above looks more relevant.

is not supported on this platform

Such diagnostic messages initially were used to mean the lack of OS feature,
such as IPv6/unix address family or socket options.
It looks somewhat odd to see this clause in relation to a library.
I believe "ssl_early_data" was the first such precedent.

Further, there are several cases of "is not available on this platform",
a slight modification of the above diagnostic message.

On the other side, it seems to be commonly used now throughout the nginx code to start fighting with it.
Notable exceptions are "ssl_stapling" and "ssl_session_ticket_key", both predate early data.
Ironically, "ssl_ocsp" uses "on this platform" although it has the same feature test as "ssl_stapling".

To sum up, I'm fine to change the message as suggested.
Note though this way it cannot be wrapped as consistently due to long naming,
which can be challenging in certain grep exercises.

Why not disable tx compression in the if (!enable) {...} block above instead of disabling it for all new ssl context and enabling it back here. Also, seems like enabling tx compression is completely safe anyway.

Why not disable tx compression in the if (!enable) {...} block above instead of disabling it for all new ssl context and enabling it back here.

Because it is meant to be disabled by default, and this is done in a logically separate preliminary change.
Besides that, ngx_ssl_certificate_compression() doesn't fall under the "for all new ssl context" criteria.

Also, seems like enabling tx compression is completely safe anyway.

Depends on what you put in "safe".
OpenSSL documentation quite clearly describes what this option intends to do.

Although the current behaviour is indeed to send precompressed certificates only,
we cannot rely on such an implementation detail that can be changed in future.

Additionally, not using SSL_OP_NO_TX_CERTIFICATE_COMPRESSION as suggested
impends an overhead in parsing and processing the compress_certificate extension
in ClientHello and in the handshake state machine, which can be avoided with no effort.

For example, see previous attempts to implement this feature,
where pre-compressed certificates was an option, in addition to using a callback.

In the previous patch the other macro SSL_OP_NO_TX_CERTIFICATE_COMPRESSION was used . While for these purposes they are equally suitable, I suggest using the same macro. Moreover, this block itself uses the TX macro 2 lines below.

Fixed, thanks.

I don't see a reason for this block, there's a return NGX_OK below.

Actually, this raises a question whether we should fail softly due to the optional nature of the feature and/or emit a warning.
Failing softly will make this consistent to SSL_CTX_compress_certs() error handling, although for different reasons.
Unlike in SSL_CTX_compress_certs(), registering the BoringSSL callback as currently implemented can barely fail on memory shortage. OTOH, we cannot rely on this as it may change in future hypothetically.

That said, a viable compromise is to emit a warning and let the user decide.

See updated in 50e7f09