HTTP/3 to upstream PoC #770

arut · 2025-07-06T16:28:22Z

The PR adds support for HTTP/3 protocol to upstream.

Module

While the HTTP/3 proxy code is contained in a separate module, a decision has been made to integrate it with the ngx_http_proxy_module and rely on its directives. This allows to avoid confusion when switching between protocols. The HTTP/3 protocol is enabled by proxy_http_version 3.0. Currently the protocol version can only be specified statically, but this can be changed later to allow for dynamic protocol selection. But even if implemented, the selection will only be possible while initializing the upstream. It will not be possible to change the protocol after trying another version.

Cache

Despite using the same directives for caching (proxy_cache etc), cache format is incompatible between HTTP/3 and HTTP/1 and is expected to be incompatible with HTTP/2 as well. When using the same cache for different protocols, alter proxy_cache_key to include protocol version indication.

Example

server {
    listen 8000;

    location / {
        proxy_http_version 3.0;
        proxy_pass https://127.0.0.1:8443;
    }
}

server {
    listen 8443 quic reuseport;

    ssl_certificate certs/example.com.crt;
    ssl_certificate_key certs/example.com.key;

    location / {
        return 200 foo;
    }
}

Multiplexing

This PR does not provide HTTP/3 connection multiplexing, meaning a single QUIC connection is created for each request. Note that without multiplexing no performance benefits are expected compared to HTTP/1 proxy. Multiplexing is currently considered a separate feature and will likely require a redesign of nginx upstream keepalive.

An effort to add multiplexing can be found here. It uses a separate set of keepalive-like directives with http3_ prefix. A proper implementation will probably rely on location-based keepalive as opposed to currently implemented upstream-based.

HanadaLee · 2025-07-07T02:43:13Z

Due to the importance of cache hit rate, it is recommended to implement a compatibility layer to handle cache protocol conversion (a switch can be provided to turn it on or off).
This is especially true for scenarios where dynamic protocol selection is planned to be implemented later, as the upstream protocol may change dynamically.

hongzhidao · 2025-07-07T03:14:21Z

Hi Roman,
Great job!
I created a draft PR of http2 for proxy module based on your design.
I feel we can separate a refactored patch for http2 and http3 to upstream. 5245508
I borrowed your code and feel free to create a new one in your PR, I'll rebase yours anytime.

arut · 2025-07-07T07:31:43Z

Due to the importance of cache hit rate, it is recommended to implement a compatibility layer to handle cache protocol conversion (a switch can be provided to turn it on or off). This is especially true for scenarios where dynamic protocol selection is planned to be implemented later, as the upstream protocol may change dynamically.

I agree this would be a nice option to have. This will come at a price though. We''ll get back to it at some point.

src/http/ngx_http_upstream.c

A mistake.

src/http/ngx_http_upstream.c

bullerdu · 2025-07-14T09:07:56Z

src/event/quic/ngx_event_quic_streams.c

-        ngx_reusable_connection(pc, 1);
-    }
-
-    if (qc->shutdown) {


What is the intention of deleting this code？Useless logic？

Some functionality is moved from QUIC to application layer, HTTP/3 in this case. For example, tracking whether it's time to close the QUIC connection.

hongzhidao · 2025-07-16T07:59:18Z

Hi @arut,
Does it make sense not to reuse ngx_http_proxy_ctx_t? I checked the implementation, we only reuse vars and tailers, right?
Take a look at this. 5052583

arut · 2025-07-16T08:13:39Z

Hi @arut, Does it make sense not to reuse ngx_http_proxy_ctx_t? I checked the implementation, we only reuse vars and tailers, right? Take a look at this. 5052583

What about the variables like $proxy_host? I agree it would be nice to keep ctx private though, if it's not too hard.

arut · 2025-07-16T08:54:20Z

Hi @arut, Does it make sense not to reuse ngx_http_proxy_ctx_t? I checked the implementation, we only reuse vars and tailers, right? Take a look at this. 5052583

I also think we can keep trailers private and have separate trailers fields for HTTP/2 and HTTP/3. It's all about vars and proxy variables.

hongzhidao · 2025-07-16T08:55:13Z

Hi @arut, Does it make sense not to reuse ngx_http_proxy_ctx_t? I checked the implementation, we only reuse vars and tailers, right? Take a look at this. 5052583

What about the variables like $proxy_host? I agree it would be nice to keep ctx private though, if it's not too hard.

Good question. It looks like ngx_http_proxy_ctx_t is required for all of them.

What about the design?
There is only one module ngx_http_proxy_module with ngx_http_proxy_ctx_t, we don't introduce ngx_http_v2_proxy_module and ngx_http_v3_proxy_module. But we can add their private fields. For example:

#if (NGX_HTTP_V2)
    http2 fields from `ngx_http_v2_proxy_ctx_t` and renamed it `ngx_http_v2_proxy_t`.
#endif

#if (NGX_HTTP_V3)
    http2 fields from `ngx_http_v3_proxy_ctx_t` and renamed it `ngx_http_v3_proxy_t`.
#endif

typedef struct {
    ...
    unsigned                       connection_type:2;

#if (NGX_HTTP_V2)
    ngx_http_v2_proxy_t            http2;
#endif
#if (NGX_HTTP_V3)
    ngx_http_v2_proxy_t            http3;
#endif

} ngx_http_proxy_ctx_t;

Then we can access them like ctx->http2, ctx->http3.

arut · 2025-07-16T09:07:46Z

Hi @arut, Does it make sense not to reuse ngx_http_proxy_ctx_t? I checked the implementation, we only reuse vars and tailers, right? Take a look at this. 5052583

What about the variables like $proxy_host? I agree it would be nice to keep ctx private though, if it's not too hard.

Good question. It looks like ngx_http_proxy_ctx_t is required for all of them.

What about the design? There is only one module ngx_http_proxy_module with ngx_http_proxy_ctx_t, we don't introduce ngx_http_v2_proxy_module and ngx_http_v3_proxy_module. But we can add their private fields. For example:
#if (NGX_HTTP_V2)
    http2 fields from `ngx_http_v2_proxy_ctx_t` and renamed it `ngx_http_v2_proxy_t`.
#endif

#if (NGX_HTTP_V3)
    http2 fields from `ngx_http_v3_proxy_ctx_t` and renamed it `ngx_http_v3_proxy_t`.
#endif

typedef struct {
    ...
    unsigned                       connection_type:2;

#if (NGX_HTTP_V2)
    ngx_http_v2_proxy_t            http2;
#endif
#if (NGX_HTTP_V3)
    ngx_http_v2_proxy_t            http3;
#endif

} ngx_http_proxy_ctx_t;
Then we can access them like ctx->http2, ctx->http3.

I'm not sure if it helps. Also, Ideally these modules should be independent.

hongzhidao · 2025-07-16T09:10:43Z

Let me try it with v2 proxy module.

bullerdu · 2025-07-18T03:39:13Z

src/event/quic/ngx_event_quic.h

ngx_quic_init_pt and ngx_quic_shutdown_pt also need to remove

bullerdu · 2025-07-18T07:07:42Z

src/event/quic/ngx_event_quic.c

-        return NULL;
-    }
+    qc = c->quic->connection;
+    conf = qc->conf;


The following 317 lines of 'qc->conf = conf;' do not require reassignment

hongzhidao · 2025-07-21T15:32:15Z

Hi @arut, Does it make sense not to reuse ngx_http_proxy_ctx_t? I checked the implementation, we only reuse vars and tailers, right? Take a look at this. 5052583

What about the variables like $proxy_host? I agree it would be nice to keep ctx private though, if it's not too hard.

Good question. It looks like ngx_http_proxy_ctx_t is required for all of them.
What about the design? There is only one module ngx_http_proxy_module with ngx_http_proxy_ctx_t, we don't introduce ngx_http_v2_proxy_module and ngx_http_v3_proxy_module. But we can add their private fields. For example:
#if (NGX_HTTP_V2)
    http2 fields from `ngx_http_v2_proxy_ctx_t` and renamed it `ngx_http_v2_proxy_t`.
#endif

#if (NGX_HTTP_V3)
    http2 fields from `ngx_http_v3_proxy_ctx_t` and renamed it `ngx_http_v3_proxy_t`.
#endif

typedef struct {
    ...
    unsigned                       connection_type:2;

#if (NGX_HTTP_V2)
    ngx_http_v2_proxy_t            http2;
#endif
#if (NGX_HTTP_V3)
    ngx_http_v2_proxy_t            http3;
#endif

} ngx_http_proxy_ctx_t;
Then we can access them like ctx->http2, ctx->http3.
I'm not sure if it helps. Also, Ideally these modules should be independent.

I agree with the related suggestions.
#771 (comment)
Will focus on buffering and cache first.

This is a preparation for introducing QUIC client support.

The function is called by event pipe which is used when proxying an HTTP connection.

This is a preparation for adding QUIC client support.

The change removes ngx_quic_run() and adds the following functions: - ngx_quic_handshake() - ngx_quic_shutdown() - ngx_quic_get_error() - ngx_quic_accept_stream() - ngx_quic_reject_streams() The first two resemble similar SSL functions. A typical QUIC workflow is now application-centered. After initiating the handshake, the application sets c->ssl->handshake which is first called once the handshake is complete. After that it's called after processing each QUIC event. Inside the handler the application can accept new streams, check QUIC errors and shut down the connection using the above mentioned functions. The change also introduces QUIC connection lingering. When closing a QUIC connection with ngx_quic_shutdown(), first the connections switches to the lingering mode. In this mode all unsent stream data is sent and acked. The maximum timeout for lingering is 3 seconds. The same valus is currently used by ngx_ssl_shutdown().

Now c->quic is used for both QUIC connections and streams. It allows to avoid using c->udp to access QUIC connection object. Also, it allows for early allocation of QUIC connection, which can hold QUIC configuration before the handshake. QUIC handshake function is now split into ngx_quic_create_connection() and ngx_quic_handshake() in preparation to adding QUIC client support. QUIC configuration is passed only to ngx_quic_create_connection(). This brings QUIC API in line with SSL API.

Similar to SSL, client mode is enabled by passing NGX_SSL_CLIENT flag to ngx_quic_create_connection().

A number of changes in HTTP/3 server code in preparation for adding HTTP/3 client support: - parse, table and uni function no longer access server configurations - parse functions no longer call action functions and use callbacks instead - added session "data" field, which stores ngx_http_connection_t reference - added session fields "max_table_capacity" and "max_blocked_streams" - session field "blocked" is renamed to "queue" - ngx_http_v3_close_connection() is moved to ngx_http_v3.c and made public - made QPACK header constants public

The module allows to use HTTP/3 protocol for proxying. HTTP/3 proxying is enabled by specifying "proxy_http_version 3.0". Example: server { listen 8000; location / { proxy_http_version 3.0; proxy_pass https://127.0.0.1:8443; } } server { listen 8443 quic reuseport; ssl_certificate certs/example.com.crt; ssl_certificate_key certs/example.com.key; location / { return 200 foo; } }

arut · 2025-07-24T16:43:17Z

Removing the host+port patch from here. It's committed to mainline: #772

arut added the poc proof of concept label Jul 6, 2025

Maryna-f5 added this to the nginx-1.29.2 milestone Jul 7, 2025

arut mentioned this pull request Jul 7, 2025

HTTP/3: fixed handling :authority and Host with port. #772

Merged

Maryna-f5 removed this from the nginx-1.29.2 milestone Jul 7, 2025

hongzhidao previously approved these changes Jul 8, 2025

View reviewed changes

src/http/ngx_http_upstream.c Show resolved Hide resolved

hongzhidao self-requested a review July 8, 2025 14:18

arut requested a review from pluknet July 8, 2025 14:26

hongzhidao reviewed Jul 8, 2025

View reviewed changes

src/http/ngx_http_upstream.c Show resolved Hide resolved

bullerdu reviewed Jul 14, 2025

View reviewed changes

bullerdu reviewed Jul 18, 2025

View reviewed changes

src/event/quic/ngx_event_quic.h Outdated

Copy link

bullerdu Jul 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ngx_quic_init_pt and ngx_quic_shutdown_pt also need to remove

bullerdu reviewed Jul 18, 2025

View reviewed changes

arut added 8 commits July 24, 2025 20:17

Events: ignoring handling events for shared connections.

b1df8a2

QUIC: renamed client/server keys to read/write.

1f0b9ad

This is a preparation for introducing QUIC client support.

QUIC: renamed client/server streams to local/remote.

1c8227d

This is a preparation for introducing QUIC client support.

QUIC: stream recv_chain support.

e933c63

The function is called by event pipe which is used when proxying an HTTP connection.

QUIC: detailed logging of transmitted packets.

49628f2

QUIC: ngx_quic_is_stream_local() function.

589e433

This is a preparation for adding QUIC client support.

arut added 5 commits July 24, 2025 20:17

QUIC: client support.

58c741b

Similar to SSL, client mode is enabled by passing NGX_SSL_CLIENT flag to ngx_quic_create_connection().

QUIC: allow changing main connection log.

89a36fe

HTTP/3: modified ngx_http_v3_encode_field_l() prototype.

ac73e8a

arut force-pushed the http3-upstream branch from d69634d to 02105aa Compare July 24, 2025 16:43

HanadaLee mentioned this pull request Jul 24, 2025

feature: proxy_ssl_verify_by_lua directives openresty/openresty#1066

Open

arut mentioned this pull request Jul 25, 2025

reverse proxy to http3/quic #496

Open

HTTP/3 to upstream PoC #770

Are you sure you want to change the base?

HTTP/3 to upstream PoC #770

Conversation

arut commented Jul 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Module

Cache

Example

Multiplexing

Uh oh!

HanadaLee commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hongzhidao commented Jul 7, 2025

Uh oh!

arut commented Jul 7, 2025

Uh oh!

Uh oh!

Uh oh!

bullerdu Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

arut Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

hongzhidao commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arut commented Jul 16, 2025

Uh oh!

arut commented Jul 16, 2025

Uh oh!

hongzhidao commented Jul 16, 2025

Uh oh!

arut commented Jul 16, 2025

Uh oh!

hongzhidao commented Jul 16, 2025

Uh oh!

bullerdu Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

bullerdu Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

hongzhidao commented Jul 21, 2025

Uh oh!

arut commented Jul 24, 2025

Uh oh!

Uh oh!

arut commented Jul 6, 2025 •

edited

Loading

HanadaLee commented Jul 7, 2025 •

edited

Loading

hongzhidao commented Jul 16, 2025 •

edited

Loading