Google Git
Sign in
chromium / chromium / src / main / . / tools / rust / README.md
blob: c9ab47ac7f7e505f9d4db3343a730b52a5fa04c6 [file] [log] [blame] [view]
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]1# //tools/rust
2
3This directory contains scripts for building, packaging, and distributing the
4Rust toolchain (the Rust compiler, and also C++/Rust FFI tools like
5[Crubit](https://github.com/google/crubit)).
6
Lukasz Anforowicz2576d8dff2022-07-22 16:23:18[diff] [blame]7[TOC]
8
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]9
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]10## Background
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]11
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]12Like with Clang, Chromium uses bleeding edge Rust tooling. We track the upstream
13projects' latest development as closely as possible. However, Chromium cannot
14use official Rust builds for various reasons which require us to match the Rust
danakjf3d99d152022-11-28 21:24:51[diff] [blame]15LLVM backend version with the Clang we use.
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]16
17It would not be reasonable to build the tooling for every Chromium build, so we
18build it centrally (with the scripts here) and distribute it for all to use
19(also fetched with the scripts here).
20
Arthur Eubanksbe10ddc62024-05-13 21:09:26[diff] [blame]21Similar to the Clang package which exists as a tarball that is unpacked into
22`third_party/llvm-build`, the Rust package exists as a tarball that is unpacked
23into `third_party/rust-toolchain`.
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]24
25## Rust build overview
26
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]27Each Rust package is built from an Rust git, usually from HEAD directly, along
28with the current Clang/LLVM revision in use in Chromium. Hence a new Rust
29package must be built whenever either Rust or Clang is updated. When building
30Rust we also build additional tools such as clippy and rustfmt, and interop
31tools including bindgen and crubit.
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]32
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]33The Rust build also includes building LLVM for rustc to use, and Clang for
34bindgen and crubit to use.
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]35
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]36The `*_upload_clang` and `*_upload_rust` trybots are used to build Clang and
37Rust respectively from the revisions specified in the Chromium source tree.
38These are uploaded to a storage bucket when the build succeeds. After being
39copied from staging to production by a developer (see
40[cs/copy_staging_to_prod_and_goma.sh](
41http://cs/copy_staging_to_prod_and_goma.sh)), they can then be fetched by
42`gclient sync`.
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]43
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]44The `update_rust.py` script is used by `gclient sync` to fetch the Rust
45toolchain for the revisions specified in the script.
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]46
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]47## Rolling Rust
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]48
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]49Follow the directions in [//docs/updating_clang.md](
50../../docs/updating_clang.md) to roll Clang and Rust together. To just
51roll Rust on its own, use the `--skip-clang` argument when running
52`upload_revision.py`.
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]53
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]54The upload_revision.py script will update the revision of Rust to be
55built and used in `update_rust.py` and will start the trybots that
56will build the Rust toolchain.
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]57
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]58After the build has succeeded and the new toolchain has been copied to
59production, the CQ will run trybots to verify that our code still builds
60and tests pass with the new Rust toolchain.
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]61
Arthur Eubanksbe10ddc62024-05-13 21:09:26[diff] [blame]62### An overview of what is updated in a Rust roll
63
64During Rust packaging, the upstream Rust sources are checked out into
65`third_party/rust-src`.
66
67During a Rust roll, a couple of things get updated. The most obvious one is
68various toolchain binaries like `rustc` that live in
69`third_party/rust-toolchain/bin`. These are the direct outputs of a Rust
70toolchain build.
71
72We also update the Rust standard library. We actually provide two copies of the
73standard library in Chromium: a prebuilt version only for use in host tools
74(e.g. build scripts, proc macros), and a version built from source as part of
75the normal Chromium build for use in target artifacts. These are the same
76version of the standard library that the Rust toolchain revision provides.
77
78The reason we have a prebuilt version of the standard library for use in host
79tools is that they are often loaded into `rustc` as a module, so to be safe we
80use the same prebuilts that the toolchain linked against. These are copied from
81the Rust toolchain build to
82`third_party/rust-toolchain/lib/rustlib/$PLATFORM/lib/*.rlib`. We use these
83when the gn arg `rust_prebuilt_stdlib` is true, which is manually set to true
84for gn host toolchains.
85
86The sources of the standard library we build from source for target artifacts
87live in `third_party/rust-toolchain/lib/rustlib/src/rust`. These are copied
88from `third_party/rust-src`. Since Chromium uses gn as its build system, we
89need some way to translate build files from Rust's build system, cargo, to gn
90rules. This is the responsibility of `gnrt`, which is a Chromium-specific tool
91that lives in [`tools/crates/gnrt`](https://crsrc.org/c/tools/crates/gnrt/),
Lukasz Anforowicz7c78315a2025-03-12 15:35:39[diff] [blame]92written in Rust. `gnrt gen` takes a cargo workspace, runs `cargo metadata`
93(or, more accurately `cargo guppy`) on
Arthur Eubanksbe10ddc62024-05-13 21:09:26[diff] [blame]94it to get information about sources and dependencies, and outputs gn rules
95corresponding to the cargo build rules. Rust has a
96[`sysroot`](https://github.com/rust-lang/rust/tree/master/library/sysroot)
97crate roughly corresponding to a top level cargo workspace we want for the
98standard library. However, we want a couple of customizations without having to
99patch the Rust sources, so we have another crate
100[`fake_root`](https://crsrc.org/c/build/rust/std/fake_root/) above that depends
101on `sysroot`.
102[`tools/rust/gnrt_stdlib.py`](https://crsrc.org/c/tools/rust/gnrt_stdlib.py)
103fetches and invokes the pinned `cargo` (see [`rustc` bootstrapping
104explanation](https://rustc-dev-guide.rust-lang.org/building/bootstrapping/what-bootstrapping-does.html),
105"pinned" is the "stage0" toolchain) to build and run `gnrt` with `fake_root` as
106the base workspace, generating an updated
107[`build/rust/std/rules/BUILD.gn`](https://crsrc.org/c/build/rust/std/rules/BUILD.gn)
108that has gn rules for the new standard library sources. For convenience when
109rolling Rust, this is one big `BUILD.gn` file as opposed to multiple files per
110crate. Note that because we do not ship cargo build files in
111`third_party/rust-toolchain`, we must run `gnrt` against `third_party/rust-src`
112instead of `third_party/rust-toolchain`. But end users do not have
113`third_party/rust-src` checked out, so we must rewrite the
114`third_party/rust-src` paths to the copies of the sources in
115`third_party/rust-toolchain/lib/rustlib/src/rust`, which is checked out by end
116users as part of the Rust toolchain.
117
118As an aside, `gnrt` is also used to generate gn build rules for
119non-standard-library Rust packages in `third_party/rust` used in Chromium's
120build. It uses
121[`third_party/rust/chromium_crates_io`](https://crsrc.org/c/third_party/rust/chromium_crates_io)
122as the base workspace and vendors sources into
123`third_party/rust/chromium_crates_io/vendor`. The `--for-std` argument to `gnrt
124gen` does different things for creating gn rules for the standard library
125versus for various non-standard-library packages, such as producing a single
126BUILD.gn file.
127
danakjeed02142023-07-19 14:59:28[diff] [blame]128### Possible failure: Missing sources or inputs
129
130A build error when building the stdlib in Chromium may look like:
131```
132FAILED: local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib
133...build command...
134ERROR: file not in GN sources: ../../third_party/rust-toolchain/lib/rustlib/src/rust/library/std/src/../../portable-simd/crates/std_float/src/lib.rs
135```
136
137Or:
138```
139FAILED: local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib
140...build command...
141ERROR: file not in GN inputs: ../../third_party/rust-toolchain/lib/rustlib/src/rust/library/std/src/../../stdarch/crates/core_arch/src/core_arch_docs.md
142```
143
144When building the stdlib in Chromium, the GN rules must have every rust source
145or other input file that makes up the crate listed in the `sources` and
146`inputs` GN variables. gnrt will walk the directory tree from the root of the
147crate and put every relevant file into the set. But sometimes a crate includes
148modules from paths outside the crate root's directory tree, with a path
149directive such as
150```rs
151#[path = "../../stuff.rs"]
152mod stuff;
153```
154or will `include!()` a file from another path, which is common for `.md` files:
155```rs
156include!("../../other_place.md")
157```
158
159The first error is saying the source file `std_float/src/lib.rs` did not
160appear in the `sources` variable. The `../../` part of the path shows that
161this is outside the crate root's directory tree. The second error is saying
162that `core_arch/src/core_arch_docs.md` did not appear in the `inputs` variable.
163
164To fix the error:
165* Determine the path that is missing, relative to the crate root. In the above
166 example this is `../../portable-simd/crates/std_float/src`. We could also use
167 `../../portable-simd` or anything in between, though that would add a lot
168 more sources to the GN rules than is necessary in this case. It's best to
169 point to the directory of the module root (where the `lib.rs` or `mod.rs`
170 is located).
Alan Zhaode04e7f2024-01-31 22:54:42[diff] [blame]171* Download the roll CL (on Gerrit, click on the 3 dots in the upper right
172 corner and click on "Download patch").
danakjeed02142023-07-19 14:59:28[diff] [blame]173* Find the failing build target crate's rules in
174 `//build/rust/std/gnrt_config.toml`. The failing crate in the above example
175 is `libstd.rlib`, so we want the `[crate.std]` section of the config file.
danakj324f546552024-09-18 16:31:11[diff] [blame]176* Determine if the target being built is a library or a build script. Build
177 script targets end with the suffix `_build_script`. For example:
178 ```
179 [13627/84339] RUST(BIN) clang_x64_for_rust_host_build_tools/compiler_builtins_compiler_builtins_vunknown_build_script
Lukasz Anforowicze4eded42025-08-04 15:58:24[diff] [blame]180 python3 ../../build/rust/gni_impl/rustc_wrapper.py --rustc=../../third_party/rust-toolchain/bin/rustc --depfi...(too long)
danakj324f546552024-09-18 16:31:11[diff] [blame]181 ERROR: file not in GN sources: ../../third_party/rust-toolchain/lib/rustlib/src/rust/library/vendor/compiler_builtins-0.1.123/configure.rs
182 ```
danakjeed02142023-07-19 14:59:28[diff] [blame]183* Determine if the missing file should go in `sources` or `inputs`.
184 * For `sources`, add the path to a `extra_src_roots` list in the crate's
185 rules. For the above example, we could add
186 `extra_src_roots = ['../../portable-simd/crates/std_float/src']`.
danakj324f546552024-09-18 16:31:11[diff] [blame]187 * Or if it was a build script target, then
188 `extra_build_script_src_roots = ['../../portable-simd/crates/std_float/src']`.
danakjeed02142023-07-19 14:59:28[diff] [blame]189 * For `inputs`, add the path to a `extra_input_roots` list in the crate's
190 rules. For the above example, we could add
191 `extra_input_roots = ['../../stdarch/crates/core_arch/src']`.
danakj324f546552024-09-18 16:31:11[diff] [blame]192 * Or if it was a build script target, then
193 `extra_build_script_input_roots = ['../../stdarch/crates/core_arch/src']`.
Alan Zhaode04e7f2024-01-31 22:54:42[diff] [blame]194* With the roll CL checked out, run `gclient sync`.
danakj324f546552024-09-18 16:31:11[diff] [blame]195
Alan Zhaode04e7f2024-01-31 22:54:42[diff] [blame]196*** note
197NOTE: `gclient sync` will download the version of the rust toolchain from the
198roll CL. In order for this to work, the upload_rust bots should've completed and
199`copy_staging_to_prod_and_goma.sh should've been run.
200***
danakj324f546552024-09-18 16:31:11[diff] [blame]201
danakjeed02142023-07-19 14:59:28[diff] [blame]202* Run `tools/rust/gnrt_stdlib.py` to use gnrt to rebuild the stdlib GN rules
203 using the updated config.
204
danakj324f546552024-09-18 16:31:11[diff] [blame]205*** note
206NOTE: All gnrt_config options are found in
207[//tools/crates/gnrt/lib/config.rs](https://source.chromium.org/chromium/chromium/src/+/main:tools/crates/gnrt/lib/config.rs).
208The `CrateConfig` type has the various per-crate config options.
209***
210
danakjbb4d0c772023-10-13 13:22:28[diff] [blame]211### Generating `BUILD.gn` files for stdlib crates
212
213If the build structure changes in any way during a roll, the GN files need
214to be regenerated.
215
216#### Simple way:
217
218Run `tools/rust/gnrt_stdlib.py`.
219
220#### Longer way
221
222This requires Rust to be installed and available in your system, typically
223through [https://rustup.rs](https://rustup.rs).
224
225To generate `BUILD.gn` files for the crates with the `gnrt` tool:
2261. Change directory to the root `src/` dir of Chromium.
2271. Build `gnrt` to run on host machine: `cargo build --release --manifest-path
228 tools/crates/gnrt/Cargo.toml --target-dir out/gnrt`.
2291. Ensure you have a checkout of the Rust source tree in `third_party/rust-src`
230 which can be done with `tools/rust/build_rust.py --sync-for-gnrt`.
2311. Run `gnrt` with the `gen` action:
232 `out/gnrt/release/gnrt gen --for-std third_party/rust-src`.
233
234This will generate the `//build/rust/std/rules/BUILD.gn` file, with the changes
235visible in `git status` and can be added with `git add`.
236
237## Local development
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]238
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]239To build the Rust toolchain locally, run `//tools/rust/build_rust.py`. It
240has additional flags to skip steps if you're making local changes and want
241to retry a build. The script will produce its outputs in
242`//third_party/rust-toolchain/`, which is the same place that `gclient sync`
243places them.
Collin Baker00a21532022-11-16 21:44:24[diff] [blame]244
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]245Building the `rust_build_tests` GN target is a good way to quickly verify the
246toolchain is working.
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]247
248## Rolling Crubit tools
249
250Steps to roll the Crubit tools (e.g. `rs_bindings_from_cc` tool)
251to a new version:
252
Lukasz Anforowicz2576d8dff2022-07-22 16:23:18[diff] [blame]253- Locally, update `CRUBIT_REVISION` in `update_rust.py`.
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]254 (Update `CRUBIT_SUB_REVISION` when the build or packaging is changed, but
255 the upstream Rust revision we build from is not changed.)
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]256
257- Locally, update `crubit_revision` in `//DEPS`, so that it matches
258 the revision from the previous bullet item.
259
260- Run manual tests locally (see the "Building and testing the tools locally"
261 section below).
Alison Gale4d9c2312024-04-26 19:15:24[diff] [blame]262 TODO(crbug.com/40226863): These manual steps should
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]263 be made obsolete once Rust-specific tryjobs cover Crubit
264 tests.
265
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]266## Building and testing Crubit locally
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]267
268### Prerequisites
269
Lukasz Anforowicz2576d8dff2022-07-22 16:23:18[diff] [blame]270#### Bazel
271
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]272`build_crubit.py` depends on Bazel.
Lukasz Anforowicz2576d8dff2022-07-22 16:23:18[diff] [blame]273
274To get Bazel, ensure that you have `checkout_bazel` set in your `.gclient` file
275and then rerun `gclient sync`:
276
277```sh
278$ cat ../.gclient
279solutions = [
280 {
281 "name": "src",
282 "url": "https://chromium.googlesource.com/chromium/src.git",
283 ...
284 "custom_vars": {
285 "checkout_bazel": True,
Adrian Taylor8a824c22023-03-01 07:04:14[diff] [blame]286 "checkout_crubit": True,
Lukasz Anforowicz2576d8dff2022-07-22 16:23:18[diff] [blame]287 },
288 },
289]
290```
291
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]292### Building
293
danakj358a7f4a2023-04-13 22:06:01[diff] [blame]294Just run `tools/rust/build_crubit.py`. So far `build_crubit.py` has only been
295tested on Linux hosts.
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]296
297### Deploying
298
Lukasz Anforowicz401dc9c2022-10-20 18:45:00[diff] [blame]299`build_crubit.py` will copy files into the directory specified in the
300(optional) `--install-to` cmdline parameter - for example:
301
302```
303$ tools/rust/build_crubit.py --install-to=third_party/rust-toolchain/bin/
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]304```
305
306### Testing
307
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]308Crubit tests are under `//build/rust/tests/test_rs_bindings_from_cc`. Until
309Crubit is built on the bots, the tests are commented out in
310`//build/rust/tests/BUILD.gn`, but they should still be built and run before
Alison Gale4d9c2312024-04-26 19:15:24[diff] [blame]311rolling Crubit. TODO(crbug.com/40226863): Rephrase this paragraph
Lukasz Anforowiczc105ecee92022-07-19 23:03:03[diff] [blame]312after Crubit is built and tested on the bots.