Fix: Not loosing the access token when calling UserCredentials#ToBuil…#993
Conversation
…als#ToBuilder()
|
🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use -- conventional-commit-lint bot |
TimurSadykov
left a comment
There was a problem hiding this comment.
Thanks for the contribution! Few minor comments.
oauth2_http/javatests/com/google/auth/oauth2/UserCredentialsTest.java
Outdated
Show resolved
Hide resolved
oauth2_http/javatests/com/google/auth/oauth2/UserCredentialsTest.java
Outdated
Show resolved
Hide resolved
|
@TimurSadykov As I was making the test a little bit more generic than just checking the access token, I found out that two methods of the builders were missing. |
|
@TimurSadykov any news on this one? |
TimurSadykov
left a comment
There was a problem hiding this comment.
Sorry for a delay.
LGTM, i will merge once the last nit is fixed.
| } | ||
|
|
||
| @Test | ||
| public void UserCredentials_ToBuilder_CopyEveryAttributes() { |
There was a problem hiding this comment.
nit, every part has to stat with lowercase, like this and a typo fix: userCredentials_toBuilder_copyEveryAttribute
oauth2_http/javatests/com/google/auth/oauth2/UserCredentialsTest.java
Outdated
Show resolved
Hide resolved
### What changes were proposed in this pull request? This PR aims to upgrade gcs-connector to 2.2.17. ### Why are the changes needed? To have the latest auth updates, - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15) - GoogleCloudDataproc/hadoop-connectors#1041 ```xml - <google.auth.version>1.12.1</google.auth.version> + <google.auth.version>1.14.0</google.auth.version> - <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version> + <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version> ``` - https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06) - googleapis/google-auth-library-java#1100 - googleapis/google-auth-library-java#993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BUILD** ``` dev/make-distribution.sh -Phadoop-cloud ``` **TEST** ``` $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1692645442153). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312) Type in expressions to have them evaluated. Type :help for more information. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #42588 from dongjoon-hyun/SPARK-44898. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
### What changes were proposed in this pull request? This PR aims to upgrade gcs-connector to 2.2.17. ### Why are the changes needed? To have the latest auth updates, - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15) - GoogleCloudDataproc/hadoop-connectors#1041 ```xml - <google.auth.version>1.12.1</google.auth.version> + <google.auth.version>1.14.0</google.auth.version> - <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version> + <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version> ``` - https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06) - googleapis/google-auth-library-java#1100 - googleapis/google-auth-library-java#993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BUILD** ``` dev/make-distribution.sh -Phadoop-cloud ``` **TEST** ``` $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1692645442153). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312) Type in expressions to have them evaluated. Type :help for more information. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42588 from dongjoon-hyun/SPARK-44898. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
### What changes were proposed in this pull request? This PR aims to upgrade gcs-connector to 2.2.17. ### Why are the changes needed? To have the latest auth updates, - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15) - GoogleCloudDataproc/hadoop-connectors#1041 ```xml - <google.auth.version>1.12.1</google.auth.version> + <google.auth.version>1.14.0</google.auth.version> - <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version> + <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version> ``` - https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06) - googleapis/google-auth-library-java#1100 - googleapis/google-auth-library-java#993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BUILD** ``` dev/make-distribution.sh -Phadoop-cloud ``` **TEST** ``` $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1692645442153). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312) Type in expressions to have them evaluated. Type :help for more information. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42588 from dongjoon-hyun/SPARK-44898. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
### What changes were proposed in this pull request? This PR aims to upgrade gcs-connector to 2.2.17. ### Why are the changes needed? To have the latest auth updates, - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15) - GoogleCloudDataproc/hadoop-connectors#1041 ```xml - <google.auth.version>1.12.1</google.auth.version> + <google.auth.version>1.14.0</google.auth.version> - <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version> + <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version> ``` - https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06) - googleapis/google-auth-library-java#1100 - googleapis/google-auth-library-java#993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BUILD** ``` dev/make-distribution.sh -Phadoop-cloud ``` **TEST** ``` $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1692645442153). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312) Type in expressions to have them evaluated. Type :help for more information. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42588 from dongjoon-hyun/SPARK-44898. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 109e9cd)
### What changes were proposed in this pull request? This PR aims to upgrade gcs-connector to 2.2.17. ### Why are the changes needed? To have the latest auth updates, - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15) - GoogleCloudDataproc/hadoop-connectors#1041 ```xml - <google.auth.version>1.12.1</google.auth.version> + <google.auth.version>1.14.0</google.auth.version> - <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version> + <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version> ``` - https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06) - googleapis/google-auth-library-java#1100 - googleapis/google-auth-library-java#993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BUILD** ``` dev/make-distribution.sh -Phadoop-cloud ``` **TEST** ``` $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1692645442153). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312) Type in expressions to have them evaluated. Type :help for more information. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42588 from dongjoon-hyun/SPARK-44898. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 109e9cd) (cherry picked from commit 0cb12e9)
### What changes were proposed in this pull request? This PR aims to upgrade gcs-connector to 2.2.17. ### Why are the changes needed? To have the latest auth updates, - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15) - GoogleCloudDataproc/hadoop-connectors#1041 ```xml - <google.auth.version>1.12.1</google.auth.version> + <google.auth.version>1.14.0</google.auth.version> - <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version> + <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version> ``` - https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06) - googleapis/google-auth-library-java#1100 - googleapis/google-auth-library-java#993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BUILD** ``` dev/make-distribution.sh -Phadoop-cloud ``` **TEST** ``` $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1692645442153). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312) Type in expressions to have them evaluated. Type :help for more information. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42588 from dongjoon-hyun/SPARK-44898. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 109e9cd) (cherry picked from commit 0cb12e9)
### What changes were proposed in this pull request? This PR aims to upgrade gcs-connector to 2.2.17. ### Why are the changes needed? To have the latest auth updates, - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15) - GoogleCloudDataproc/hadoop-connectors#1041 ```xml - <google.auth.version>1.12.1</google.auth.version> + <google.auth.version>1.14.0</google.auth.version> - <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version> + <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version> ``` - https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06) - googleapis/google-auth-library-java#1100 - googleapis/google-auth-library-java#993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BUILD** ``` dev/make-distribution.sh -Phadoop-cloud ``` **TEST** ``` $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1692645442153). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312) Type in expressions to have them evaluated. Type :help for more information. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42588 from dongjoon-hyun/SPARK-44898. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 109e9cd) (cherry picked from commit 0cb12e9)
### What changes were proposed in this pull request? This PR aims to upgrade gcs-connector to 2.2.17. ### Why are the changes needed? To have the latest auth updates, - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15) - GoogleCloudDataproc/hadoop-connectors#1041 ```xml - <google.auth.version>1.12.1</google.auth.version> + <google.auth.version>1.14.0</google.auth.version> - <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version> + <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version> ``` - https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06) - googleapis/google-auth-library-java#1100 - googleapis/google-auth-library-java#993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BUILD** ``` dev/make-distribution.sh -Phadoop-cloud ``` **TEST** ``` $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1692645442153). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312) Type in expressions to have them evaluated. Type :help for more information. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42588 from dongjoon-hyun/SPARK-44898. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 109e9cd) (cherry picked from commit 0cb12e9)
### What changes were proposed in this pull request? This PR aims to upgrade gcs-connector to 2.2.17. ### Why are the changes needed? To have the latest auth updates, - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15) - GoogleCloudDataproc/hadoop-connectors#1041 ```xml - <google.auth.version>1.12.1</google.auth.version> + <google.auth.version>1.14.0</google.auth.version> - <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version> + <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version> ``` - https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06) - googleapis/google-auth-library-java#1100 - googleapis/google-auth-library-java#993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BUILD** ``` dev/make-distribution.sh -Phadoop-cloud ``` **TEST** ``` $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1692645442153). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312) Type in expressions to have them evaluated. Type :help for more information. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42588 from dongjoon-hyun/SPARK-44898. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 109e9cd) (cherry picked from commit 0cb12e9)
Fixes #992 ☕️