Skip to content

[SPARK-52892][SPARK-52893][SPARK-52894][PYTHON] Support TIME in the hour, minute, and second functions in PySpark #51586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from

Conversation

uros-db
Copy link
Contributor

@uros-db uros-db commented Jul 20, 2025

What changes were proposed in this pull request?

Implement the hour, minute, and second functions for TIME type in PySpark & PySpark Connect API.

Why are the changes needed?

Expand API support for the hour, minute, and second functions for TIME type.

Does this PR introduce any user-facing change?

Yes, the new functions for TIME type are now available in Python API.

How was this patch tested?

Added appropriate Python function tests.

  • pyspark.sql.tests.test_functions
  • pyspark.sql.tests.connect.test_parity_functions

Was this patch authored or co-authored using generative AI tooling?

No.

@uros-db uros-db marked this pull request as draft July 20, 2025 23:08
@uros-db uros-db changed the title [DRAFT][SPARK-52892][PYTHON] Implement the hour, minute, and second functions in PySpark [DRAFT][SPARK-52892][SPARK-52893][SPARK-52894][PYTHON] Implement the hour, minute, and second functions in PySpark Jul 20, 2025
Copy link
Contributor Author

@uros-db uros-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is waiting for the corresponding Scala functions PR: #51576.

@uros-db uros-db marked this pull request as ready for review July 27, 2025 18:58
@uros-db
Copy link
Contributor Author

uros-db commented Jul 27, 2025

Scala function PR is merged: #51576. Continuing the work on PySpark.

@uros-db uros-db changed the title [DRAFT][SPARK-52892][SPARK-52893][SPARK-52894][PYTHON] Implement the hour, minute, and second functions in PySpark [WIP][SPARK-52892][SPARK-52893][SPARK-52894][PYTHON] Implement the hour, minute, and second functions in PySpark Jul 27, 2025
@dongjoon-hyun dongjoon-hyun marked this pull request as draft July 27, 2025 21:03
@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Jul 27, 2025

Thank you, @uros-db . According to the PR title, I converted this to Draft status because GitHub Draft status provides more features like preventing merge mistake. Please make it ready when the PR is ready.

@uros-db uros-db changed the title [WIP][SPARK-52892][SPARK-52893][SPARK-52894][PYTHON] Implement the hour, minute, and second functions in PySpark [SPARK-52892][SPARK-52893][SPARK-52894][PYTHON] Support TIME in the hour, minute, and second functions in PySpark Jul 27, 2025
@uros-db uros-db marked this pull request as ready for review July 27, 2025 22:18
@uros-db
Copy link
Contributor Author

uros-db commented Jul 27, 2025

@zhengruifeng Please review. cc @MaxGekk

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @uros-db .

@dongjoon-hyun
Copy link
Member

cc @peter-toth

@uros-db
Copy link
Contributor Author

uros-db commented Jul 28, 2025

All checks passed, ready to merge. @dongjoon-hyun @HyukjinKwon @zhengruifeng

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants