cuddly-jelly-27016
06/19/2025, 10:35 AMContainerTask, we discovered that the pod stuck on uploader container for a long time after the main task succeed. The uploader should not take so much time. This is because uploader sidecar starts after the task container finishes, so that it cannot detect any active process (the main container already completed), and it waits until timeout (1m40s).
[image](https://private-user-images.githubusercontent.com/60069744/455231980-ffd84e3b-5eb5-47e4-a876-805522c06bff.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTA5MTQ0ODUsIm5iZiI6MTc1MDkxNDE4NSwicGF0aCI6Ii82MDA2OTc0NC80NTUyMzE5ODAtZmZkODRlM2ItNWViNS00N2U0LWE4NzYtODA1NTIyYzA2YmZmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA2MjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNjI2VDA1MDMwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTgyNzMxM2IzMzkwODQwN2Q4MDE5YmYwMjBhZGNkNmU5ODJhMDZhMWY1YWQxMWVmODU4ZjU1YWM2YWRmODkwNGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.0dK77TGeJ-PvLalTk2wI2WajsQm-eegdoAkAspZnMRs)
## What changes were proposed in this pull request?
Convert the uploader container from "main container" to "sidecar container" (which is the init container with restartPolicy set to Always). This has following benefit:
1. Ensure the uploader container starts before main container
2. We can trap the SIGTERM signal to know when the main container completed. Therefore, we do not need to check if the main container start
• Ref: https://kubernetes.io/docs/tutorials/configuration/pod-sidecar-containers/#benefits-of-a-built-in-sidecar-container
In the PR, I made following modification:
1. Change uploader container to "sidecar contianer"
2. Add a new watcher: SignalWatcher
3. Remove the WaitToStart logic for uploader
4. Use WaitToExit in Signal Watcher to check if the main container completed
5. Remove kube api watcher, file watcher
6. Let copilot use the logger level set in config
## How was this patch tested?
Manual test with following flytekit script:
command:
pyflyte run --remote containertask.py wf
script:
import functools
from flytekit import ContainerTask, kwtypes, workflow
calculate_ellipse_area_python_template_style = ContainerTask(
name="calculate_ellipse_area_python_template_style",
input_data_dir="/var/inputs",
output_data_dir="/var/outputs",
inputs=kwtypes(a=float, b=float),
outputs=kwtypes(area=float),
image="ghcr.io/flyteorg/rawcontainers-python:v2",
command=[
"python",
"calculate-ellipse-area.py",
"{{.inputs.a}}",
"{{.inputs.b}}",
"/var/outputs",
],
)
@workflow
def wf(a: float = 0.5, b: float = 0.4) -> float:
res = calculate_ellipse_area_python_template_style(a, b)
return res
Result: see screenshot below
### Labels
Please add one or more of the following labels to categorize your PR:
• added: For new features.
• changed: For changes in existing functionality.
• deprecated: For soon-to-be-removed features.
• removed: For features being removed.
• fixed: For any bug fixed.
• security: In case of vulnerabilities
This is important to improve the readability of release notes.
### Setup process
### Screenshots
With my modification, the output is upload immediately after main container completed (takes less than a second to finish the upload)
[image](https://private-user-images.githubusercontent.com/60069744/459221195-a06216ad-7650-443d-b2ff-5897e4c6a451.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTA5MTQ0ODUsIm5iZiI6MTc1MDkxNDE4NSwicGF0aCI6Ii82MDA2OTc0NC80NTkyMjExOTUtYTA2MjE2YWQtNzY1MC00NDNkLWIyZmYtNTg5N2U0YzZhNDUxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA2MjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNjI2VDA1MDMwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI2YTZkZWFiMDQ5ZjFhOGU4NDY4Y2ZjNmNhNTlkZjRiNTY1YjNhZjI2MjVjODRjNmZhZWQwZDMxYzgzNmQ5OTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.GvGlwq1W2ddM4f1KqX70dchLNeqYwzaTARU2p_VymrQ)
## Check all the applicable boxes
• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.
## Related PRs
## Docs link
https://bito.ai#summarystart|
### Summary by Bito
This pull request enhances the uploader container by converting it into a sidecar container for improved lifecycle management. It introduces a new SignalWatcher for efficient termination signal handling, simplifies logging, and removes unnecessary watchers to streamline the codebase. These modifications aim to boost performance and significantly reduce upload times, resulting in a better user experience.
flyteorg/flytecuddly-jelly-27016
06/19/2025, 10:35 AMcuddly-jelly-27016
06/24/2025, 5:07 PM