<#6501 Add SignalWatcher in Copilot> Pull request ...
# flyte-github
c
#6501 Add SignalWatcher in Copilot Pull request opened by machichima ## Tracking issue ## Why are the changes needed? When running
ContainerTask
, we discovered that the pod stuck on
uploader
container for a long time after the main task succeed. The
uploader
should not take so much time. This is because
uploader
sidecar starts after the task container finishes, so that it cannot detect any active process (the main container already completed), and it waits until timeout (1m40s). [image](https://private-user-images.githubusercontent.com/60069744/455231980-ffd84e3b-5eb5-47e4-a876-805522c06bff.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTA5MTQ0ODUsIm5iZiI6MTc1MDkxNDE4NSwicGF0aCI6Ii82MDA2OTc0NC80NTUyMzE5ODAtZmZkODRlM2ItNWViNS00N2U0LWE4NzYtODA1NTIyYzA2YmZmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA2MjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNjI2VDA1MDMwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTgyNzMxM2IzMzkwODQwN2Q4MDE5YmYwMjBhZGNkNmU5ODJhMDZhMWY1YWQxMWVmODU4ZjU1YWM2YWRmODkwNGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.0dK77TGeJ-PvLalTk2wI2WajsQm-eegdoAkAspZnMRs) ## What changes were proposed in this pull request? Convert the
uploader
container from "main container" to "sidecar container" (which is the init container with restartPolicy set to Always). This has following benefit: 1. Ensure the
uploader
container starts before main container 2. We can trap the
SIGTERM
signal to know when the main container completed. Therefore, we do not need to check if the main container start • Ref: https://kubernetes.io/docs/tutorials/configuration/pod-sidecar-containers/#benefits-of-a-built-in-sidecar-container In the PR, I made following modification: 1. Change uploader container to "sidecar contianer" 2. Add a new watcher:
SignalWatcher
3. Remove the
WaitToStart
logic for uploader 4. Use
WaitToExit
in Signal Watcher to check if the main container completed 5. Remove kube api watcher, file watcher 6. Let copilot use the logger level set in config ## How was this patch tested? Manual test with following flytekit script: command: pyflyte run --remote containertask.py wf script: import functools from flytekit import ContainerTask, kwtypes, workflow calculate_ellipse_area_python_template_style = ContainerTask( name="calculate_ellipse_area_python_template_style", input_data_dir="/var/inputs", output_data_dir="/var/outputs", inputs=kwtypes(a=float, b=float), outputs=kwtypes(area=float), image="ghcr.io/flyteorg/rawcontainers-python:v2", command=[ "python", "calculate-ellipse-area.py", "{{.inputs.a}}", "{{.inputs.b}}", "/var/outputs", ], ) @workflow def wf(a: float = 0.5, b: float = 0.4) -> float: res = calculate_ellipse_area_python_template_style(a, b) return res Result: see screenshot below ### Labels Please add one or more of the following labels to categorize your PR: • added: For new features. • changed: For changes in existing functionality. • deprecated: For soon-to-be-removed features. • removed: For features being removed. • fixed: For any bug fixed. • security: In case of vulnerabilities This is important to improve the readability of release notes. ### Setup process ### Screenshots With my modification, the output is upload immediately after main container completed (takes less than a second to finish the upload) [image](https://private-user-images.githubusercontent.com/60069744/459221195-a06216ad-7650-443d-b2ff-5897e4c6a451.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTA5MTQ0ODUsIm5iZiI6MTc1MDkxNDE4NSwicGF0aCI6Ii82MDA2OTc0NC80NTkyMjExOTUtYTA2MjE2YWQtNzY1MC00NDNkLWIyZmYtNTg5N2U0YzZhNDUxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA2MjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNjI2VDA1MDMwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI2YTZkZWFiMDQ5ZjFhOGU4NDY4Y2ZjNmNhNTlkZjRiNTY1YjNhZjI2MjVjODRjNmZhZWQwZDMxYzgzNmQ5OTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.GvGlwq1W2ddM4f1KqX70dchLNeqYwzaTARU2p_VymrQ) ## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. ## Related PRs ## Docs link https://bito.ai#summarystart| ### Summary by Bito This pull request enhances the uploader container by converting it into a sidecar container for improved lifecycle management. It introduces a new SignalWatcher for efficient termination signal handling, simplifies logging, and removes unnecessary watchers to streamline the codebase. These modifications aim to boost performance and significantly reduce upload times, resulting in a better user experience. flyteorg/flyte
#6501 [WIP] Improve Copilot Pull request ready for review by machichima flyteorg/flyte