<#6521 Fix workflow equality check> Pull request o...
# flyte-github
c
#6521 Fix workflow equality check Pull request opened by Sovietaced ## Tracking issue Closes #6520 ## Why are the changes needed? The core propeller logic has a branch which compares the workflow read from etcD to the workflow that may have been mutated by the propeller state machine logic.The intention is that if the workflow status is the same it does not bother writing the workflow to etcD. The comparator logic for this operation includes bugs where two workflow that have the same state are considered different and thus trigger an etcD write (or at least an API call). As such, a shockingly high amount of our workflow update are redundant, wasted API calls (~60%). [Screenshot 2025-07-03 at 1 47 34 PM](https://private-user-images.githubusercontent.com/1274471/462207964-2a44bd07-4aaa-43c2-9963-eac2af0f0bd8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTMzNDUzOTMsIm5iZiI6MTc1MzM0NTA5MywicGF0aCI6Ii8xMjc0NDcxLzQ2MjIwNzk2NC0yYTQ0YmQwNy00YWFhLTQzYzItOTk2My1lYWMyYWYwZjBiZDgucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDcyNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTA3MjRUMDgxODEzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MTY5ZDczNzQyOTFlM2E4MGRmNWE2NmRlZGUwYjc0ODBiNzEyM2M3M2RkZGJjZWZkYzQ0Y2JlNTVmNWE1ZWY2YiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.U6FnZWTP4fite0Cx9g5CRX1w863n6rE07mGnqH_WCM4) [Screenshot 2025-07-03 at 1 47 50 PM](https://private-user-images.githubusercontent.com/1274471/462207986-1e560072-a16b-4987-a6ef-2d1d10ad25ee.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTMzNDUzOTMsIm5iZiI6MTc1MzM0NTA5MywicGF0aCI6Ii8xMjc0NDcxLzQ2MjIwNzk4Ni0xZTU2MDA3Mi1hMTZiLTQ5ODctYTZlZi0yZDFkMTBhZDI1ZWUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDcyNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTA3MjRUMDgxODEzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OWM0MmI4ZGVmYmRiZGUwZjBlYjBmZTQyYjUxODNhZmRlNWJhZmQxOWQ1YmNmMDExZmI2ZTVlYjEyNGMxM2RkZSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.KbpeFuVd7gt0RwlriUTPBTn7x7x1NRbz7aQQNAQSGQQ) ## What changes were proposed in this pull request? This pull request proposes to only set the
dirty
flags in TaskNodeStatus when a value is set that differs from the existing value. This should prevent workflows from being marked dirty during regular workflow processing, like from this call site. For posterity, I also eliminated fields now serialized to etcD from the
.Equals
comparator. Depending on the running workflows this pull request has an opportunity to massively reduce k8s API and etcD load at scale. Previously k8s API and etcD load would scale linearly with running workflows due to how workflows are regularly queued and processed. With this change we have been able to run thousands of concurrent workflows with minimal k8s and etcD load (4-5 operations per second). ## How was this patch tested? Tested in our non-production environment where it has eliminated redundant workflow updates. Will be landing in our production environment soon. No more redundant updates in non-prod [Screenshot 2025-07-03 at 1 58 56 PM](https://private-user-images.githubusercontent.com/1274471/462210042-543c0534-87d3-49d4-a1b0-106bddd202ba.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTMzNDUzOTMsIm5iZiI6MTc1MzM0NTA5MywicGF0aCI6Ii8xMjc0NDcxLzQ2MjIxMDA0Mi01NDNjMDUzNC04N2QzLTQ5ZDQtYTFiMC0xMDZiZGRkMjAyYmEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDcyNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTA3MjRUMDgxODEzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MzIzMTFjN2Y1OTYzODVhNmY0M2RiMmYyMGU3MDA3MGI3YjdmNGIwYjc1MzJiNzVjZjkwYTg2NzVlMWFhNTgxNyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.W9t-JAyZAsJd8664FwLcjhX75dst2FHn7JSOLRaK6Gk) Also eliminates many wasted API calls to update the workflow. <https://private-user-images.githubusercontent.com/1274471/462212214-50a340a6-e9f7-47de-9b7f-130c182ff10d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTMzNDUzOTMsIm5iZiI6MTc1MzM0NTA5MywicGF0aCI6Ii8xMjc0NDcxLzQ2MjIxMjIxNC01MGEzNDBhNi1lOWY3LTQ3ZGUtOWI3Zi0xMzBjMTgyZmYxMGQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDcyNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTA3MjRUMDgxODEzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZTJjMDEyMmFhY2UyZjUwMWU1NzQ0MjAwOWQwZjRkOWYxOThkODNlNTAyZjRiZmRiZDQ3NmZiNmRmNTY1ZWZiMCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.xp-VdG8EGQl4kmIUQs7F5W0uHIBh_2mEfN_ttM8uHa8|[Screenshot 2025-07-03 at 2 06 51 PM](https://private-user-images.githubusercontent.com/1274471/462212214-50a340a6-e9f7-47de-9b7f-130c182ff10d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTMzNDUzOTMsIm5iZiI6MTc1MzM0NTA5MywicGF0aCI6Ii8xMjc0NDcxLzQ2MjIxMjIxNC01MGEzNDBhNi1lOWY3LTQ3ZGUtOWI3Zi0xMzBjMTgyZmYxMGQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2Jlg… flyteorg/flyte