cuddly-jelly-27016
07/10/2025, 2:57 PMmain branch
### Describe the bug
### Summary
The PodTemplateStore cache returns direct references to cached objects instead of copies, causing resource configurations from one workflow to leak into subsequent workflows. This results in workflows without inline resources unexpectedly inheriting resource specifications from previously executed workflows.
### Description
When multiple workflows execute using the same pod template, the second and subsequent workflows may inherit resource configurations (CPU, memory, GPU, etc.) from earlier workflows, even when no inline resources are specified. This happens because the PodTemplateStore.LoadOrDefault() method returns a pointer to the cached pod template object, which gets mutated during resource processing.
### Root Cause
In flyteplugins/go/tasks/pluginmachinery/flytek8s/pod_template_store.go, the LoadOrDefault() method returns a direct reference to the cached object:
func (p *PodTemplateStore) LoadOrDefault(namespace string, podTemplateName string) *v1.PodTemplate {
// ...
if podTemplate, ok := podTemplates.Load(namespace); ok {
return podTemplate.(*v1.PodTemplate) // ← BUG: Returns pointer to cached object
}
// ...
}
### Expected behavior
flytepropeller doesn't preserve resources different between runs
### Additional context to reproduce
1. Execute a workflow (Workflow A) that specifies inline resources (e.g., high CPU/memory requirements)
2. Execute a different workflow (Workflow B) using the same pod template but without inline resources
3. Observe that Workflow B unexpectedly uses the resource specifications from Workflow A
4. Restart flytepropeller to clear the cache
5. Execute Workflow B again - it now works correctly until another workflow with inline resources executes
### Screenshots
No response
### Are you sure this issue hasn't been raised already?
• Yes
### Have you read the Code of Conduct?
• Yes
flyteorg/flytecuddly-jelly-27016
07/13/2025, 4:50 PM