Hello, good morning! I was wondering if the team i...
# ask-the-community
s
Hello, good morning! I was wondering if the team is planning to hold office hours today? My team has a quick question on max task nodes and figured it might be a good place to address it. Thanks!! @Dan Rammer (hamersaw) @Samhita Alla
d
Hey @Seth Baer, we have office hours setup for Wednesdays, but we have been discussing revisiting the schedule to better trade-off our time devotion with user participation. If you have any thoughts on how we might improve availability, we would be happy to hear them!
Regarding your question - is it something we can discuss here on slack? Otherwise we might be able to setup a brief call later today.
s
Oh right, no worries! Yes, I think we can discuss here and if there's still questions we can attend the regular office hours tomorrow. We recently ran into the node limit for a workflow (it seems to be set to 100). We looked around and made this change to the Admin config map,
Copy code
registration.yaml: |
    maxWorkflowNodes: 1500
but still hit the same error during registration. For context, the error looked like this during registration:
Copy code
InvalidArgument desc = number of nodes in workflow
exceeds limit (1426 > 100)
d
Ok, it looks like FlyteAdmin isn't recognizing the configuration. I think this may need to be declared at another level, but double checking just to make sure. Something like:
Copy code
registration.yaml: |
    registration:
      maxWorkflowNodes: 1500
Yeah, so this works locally. We have been discussing solutions to be strict in parsing configuration options - so the binary fails if there is a value defined incorrectly. But there are issues with supporting different environments (ex. having all configuration in a single
ConfigMap
). Perhaps it's something we need to revisist.
s
Ah, I see! Let me try that variation in how we deployed the yaml. I do feel it would be helpful to know if errors are coming from the way we define values in the manifests. We're also trying to get GPUs on our platform but it's been a little hard to know if the problem is in the definitions, in our underlying kubernetes infra, or elsewhere. But of course, there's tradeoffs as you mentioned!
@Dan Rammer (hamersaw) That worked!! Thanks very much for the guidance
106 Views