https://flyte.org logo
Powered by
a

Alex Bain

02/08/2022, 11:00 PM
Hey Flyte team, we (at Woven Planet) are hoping to start making good use of Flyte map tasks. I went through the Flyte OSS roadmap issues and saw that a lot of recent work has been done to get them up to speed (such as https://github.com/flyteorg/flyte/issues/1838 to propagate the status of subtasks). However, I can see on the latest 
flyteconsole:v0.41.0
that in the console, there are no log links to subtasks or any UI to show the status of subtasks. Can I ask how soon you plan to land these features in flyteconsole and if there are GitHub issues or PRs that I can follow / contribute to?
k

Ketan (kumare3)

02/08/2022, 11:02 PM
cc @Dan Rammer (hamersaw)
so @Alex Bain for 0.19.3 - End of Feb release we plan the following things in map tasks 1. Map sub-task retries 2. Map sub-task log link and other metadata 3. Map task - intra task checkpointing 4. Map task concurrency limits
We will also try to get the UI in, but that will probably land in March - cc @Jason Porter
a

Alex Bain

02/08/2022, 11:05 PM
@Ketan (kumare3) thanks for the update. I see Github issues for 1, 3, 4 but if you issues open for #2 and other UI issues let me know so I can follow along
We can contribute time this quarter for testing (and fixes)
k

Ketan (kumare3)

02/08/2022, 11:12 PM
ohh wait is #2 already checked in
@Alex Bain what version are you guys on?
it seemed you have not upgraded in a while?
d

Dan Rammer (hamersaw)

02/08/2022, 11:40 PM
With the newest map task fixes log links for the subtasks are shown under the top-level map task in the UI (as pictured). The subtask phases are being reported from the execution, but there's still some plumbing work to get them available to the UI. I suspect the plan is to update the UI so map task subtasks are displayed similar to dynamic worflows. @Jason Porter can probably talk more about this, as well as the timeline.
j

Jason Porter

02/09/2022, 12:25 AM
Yes, we have a new UX coming for mapped tasks; aiming for march on this 👍
a

Alex Bain

02/09/2022, 1:41 AM
Our prod environment is old, but in our dev environment I'm on v0.19.2 with 
flyteconsole:v0.41.0
. Let me double check if I get the log links for subtasks...
^^^^ Will wait to double check this tomorrow, I'm done for today
👍 1
k

Ketan (kumare3)

02/09/2022, 4:55 AM
@Alex Bain i also just noticed, but contributions welcome 😄
a

Alex Bain

02/11/2022, 7:11 PM
@Dan Rammer (hamersaw) @Jason Porter I can indeed run map tasks and get the k8s log links. We actually use Stackdriver log links for all Flyte tasks. I think I have my Propeller deployment YAML setup correctly to config 
k8s-array
plugin tasks to use Stackdriver links, but I still get the k8s log links (like in the screenshot a couple of comments above) when I execute my map tasks.
👀 1
Could you confirm if Stackdriver links for subtasks should work in flyteconsole for map tasks yet? If not, let me request support for that with the next set of UI enhancements for map tasks!
(Note: I'm not 100% sure that I have my k8s deployment yaml updated correctly with the right stackdriver config)
d

Dan Rammer (hamersaw)

02/11/2022, 7:26 PM
Hey @Alex Bain are Stackdriver logs working in regular k8s tasks (ie. container, sidecar) and just not working in map tasks? It looks like there is a separate config to specifically enable stackdriver logs in map tasks. So it should be as simple as copying the configuration if it's working elsewhere. Can you paste the plugins configuration yaml?
a

Alex Bain

02/11/2022, 10:31 PM
That's right - Stackdriver logs are working for our regular k8s tasks, but I can't get them to work for map tasks. I could have my YAML config wrong! My Propeller 
deployment.yaml
has two ConfigMaps, the first is:
Copy code
apiVersion: v1
kind: ConfigMap
metadata:
  name: flyte-propeller-config
  namespace: "prod"
data:
  propeller: |-
    propeller:
      ... (stuff here)
    tasks:
      max-plugin-phase-versions: 1000000
      task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - spark
          - k8s-array
        default-for-task-types:
          container: container
          sidecar: sidecar
          spark: spark
          container_array: k8s-array
    k8s-array:
      logs:
        stackdriver-enabled: true
        gcp-project: lyft-level5
        stackdriver-logresourcename: aws_ec2_instance
^^^^ I added this 
k8s-array
block above, I sort of just took my best guess on declaring this here.
The 
logs
block there is the same block that we have in a second ConfigMap for the existing k8s task plugins (so this ConfigMap should be ok). It looks like:
Copy code
apiVersion: v1
kind: ConfigMap
metadata:
  name: flyte-plugin-config
  namespace: "prod"
data:
  plugins: |-
    plugins:
      enabled-plugins:
        - container
        - sidecar
        - k8s-array
        - spark
      logs:
        stackdriver-enabled: true
        gcp-project: lyft-level5
        stackdriver-logresourcename: aws_ec2_instance
@Dan Rammer (hamersaw) see anything wrong (especially in that first ConfigMap for Propeller)? I took my best guess on how to configure it.
d

Dan Rammer (hamersaw)

02/11/2022, 11:14 PM
I think you are very close. The 
k8s-array
block should be defined in the plugin configuration instead of the propeller configuration. Also, I don't believe the 
enabled-plugins
block in the plugin configuration is necessary.
So your propeller configuration can look something like:
Copy code
apiVersion: v1
kind: ConfigMap
metadata:
  name: flyte-propeller-config
  namespace: "prod"
data:
  propeller: |-
    propeller:
      ... (stuff here)
    tasks:
      max-plugin-phase-versions: 1000000
      task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - spark
          - k8s-array
        default-for-task-types:
          container: container
          sidecar: sidecar
          spark: spark
          container_array: k8s-array
And the plugins configuration should be:
Copy code
apiVersion: v1
kind: ConfigMap
metadata:
  name: flyte-plugin-config
  namespace: "prod"
data:
  plugins: |-
    plugins:
      k8s-array:
        logs:
          stackdriver-enabled: true
          gcp-project: lyft-level5
          stackdriver-logresourcename: aws_ec2_instance
      logs:
        stackdriver-enabled: true
        gcp-project: lyft-level5
        stackdriver-logresourcename: aws_ec2_instance
Basically, the configuration in propeller is because all of the Flyte plugins are currently compiled into the binary (something we're hopefully migrating away from). The configuration allows users to enable / disable certain plugins. Alternatively, the actual plugins configuration enabled fine-tuning of individual plugin behavior.
I think this should fix your problem, please let me know if it doesn't work!
17 Views
Powered by