Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hi all, coming back to Flyte after a couple months working on other things.

Anyone found the FrameProfileRenderer to be really slow? I’m trying to generate statistics for a workflow with &lt;100k rows and its been running for over an hour now. Seems to chew up whatever memory I throw and it but its only using one thread.

Any ideas how I could improve this? Or is there something else I should be doing?

Not familiar as much but it seems to use <https://docs.profiling.ydata.ai/latest/|https://docs.profiling.ydata.ai/latest/>
Under the hood... Maybe the project's page offer some insight?

It seems they have a <https://docs.profiling.ydata.ai/latest/features/big_data/#minimal-mode|minimal mode> but currently no way to configure it for the plugin

Literally just the flag to tell it to remove the really expensive statistics

Either way, for now looks like I can’t use this profiler on any moderately sized dataset.

I see.. would you be willing to open a PR against flytekit? 
You can copy that renderer into your own code, make the change and use it in your type annotation until we upstream the change

Unlikely to have time for it but if it keeps bothering me then I’ll give the minimal mode thing a try

Do you know the exact code/flag that needs to be set? 

```profile = ProfileReport(large_dataset, minimal=True)```

hi <@U05EFL3CVGQ> this absolutely makes sense! would you mind making a github issue here? <https://github.com/flyteorg/flyte/issues/new?assignees=&amp;labels=enhancement%2Cuntriaged&amp;projects=&amp;template=feature_request.yaml&amp;title=%5BCore+feature%5D+|https://github.com/flyteorg/flyte/issues/new?assignees=&amp;labels=enhancement%2Cuntri[…]ojects=&amp;template=feature_request.yaml&amp;title=%5BCore+feature%5D+>

Changes would need to be implemented here:
<https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-deck-standard/flytekitplugins/deck/renderer.py#L61>

<https://github.com/flyteorg/flyte/issues/5993>