Hi team I want to use map tasks to process the data from a S Flyte #flyte-support

Hi team I want to use map_tasks to process the dat...

little-cricket-84530

01/10/2023, 7:50 PM

Hi team I want to use map_tasks to process the data from a StructuredDataset.. It’s a large dataset (parquet in s3) so i’ll need to use

iter()

to access all of it and would like each map task to process 1 chunk. Can map task take a generator instead of a list as the input?

little-cricket-84530

01/10/2023, 7:57 PM

There was a related discussion here

thankful-minister-83577

01/10/2023, 9:00 PM

@broad-monitor-993 @glamorous-carpet-83516 let’s chat about this?

👍 1

broad-monitor-993

01/10/2023, 9:02 PM

let’s!

thankful-minister-83577

01/10/2023, 9:02 PM

free in 30?

👍 1

broad-monitor-993

01/10/2023, 9:12 PM

yep

little-cricket-84530

01/10/2023, 9:16 PM

So am assuming you will let me know what you decide? 🙂

thankful-minister-83577

01/10/2023, 9:17 PM

oh you can join too… we’ll let you know

thankful-minister-83577

01/10/2023, 9:17 PM

but it might be boring for you

little-cricket-84530

01/10/2023, 9:17 PM

I can join.. in case you need details of my use case

thankful-minister-83577

01/10/2023, 9:17 PM

i’ll post the link here when it happens

thankful-minister-83577

01/10/2023, 9:17 PM

yup sounds good

little-cricket-84530

01/10/2023, 9:17 PM

Thanks

freezing-airport-6809

01/10/2023, 10:25 PM

can i join?

thankful-minister-83577

01/10/2023, 10:25 PM

we’re already done

thankful-minister-83577

01/10/2023, 10:25 PM

had two meetings

freezing-airport-6809

01/10/2023, 10:25 PM

haha

thankful-minister-83577

01/10/2023, 10:25 PM

@broad-monitor-993 is writing up the issue

broad-monitor-993

01/10/2023, 10:29 PM

https://github.com/flyteorg/flyte/issues/3226

👍🏼 1

👍 1

154 Views

Open in Slack

Previous Next