Hi ! I am working on a task that is gonna be run in map task config, each instance will generate fe...

important-hamburger-34837

07/02/2023, 12:40 PM

Hi ! I am working on a task that is gonna be run in map task config, each instance will generate few files and will store them in /tmp/some_path, it returns a flyte directory from that path. however I keep getting this error:

Copy code

Cannot convert from /tmp/some_path to typing.Optional[flytekit.types.directory.types.FlyteDirectory]\n\nSYSTEM ERROR! Contact platform administrators."

please note that sometimes it works . does it have to use context path instead ?

glamorous-carpet-83516

07/02/2023, 11:38 PM

string is fine. it’s wired. could you share the code snippet. happy to investigate at it

freezing-airport-6809

07/03/2023, 5:16 AM

I think this is the optional bug that I fixed recently? Version will be important

important-hamburger-34837

07/03/2023, 9:20 AM

Copy code

def scrape(scraper_input: ScraperMapInput) -> Optional[FlyteDirectory]: 

scraper.scrape() # writes multiple files to a output_path directory   
remote_dir = os.path.join('<gs://datasets>', output_path)
    return FlyteDirectory(path=output_path, remote_directory=remote_dir)

# wf: 
result = map_task(scrape, min_success_ratio=0.25)(scraper_input=scraper_inputs)

important-hamburger-34837

07/03/2023, 9:22 AM

@freezing-airport-6809 flytekit 1.5.0

glamorous-carpet-83516

07/04/2023, 6:38 AM

could you try flytekit 1.7.1 instead

important-hamburger-34837

07/04/2023, 10:27 AM

i used 1.7.0 as its on pip index(latest). didnt work

elegant-toddler-67101

07/04/2023, 12:45 PM

Btw (I work with @important-hamburger-34837) - we use Flyte helm chart on GKE cluster version 1.6.2, I wondered maybe this can be related?

elegant-toddler-67101

07/04/2023, 2:16 PM

We upgraded the chart to 1.7.0 and still getting the error…

elegant-toddler-67101

07/04/2023, 2:17 PM

Do you have any idea why it happens? It is a major bug and it blocks us from continue the work with Flyte

glamorous-carpet-83516

07/05/2023, 12:32 AM

looking

glamorous-carpet-83516

07/05/2023, 1:34 AM

few questions 1. is

/tmp/some_path

a directory 2. did you see any other errors in the pod?

important-hamburger-34837

07/05/2023, 10:37 AM

1. yes 2. turned out its because of gcs account was not permitted to delete objects. error was not showing up until I removed Optional datatype

important-hamburger-34837

07/05/2023, 10:37 AM

Thanks

153 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.