https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
d

Daniel Xenes

04/12/2023, 3:09 PM
Hi everyone, trying to set up a simple workflow using flytesnacks. I recently built a new image and using it to run the individual tasks but now I’m getting this error when the workflow starts.
Copy code
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 497, in fast_execute_task_cmd
    _download_distribution(additional_distribution, dest_dir)
  File "/usr/local/lib/python3.9/site-packages/flytekit/tools/fast_registration.py", line 111, in download_distribution
    FlyteContextManager.current_context().file_access.get_data(additional_distribution, os.path.join(destination, ""))
  File "/usr/local/lib/python3.9/site-packages/flytekit/core/data_persistence.py", line 303, in get_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://my-s3-bucket/flytesnacks/development/TLGPVQIJ7UHT2DCEZO3ZBNK77U======/fast0a28e459bf705dea2398e8d5270d09d1.tar.gz> to /root/ (recursive=False).
Has anyone encountered something like this before? Seems like either the container or the workflow can’t access the tar
This is the dockerfile I’m using
Copy code
FROM <http://ghcr.io/flyteorg/flytekit:py3.9-latest|ghcr.io/flyteorg/flytekit:py3.9-latest>

# Install Python dependencies
USER root
RUN pip install --no-cache-dir cloud-volume scikit-image
USER flytekit
k

Ketan (kumare3)

04/12/2023, 3:11 PM
Hmm I guess need more info. How did you launch it
Seems the fast patched zip Is not available
d

Daniel Xenes

04/12/2023, 3:13 PM
Hi Ketan, So after building the image (em-membrane-image) and pushing it to the local repository included with the flyte demo, I added the container image to the task decorator on my tasks
Copy code
@task(container_image="localhost:30000/em-membrane-image:latest",
      cache=True, cache_version="1.0", limits=Resources(mem="4Gi"))
I then registered the workflow
Copy code
pyflyte register -p flytesnacks sobel_membrane.py
And then executed the workflow through the GUI. I get this error in the first step
Let me also include the full error. Seems like the traceback is actually being cause by an error with s3fs, which is making me think that maybe the minio access is not working properly:
Copy code
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-yfc_gvjn because the default path (/home/flytekit/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
/usr/local/lib/python3.9/site-packages/botocore/utils.py:1720: FutureWarning: The S3RegionRedirector class has been deprecated for a new internal replacement. A future version of botocore may remove this class.
  warnings.warn(
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 112, in _error_wrapper
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/client.py", line 358, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flytekit/core/data_persistence.py", line 301, in get_data
    self.get(remote_path, to_path=local_path, recursive=is_multipart)
  File "/usr/local/lib/python3.9/site-packages/flytekit/core/data_persistence.py", line 205, in get
    return file_system.get(from_path, to_path, recursive=recursive)
  File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 115, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 100, in sync
    raise return_result
  File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 55, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 562, in _get
    return await _run_coros_in_chunks(
  File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 246, in _run_coros_in_chunks
    await asyncio.gather(*chunk, return_exceptions=return_exceptions),
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
  File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 1171, in _get_file
    body, content_length = await _open_file(range=0)
  File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 1162, in _open_file
    resp = await self._call_s3(
  File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 347, in _call_s3
    return await _error_wrapper(
  File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 139, in _error_wrapper
    raise err
PermissionError: Access Denied.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 497, in fast_execute_task_cmd
    _download_distribution(additional_distribution, dest_dir)
  File "/usr/local/lib/python3.9/site-packages/flytekit/tools/fast_registration.py", line 111, in download_distribution
    FlyteContextManager.current_context().file_access.get_data(additional_distribution, os.path.join(destination, ""))
  File "/usr/local/lib/python3.9/site-packages/flytekit/core/data_persistence.py", line 303, in get_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://my-s3-bucket/flytesnacks/development/TLGPVQIJ7UHT2DCEZO3ZBNK77U======/fast0a28e459bf705dea2398e8d5270d09d1.tar.gz> to /root/ (recursive=False).
Original exception: Access Denied.
k

Ketan (kumare3)

04/13/2023, 4:21 AM
Access Denied?
i think this could be due to - missing storage config?
this is minio right?
d

Daniel Xenes

04/13/2023, 6:09 PM
yep minio, I think you’re on to something though. This is my config-sandbox.yaml in ~/.flyte/
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:30080
  authType: Pkce
  insecure: true
console:
  endpoint: <http://localhost:30080>
logger:
  show-source: true
  level: 0
I notice in this one though there is storage block for minio
adding that didn’t seem to fix it though, even when I changed the endpoint to localhost:30002
s

Samhita Alla

04/14/2023, 9:31 AM
Can you not set USER in your dockerfile? I think it should work out of the box.
It isn't working for me either. I see the same error in demo cluster.
k

Ketan (kumare3)

04/14/2023, 1:27 PM
You should not need the storage block @Daniel Xenes. Sorry for the delay
@Samhita Alla what’s not working - the storage api? Cc @Yee
s

Samhita Alla

04/14/2023, 1:27 PM
FlyteDirectory. FlyteFile's working fine.
When I'm trying to download FlyteDirectory, it's giving me an access denied error.
Copy code
import os

import flytekit
from flytekit import task, workflow
from flytekit.types.directory import FlyteDirectory


@task
def t1() -> FlyteDirectory:
    working_dir = flytekit.current_context().working_directory
    dir = os.path.join(working_dir, "flyte_directory")
    os.makedirs(dir, exist_ok=True)
    return FlyteDirectory(dir)


@task
def t2(ff: FlyteDirectory) -> str:
    return ff.download()


@workflow
def wf() -> str:
    return t2(ff=t1())
k

Ketan (kumare3)

04/14/2023, 1:49 PM
Aah interesting, can you downgrade flytekit and try
I think this could be a bug in fsspec
s

Samhita Alla

04/14/2023, 1:52 PM
The error's appearing because the directory is empty. Not sure why the exception is "access denied" though. Able to resolve it after I added a file to the directory.
image.png
k

Ketan (kumare3)

04/14/2023, 1:54 PM
Aah that’s what fsspec does
I think we will have to improve the error
7 Views
Powered by