Mathias Andersen
04/04/2023, 7:20 AMstorage:
type: custom
enableMultiContainer: true
limits:
maxDownloadMBs: 500000
type: custom
bucketName: "{{ .Values.userSettings.azure.containerName }}"
custom:
container: "{{ .Values.userSettings.azure.containerName }}"
enable-multicontainer: true
connection: {}
type: stow
stow:
kind: azure
config:
account: "{{ .Values.userSettings.azure.storageAccountName }}"
key: "{{ .Values.userSettings.azure.storageAccountKey }}"
Trying to run the init workflow examples throws:
Traceback (most recent call last): │
│ File "/opt/venv/bin/pyflyte-execute", line 8, in <module> │
│ sys.exit(execute_task_cmd()) │
│ File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__ │
│ return self.main(*args, **kwargs) │
│ File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main ││ rv = self.invoke(ctx) ││ File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke ││ return ctx.invoke(self.callback, **ctx.params) ││ File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke ││ return __callback(*args, **kwargs) ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 470, in execute_task_cmd ││ _execute_task( ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point ││ return wrapped(*args, **kwargs) ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task ││ _handle_annotated_task(ctx, _task_def, inputs, output_prefix) ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 291, in _handle_annotated_task ││ _dispatch_execute(ctx, task_def, inputs, output_prefix) ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 160, in _dispatch_execute ││ ctx.file_access.put_data(ctx.execution_state.engine_dir, output_prefix, is_multipart=True) ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 476, in put_data ││ raise FlyteAssertion( ││ flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyte-z58pqpy5/sandbox/local_flytekit/engine_dir to <abfs://flyte-workflows/metadata/propeller/>...
Original exception: No plugin found for matching protocol of path <abfs://flyte-workflows/metadata/propeller/>...
Are there specific steps to enable/support the abfs protocol, beyond configuring stow as above?Nick Müller (MorpheusXAUT)
04/04/2023, 9:14 AMtype: custom
in the limits
blocks that should not be required, I believe.
We're using a fork of adlfs in our flyte deployments since some FlyteDirectory
handling was broken with the upstream version (our PR has finally been merged, but no new release was created yet, so we still couldn't contribute it back to upstream flytekit), however that should not be the issue you're seeing right here.Mathias Andersen
04/04/2023, 9:27 AMNick Müller (MorpheusXAUT)
04/04/2023, 9:30 AMmaster
branch state at the time of forkingMathias Andersen
04/04/2023, 9:36 AMNick Müller (MorpheusXAUT)
04/04/2023, 9:37 AMMathias Andersen
04/04/2023, 12:57 PMpyflyte run --remote workflows/example.py wf --name 'hello'
File "/home/user/.local/share/virtualenvs/first-attempt-to-flyte-Ae80d762/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 447, in put_data
raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmprmqf8xzd/script_mode.tar.gz to <https://my.blob.core.windows.net/flyte-workflows/flytesnacks/development/FOVDCPZCJ447LTKJTX5HCLBKPA======/scriptmode.tar.gz?se=2023-04-04T13%3A50%3A53Z&sig=V7%2B9c7Y5e4Oc0sf2XbxVXOtWssJIdCmxst4wtWLzzyI%3D&sp=aw&spr=https&sr=b&sv=2019-12-12> (recursive=False).
Original exception: Value error! Received: 400. Request to send data failed.
Ketan (kumare3)
Nick Müller (MorpheusXAUT)
04/04/2023, 1:47 PMKetan (kumare3)
Nick Müller (MorpheusXAUT)
04/04/2023, 1:49 PMFlyteDirectory
didn't work properly, so contributing all changes upstream would've not been very usefulKetan (kumare3)
Nick Müller (MorpheusXAUT)
04/04/2023, 1:51 PMMathias Andersen
04/04/2023, 2:28 PMOriginal exception: No plugin found for matching protocol of path abfs://...awqq2pc7mkjlnmql7vcf/n0/data/0
Which I would interpret as a failure to write data, but this path contains an inputs.pb file with expected content!Ketan (kumare3)
Mathias Andersen
04/04/2023, 2:53 PMNick Müller (MorpheusXAUT)
04/04/2023, 2:53 PMMathias Andersen
04/04/2023, 2:58 PMOriginal exception: unable to connect to account for Must provide either a connection_string or account_name with credentials!!
I guess I need to review the fsspec config
Files were still produced though.Ketan (kumare3)
Nick Müller (MorpheusXAUT)
04/05/2023, 1:13 PMKetan (kumare3)
Nick Müller (MorpheusXAUT)
04/05/2023, 1:30 PMMathias Andersen
04/11/2023, 11:50 AMpyflyte register workflows
and
pyflyte run --remote workflows/example.py wf --name myname
throws my original error: "Value error! Received: 400. Request to send data https://... Failed to put data from /tmp/tmp52zl9lno/script_mode.tar.gz to https://..."
I guess pyflyte cli does not use the data persistence layer from flytekitplugins-data-fsspec (or 1.5, I just updated)? @Ketan (kumare3)
Update:
pyflyte register workflows --non-fast
works. No idea why fast-register breaks, but it explains why run --remote
fails as well.No plugin found for matching protocol of path abfs://...
Jeongwon Song
04/12/2023, 1:53 PMpyflyte run --remote
I am on flyte 1.4.3 and flytekit 1.4.2. I am not sure if this is because the signed url collides with blob url dns.Mathias Andersen
04/12/2023, 2:10 PMKetan (kumare3)
Mathias Andersen
04/12/2023, 2:15 PMYee
abfs
not adlfs
(flytekit) ytong@Yees-MBP:~/go/src/github.com/flyteorg/flytekit [flyte-sandbox] (master) $ ipython
Python 3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.11.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from flytekit.core.context_manager import FlyteContextManager
In [2]: ctx = FlyteContextManager.current_context()
In [3]: ctx.file_access.get_filesystem("abfs")
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File ~/envs/flytekit/lib/python3.10/site-packages/adlfs/spec.py:447, in AzureBlobFileSystem.do_connect(self)
446 else:
--> 447 raise ValueError(
448 "Must provide either a connection_string or account_name with credentials!!"
449 )
451 except RuntimeError:
Mathias Andersen
04/18/2023, 12:59 PM