millions-queen-16335
04/04/2023, 7:20 AMstorage:
type: custom
enableMultiContainer: true
limits:
maxDownloadMBs: 500000
type: custom
bucketName: "{{ .Values.userSettings.azure.containerName }}"
custom:
container: "{{ .Values.userSettings.azure.containerName }}"
enable-multicontainer: true
connection: {}
type: stow
stow:
kind: azure
config:
account: "{{ .Values.userSettings.azure.storageAccountName }}"
key: "{{ .Values.userSettings.azure.storageAccountKey }}"
Trying to run the init workflow examples throws:
Traceback (most recent call last): │
│ File "/opt/venv/bin/pyflyte-execute", line 8, in <module> │
│ sys.exit(execute_task_cmd()) │
│ File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__ │
│ return self.main(*args, **kwargs) │
│ File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main ││ rv = self.invoke(ctx) ││ File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke ││ return ctx.invoke(self.callback, **ctx.params) ││ File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke ││ return __callback(*args, **kwargs) ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 470, in execute_task_cmd ││ _execute_task( ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point ││ return wrapped(*args, **kwargs) ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task ││ _handle_annotated_task(ctx, _task_def, inputs, output_prefix) ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 291, in _handle_annotated_task ││ _dispatch_execute(ctx, task_def, inputs, output_prefix) ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 160, in _dispatch_execute ││ ctx.file_access.put_data(ctx.execution_state.engine_dir, output_prefix, is_multipart=True) ││ File "/opt/venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 476, in put_data ││ raise FlyteAssertion( ││ flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyte-z58pqpy5/sandbox/local_flytekit/engine_dir to <abfs://flyte-workflows/metadata/propeller/>...
Original exception: No plugin found for matching protocol of path <abfs://flyte-workflows/metadata/propeller/>...
Are there specific steps to enable/support the abfs protocol, beyond configuring stow as above?millions-queen-16335
04/04/2023, 7:22 AMincalculable-zebra-90365
04/04/2023, 9:14 AMtype: custom
in the limits
blocks that should not be required, I believe.
We're using a fork of adlfs in our flyte deployments since some FlyteDirectory
handling was broken with the upstream version (our PR has finally been merged, but no new release was created yet, so we still couldn't contribute it back to upstream flytekit), however that should not be the issue you're seeing right here.incalculable-zebra-90365
04/04/2023, 9:16 AMincalculable-zebra-90365
04/04/2023, 9:17 AMincalculable-zebra-90365
04/04/2023, 9:17 AMmillions-queen-16335
04/04/2023, 9:27 AMincalculable-zebra-90365
04/04/2023, 9:30 AMincalculable-zebra-90365
04/04/2023, 9:31 AMmaster
branch state at the time of forkingincalculable-zebra-90365
04/04/2023, 9:32 AMincalculable-zebra-90365
04/04/2023, 9:33 AMmillions-queen-16335
04/04/2023, 9:36 AMmillions-queen-16335
04/04/2023, 9:37 AMincalculable-zebra-90365
04/04/2023, 9:37 AMmillions-queen-16335
04/04/2023, 12:57 PMpyflyte run --remote workflows/example.py wf --name 'hello'
File "/home/user/.local/share/virtualenvs/first-attempt-to-flyte-Ae80d762/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 447, in put_data
raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmprmqf8xzd/script_mode.tar.gz to <https://my.blob.core.windows.net/flyte-workflows/flytesnacks/development/FOVDCPZCJ447LTKJTX5HCLBKPA======/scriptmode.tar.gz?se=2023-04-04T13%3A50%3A53Z&sig=V7%2B9c7Y5e4Oc0sf2XbxVXOtWssJIdCmxst4wtWLzzyI%3D&sp=aw&spr=https&sr=b&sv=2019-12-12> (recursive=False).
Original exception: Value error! Received: 400. Request to send data failed.
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
incalculable-zebra-90365
04/04/2023, 1:47 PMfreezing-airport-6809
freezing-airport-6809
incalculable-zebra-90365
04/04/2023, 1:49 PMFlyteDirectory
didn't work properly, so contributing all changes upstream would've not been very usefulfreezing-airport-6809
incalculable-zebra-90365
04/04/2023, 1:51 PMincalculable-zebra-90365
04/04/2023, 1:53 PMmillions-queen-16335
04/04/2023, 2:28 PMmillions-queen-16335
04/04/2023, 2:29 PMmillions-queen-16335
04/04/2023, 2:36 PMmillions-queen-16335
04/04/2023, 2:49 PMOriginal exception: No plugin found for matching protocol of path abfs://...awqq2pc7mkjlnmql7vcf/n0/data/0
Which I would interpret as a failure to write data, but this path contains an inputs.pb file with expected content!freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
millions-queen-16335
04/04/2023, 2:53 PMincalculable-zebra-90365
04/04/2023, 2:53 PMincalculable-zebra-90365
04/04/2023, 2:54 PMincalculable-zebra-90365
04/04/2023, 2:55 PMmillions-queen-16335
04/04/2023, 2:58 PMmillions-queen-16335
04/04/2023, 3:06 PMOriginal exception: unable to connect to account for Must provide either a connection_string or account_name with credentials!!
I guess I need to review the fsspec config
Files were still produced though.millions-queen-16335
04/04/2023, 4:08 PMmillions-queen-16335
04/05/2023, 9:42 AMfreezing-airport-6809
incalculable-zebra-90365
04/05/2023, 1:13 PMincalculable-zebra-90365
04/05/2023, 1:16 PMfreezing-airport-6809
incalculable-zebra-90365
04/05/2023, 1:30 PMincalculable-zebra-90365
04/05/2023, 1:31 PMmillions-queen-16335
04/11/2023, 11:50 AMmillions-queen-16335
04/11/2023, 12:04 PMpyflyte register workflows
and
pyflyte run --remote workflows/example.py wf --name myname
throws my original error: "Value error! Received: 400. Request to send data https://... Failed to put data from /tmp/tmp52zl9lno/script_mode.tar.gz to https://..."
I guess pyflyte cli does not use the data persistence layer from flytekitplugins-data-fsspec (or 1.5, I just updated)? @freezing-airport-6809
Update:
pyflyte register workflows --non-fast
works. No idea why fast-register breaks, but it explains why run --remote
fails as well.millions-queen-16335
04/12/2023, 8:25 AMNo plugin found for matching protocol of path abfs://...
billions-mechanic-13717
04/12/2023, 1:53 PMpyflyte run --remote
I am on flyte 1.4.3 and flytekit 1.4.2. I am not sure if this is because the signed url collides with blob url dns.billions-mechanic-13717
04/12/2023, 2:10 PMmillions-queen-16335
04/12/2023, 2:10 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
millions-queen-16335
04/12/2023, 2:15 PMmillions-queen-16335
04/12/2023, 2:21 PMthankful-minister-83577
thankful-minister-83577
abfs
not adlfs
thankful-minister-83577
thankful-minister-83577
(flytekit) ytong@Yees-MBP:~/go/src/github.com/flyteorg/flytekit [flyte-sandbox] (master) $ ipython
Python 3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.11.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from flytekit.core.context_manager import FlyteContextManager
In [2]: ctx = FlyteContextManager.current_context()
In [3]: ctx.file_access.get_filesystem("abfs")
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File ~/envs/flytekit/lib/python3.10/site-packages/adlfs/spec.py:447, in AzureBlobFileSystem.do_connect(self)
446 else:
--> 447 raise ValueError(
448 "Must provide either a connection_string or account_name with credentials!!"
449 )
451 except RuntimeError:
millions-queen-16335
04/18/2023, 12:59 PM