https://flyte.org logo
j

James Evers

08/24/2022, 9:25 PM
hi everyone, i'm quite new to flyte and am having some trouble running the feast integration example in the sandbox. i'm not able to get much out of the error, but here it is if anyone could help to steer me in the right direction:
Copy code
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f167c07ba12604be28d7-n0-0] terminated with exit code (1). Reason [Error]. Message: 
tar: Removing leading `/' from member names

Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 502, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f167c07ba12604be28d7/n0/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f167c07ba12604be28d7/n0/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/test/n1/f167c07ba12604be28d7-n0-0>', '--checkpoint-path', '<s3://my-s3-bucket/test/n1/f167c07ba12604be28d7-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/ht/flytesnacks/development/XB5EEQT46XRRFMME7UC7ROF2JA======/scriptmode.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'feast_integration.feast_workflow', 'task-name', 'dummy_thing']' died with <Signals.SIGKILL: 9>.
.
👀 1
k

Ketan (kumare3)

08/24/2022, 11:19 PM
Seems the tar download failed. What version of flytekit is this
Cc @Yee
s

Samhita Alla

08/25/2022, 7:34 AM
It isn’t related to tar file. tar is just a warning msg. I guess you’ll have to increase the task resources, like
@task(requests=Resources(mem="500Mi", cpu="2"))
. Would you mind letting me know what this task encompasses?
k

Ketan (kumare3)

08/25/2022, 2:26 PM
Ohh that’s a great catch - Samhita. Hmm we should improve the error visibility
☝️ 1
n

Niels Bantilan

08/25/2022, 2:27 PM
Hmm we should improve the error visibility
I feel like a quick win here would be to use
rich
to pretty print tracebacks: https://rich.readthedocs.io/en/stable/traceback.html
👍 1
j

James Evers

08/25/2022, 2:28 PM
thanks for checking this out folks, the task i ran was just a test:
Copy code
@task
def dummy(bucket_name: str) -> str:
    <http://logger.info|logger.info>("test!")
    return bucket_name
Copy code
flytekit==0.32.3
@Ketan (kumare3)
n

Niels Bantilan

08/25/2022, 2:29 PM
@James Evers can you try using flytekit
1.1.*
?
j

James Evers

08/25/2022, 2:30 PM
sure thing
n

Niels Bantilan

08/25/2022, 2:35 PM
for the cookbook tutorial i was running need an update
yes we use
pip-compile
to pin all package versions, we currently don’t have an automatic way of doing this, we can look into that
j

James Evers

08/25/2022, 2:35 PM
i could help out as well
❤️ 2
4 Views