I m trying to do some tests in flyte sandbox but `pyflyte co Flyte #ask-the-community

I'm trying to do some tests in flyte sandbox, but ...

Felix Ruess

03/03/2023, 10:18 AM

I'm trying to do some tests in flyte sandbox, but

pyflyte --config ~/.flyte/config-sandbox.yaml register ...

gives me exceptions: failed to put data... Is this supposed to work out of the box? I can't seem to find where I configure credentials as that seems to be the problem.

Felix Ruess

03/03/2023, 11:02 AM

Copy code

flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmpj_j4utc7/fast451f92eba04c8d8928e00c0180d0e672.tar.gz to <http://localhost:30002/my-s3-bucket/flytesnacks/development/UKZS3GQI43IDNXA4T4SDQUDSHI%3D%3D%3D%3D%3D%3D/fast451f92eba04c8d8928e00c0180d0e672.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230303%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230303T104759Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=2d58a4003c06e3dec4b97659ddd5c402bb3cc7d03b9b17b84979c670e43233bb> (recursive=False).

Original exception: 400, message='Bad Request', url=URL('<http://localhost:30002/my-s3-bucket/flytesnacks/development/UKZS3GQI43IDNXA4T4SDQUDSHI======/fast451f92eba04c8d8928e00c0180d0e672.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio/20230303/us-east-1/s3/aws4_request&X-Amz-Date=20230303T104759Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=2d58a4003c06e3dec4b97659ddd5c402bb3cc7d03b9b17b84979c670e43233bb>')

somehow signed urls are not working?

Felix Ruess

03/03/2023, 2:43 PM

@Dan Rammer (hamersaw) @Yee any pointers here so I can test?

Dan Rammer (hamersaw)

03/03/2023, 5:07 PM

what does your

~/.flyte/config-sandbox.yaml

look like? it should just work out of the box

Felix Ruess

03/03/2023, 5:12 PM

looks good:

Copy code

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:30080
  authType: Pkce
  insecure: true
console:
  endpoint: <http://localhost:30080>
logger:
  show-source: true
  level: 0

Felix Ruess

03/03/2023, 5:12 PM

minio is running and I can view the bucket...

Dan Rammer (hamersaw)

03/03/2023, 5:14 PM

oh sure, sorry. been a long morning 😅 @Yee ideas on registering into sandbox issue?

Yee

03/03/2023, 5:28 PM

Yee

03/03/2023, 5:28 PM

no, i’ve not seen this before

Yee

03/03/2023, 5:28 PM

which version of flytekit are you on?

Felix Ruess

03/03/2023, 5:29 PM

v1.3.2-15-g569b5633 so my containertask-podtemplate branch

Yee

03/03/2023, 5:30 PM

yeah there should be no reason you’re seeing that.

Yee

03/03/2023, 5:30 PM

let me try to repro

Felix Ruess

03/03/2023, 5:30 PM

I already tore down the demo sandbox again and deleted the flyte-sandbox docker volume for good measure, still the same

Felix Ruess

03/03/2023, 5:32 PM

this was with

flytectl demo start --image <http://ghcr.io/flyteorg/flyte-sandbox-bundled:sha-e3703333bbef6153f5ec15d3c0c6500009c41bdf|ghcr.io/flyteorg/flyte-sandbox-bundled:sha-e3703333bbef6153f5ec15d3c0c6500009c41bdf>

that you posted

Yee

03/03/2023, 5:37 PM

can you check out master and try again?

Yee

03/03/2023, 5:37 PM

just try

Copy code

$ pyflyte --config ~/.flyte/config-sandbox.yaml register --version fr8 core/flyte_basics/hello_world.py

Yee

03/03/2023, 5:38 PM

um, i’m in

~/go/src/github.com/flyteorg/flytesnacks/cookbook

but you can just copy that file anywhere

Felix Ruess

03/03/2023, 5:40 PM

ah, and I did a

pip install git+<https://github.com/flyteorg/flyteidl.git@feature/pod-data-loading-config>

to thet the flyteidl pr from @Dan Rammer (hamersaw) in...

Felix Ruess

03/03/2023, 5:40 PM

gotta run now, will try it tomorrow or sunday, thanks @Yee

Yee

03/03/2023, 5:40 PM

okay sure

Yee

03/03/2023, 5:40 PM

i’m not able to repro though btw

Yee

03/03/2023, 5:40 PM

Copy code

$ pyflyte --config ~/.flyte/config-sandbox.yaml register --version fr8 core/ress.py
{"asctime": "2023-03-03 09:33:22,695", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Running pyflyte register from /Users/ytong/go/src/github.com/flyteorg/flytesnacks/cookbook with images ImageConfig(default_image=Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.10-latest'), images=[Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.10-latest')]) and image destination folder /root on 1 package(s) ('/Users/ytong/go/src/github.com/flyteorg/flytesnacks/cookbook/core/ress.py',)
Registering against localhost:30080
Detected Root /Users/ytong/go/src/github.com/flyteorg/flytesnacks/cookbook, using this to create deployable package...
No output path provided, using a temporary directory at /var/folders/jx/54tww2ls58n8qtlp9k31nbd80000gp/T/tmp6d5w6095 instead
Loading packages ['core.ress'] under source root /Users/ytong/go/src/github.com/flyteorg/flytesnacks/cookbook
Successfully serialized 1 flyte objects
[✔] Registration ellipse-area-metadata-shell type TASK successful with version fr8
Successfully registered 1 entities

Felix Ruess

03/03/2023, 5:41 PM

k, will try with a fresh flytekit then

Yee

03/03/2023, 5:41 PM

let us knwo

Felix Ruess

03/05/2023, 8:42 PM

ok, so with a fresh flytekit venv that part works now... no idea what went wrong the first time... but it fails in flyte-copilot-downloader with

Copy code

time="2023-03-05T20:26:06Z" level=info msg="[0] Couldn't find a config file []. Relying on env vars and pflags."
{"json":{},"level":"info","msg":"Downloading inputs from [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>]","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"error","msg":"Failed to read from the raw store [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>] Error: getItem, getting the object: RequestError: send request failed\ncaused by: Head <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>: dial tcp [::1]:30002: connect: connection refused","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"error","msg":"Failed to download inputs from [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>], err [path:<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>: getItem, getting the object: RequestError: send request failed\ncaused by: Head <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>: dial tcp [::1]:30002: connect: connection refused]","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"info","msg":"Exited downloading inputs from [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>]","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"error","msg":"Downloading failed, err failed to download input metadata message from remote store: path:<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>: getItem, getting the object: RequestError: send request failed\ncaused by: Head <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>: dial tcp [::1]:30002: connect: connection refused","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"info","msg":"Uploading Error file to path [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb>], errFile: errors.pb","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"error","msg":"Failed to write to the raw store [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb>] Error: Failed to write data [493b] to path [metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb].: PutObject, putting object: RequestError: send request failed\ncaused by: Put <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb>: dial tcp [::1]:30002: connect: connection refused","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"error","msg":"Failed to write error document, err :Failed to write data [493b] to path [metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb].: PutObject, putting object: RequestError: send request failed\ncaused by: Put <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb>: dial tcp [::1]:30002: connect: connection refused","ts":"2023-03-05T20:26:06Z"}
Error: Failed to write data [493b] to path [metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb].: PutObject, putting object: RequestError: send request failed
caused by: Put <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb>: dial tcp [::1]:30002: connect: connection refused

seems it tries to connect via IPv6 instead of IPv4, will further try to debug tomorrow..

Felix Ruess

03/06/2023, 10:58 AM

Shouldn't the flyte-copilot-downloader init container (and the sidecar) be using

<http://flyte-sandbox-minio.flyte:9000>

instead of

<http://localhost:30002>

Yee

03/06/2023, 5:08 PM

yeah it should be

<http://flyte-sandbox-minio.flyte:9000>

Yee

03/06/2023, 5:08 PM

that’s what pods use. trying to track that down

jeev

03/06/2023, 5:38 PM

this has to do with how the sandbox is configured to generate signed URLs that can be used from outside the sandbox container (within the user's machine). investigating.

Felix Ruess

03/06/2023, 5:40 PM

thx, thought so, could not directly find where, so thanks for looking into it

Yee

03/06/2023, 6:12 PM

we’ll get back to you later today @Felix Ruess

jeev

03/07/2023, 3:24 AM

this PR fixes: https://github.com/flyteorg/flyte/pull/3410

jeev

03/07/2023, 5:17 AM

@Ketan (kumare3): cc

Ketan (kumare3)

03/07/2023, 5:20 AM

uhh interesting, how did sandbox work with this config?

Felix Ruess

03/07/2023, 8:20 AM

ok, thanks! @Yee can you build another sandbox image with this fix and the podtemplate backend stuff so I can test it?

jeev

03/07/2023, 1:06 PM

we’ll clean up and merge soon. that will push a new image.

Felix Ruess

03/07/2023, 1:16 PM

nice, could you also build a new image with the pod template backend PRs included?

jeev

03/07/2023, 1:24 PM

ah not familiar with that. i’ll defer to @Yee

Yee

03/07/2023, 6:01 PM

yeah this is the thing i was talking about jeev

Yee

03/07/2023, 6:02 PM

building a sandbox-bundled image with all the backend changes in instances where flytekit changes also need backend changes

Yee

03/07/2023, 6:02 PM

i’ll build an image this afternoon - dealing with some bugs rn

Felix Ruess

03/07/2023, 6:23 PM

awesome @Yee, then I can pick up the PR testing tomorrow

Yee

03/08/2023, 12:21 AM

updated https://github.com/flyteorg/flyte/pull/3382 - jobs are queued though

Yee

03/08/2023, 12:22 AM

will check back later

Yee

03/08/2023, 2:19 AM

seems to have built, but in the middle of something so haven’t tested it

Yee

03/08/2023, 2:19 AM

<http://ghcr.io/flyteorg/flyte-sandbox-bundled:sha-ae54328f88f8036cf1531bdea828ee43588aee6e|ghcr.io/flyteorg/flyte-sandbox-bundled:sha-ae54328f88f8036cf1531bdea828ee43588aee6e>

Felix Ruess

03/09/2023, 7:00 AM

thanks, works!

278 Views

Open in Slack

Previous Next