https://flyte.org logo
Join Slack
Powered by
I'm trying to do some tests in flyte sandbox, but ...
# flyte-support
q
I'm trying to do some tests in flyte sandbox, but 
pyflyte --config ~/.flyte/config-sandbox.yaml register ...
gives me exceptions: failed to put data... Is this supposed to work out of the box? I can't seem to find where I configure credentials as that seems to be the problem.
Copy code
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmpj_j4utc7/fast451f92eba04c8d8928e00c0180d0e672.tar.gz to <http://localhost:30002/my-s3-bucket/flytesnacks/development/UKZS3GQI43IDNXA4T4SDQUDSHI%3D%3D%3D%3D%3D%3D/fast451f92eba04c8d8928e00c0180d0e672.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230303%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230303T104759Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=2d58a4003c06e3dec4b97659ddd5c402bb3cc7d03b9b17b84979c670e43233bb> (recursive=False).

Original exception: 400, message='Bad Request', url=URL('<http://localhost:30002/my-s3-bucket/flytesnacks/development/UKZS3GQI43IDNXA4T4SDQUDSHI======/fast451f92eba04c8d8928e00c0180d0e672.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio/20230303/us-east-1/s3/aws4_request&X-Amz-Date=20230303T104759Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=2d58a4003c06e3dec4b97659ddd5c402bb3cc7d03b9b17b84979c670e43233bb>')
somehow signed urls are not working?
@hallowed-mouse-14616 @thankful-minister-83577 any pointers here so I can test?
h
what does your 
~/.flyte/config-sandbox.yaml
look like? it should just work out of the box
q
looks good:
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:30080
  authType: Pkce
  insecure: true
console:
  endpoint: <http://localhost:30080>
logger:
  show-source: true
  level: 0
minio is running and I can view the bucket...
h
oh sure, sorry. been a long morning 😅 @thankful-minister-83577 ideas on registering into sandbox issue?
t
um
no, i’ve not seen this before
which version of flytekit are you on?
q
v1.3.2-15-g569b5633 so my containertask-podtemplate branch
t
yeah there should be no reason you’re seeing that.
let me try to repro
q
I already tore down the demo sandbox again and deleted the flyte-sandbox docker volume for good measure, still the same
this was with 
flytectl demo start --image <http://ghcr.io/flyteorg/flyte-sandbox-bundled:sha-e3703333bbef6153f5ec15d3c0c6500009c41bdf|ghcr.io/flyteorg/flyte-sandbox-bundled:sha-e3703333bbef6153f5ec15d3c0c6500009c41bdf>
that you posted
t
can you check out master and try again?
just try
Copy code
$ pyflyte --config ~/.flyte/config-sandbox.yaml register --version fr8 core/flyte_basics/hello_world.py
um, i’m in 
~/go/src/github.com/flyteorg/flytesnacks/cookbook
but you can just copy that file anywhere
q
ah, and I did a 
pip install git+<https://github.com/flyteorg/flyteidl.git@feature/pod-data-loading-config>
to thet the flyteidl pr from @hallowed-mouse-14616 in...
gotta run now, will try it tomorrow or sunday, thanks @thankful-minister-83577
t
okay sure
i’m not able to repro though btw
Copy code
$ pyflyte --config ~/.flyte/config-sandbox.yaml register --version fr8 core/ress.py
{"asctime": "2023-03-03 09:33:22,695", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Running pyflyte register from /Users/ytong/go/src/github.com/flyteorg/flytesnacks/cookbook with images ImageConfig(default_image=Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.10-latest'), images=[Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.10-latest')]) and image destination folder /root on 1 package(s) ('/Users/ytong/go/src/github.com/flyteorg/flytesnacks/cookbook/core/ress.py',)
Registering against localhost:30080
Detected Root /Users/ytong/go/src/github.com/flyteorg/flytesnacks/cookbook, using this to create deployable package...
No output path provided, using a temporary directory at /var/folders/jx/54tww2ls58n8qtlp9k31nbd80000gp/T/tmp6d5w6095 instead
Loading packages ['core.ress'] under source root /Users/ytong/go/src/github.com/flyteorg/flytesnacks/cookbook
Successfully serialized 1 flyte objects
[✔] Registration ellipse-area-metadata-shell type TASK successful with version fr8
Successfully registered 1 entities
q
k, will try with a fresh flytekit then
t
let us knwo
q
ok, so with a fresh flytekit venv that part works now... no idea what went wrong the first time... but it fails in flyte-copilot-downloader with
Copy code
time="2023-03-05T20:26:06Z" level=info msg="[0] Couldn't find a config file []. Relying on env vars and pflags."
{"json":{},"level":"info","msg":"Downloading inputs from [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>]","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"error","msg":"Failed to read from the raw store [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>] Error: getItem, getting the object: RequestError: send request failed\ncaused by: Head <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>: dial tcp [::1]:30002: connect: connection refused","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"error","msg":"Failed to download inputs from [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>], err [path:<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>: getItem, getting the object: RequestError: send request failed\ncaused by: Head <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>: dial tcp [::1]:30002: connect: connection refused]","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"info","msg":"Exited downloading inputs from [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>]","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"error","msg":"Downloading failed, err failed to download input metadata message from remote store: path:<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>: getItem, getting the object: RequestError: send request failed\ncaused by: Head <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/inputs.pb>: dial tcp [::1]:30002: connect: connection refused","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"info","msg":"Uploading Error file to path [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb>], errFile: errors.pb","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"error","msg":"Failed to write to the raw store [<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb>] Error: Failed to write data [493b] to path [metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb].: PutObject, putting object: RequestError: send request failed\ncaused by: Put <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb>: dial tcp [::1]:30002: connect: connection refused","ts":"2023-03-05T20:26:06Z"}
{"json":{},"level":"error","msg":"Failed to write error document, err :Failed to write data [493b] to path [metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb].: PutObject, putting object: RequestError: send request failed\ncaused by: Put <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb>: dial tcp [::1]:30002: connect: connection refused","ts":"2023-03-05T20:26:06Z"}
Error: Failed to write data [493b] to path [metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb].: PutObject, putting object: RequestError: send request failed
caused by: Put <http://localhost:30002/my-s3-bucket/metadata/propeller/flytesnacks-development-ahwn6q5rj8phfxhqfgf6/ellipse-area-metadata-shell/data/3/errors.pb>: dial tcp [::1]:30002: connect: connection refused
seems it tries to connect via IPv6 instead of IPv4, will further try to debug tomorrow..
Shouldn't the flyte-copilot-downloader init container (and the sidecar) be using 
<http://flyte-sandbox-minio.flyte:9000>
instead of 
<http://localhost:30002>
?
t
yeah it should be 
<http://flyte-sandbox-minio.flyte:9000>
that’s what pods use. trying to track that down
f
this has to do with how the sandbox is configured to generate signed URLs that can be used from outside the sandbox container (within the user's machine). investigating.
q
thx, thought so, could not directly find where, so thanks for looking into it
t
we’ll get back to you later today @quaint-diamond-37493
f
@freezing-airport-6809: cc
f
uhh interesting, how did sandbox work with this config?
q
ok, thanks! @thankful-minister-83577 can you build another sandbox image with this fix and the podtemplate backend stuff so I can test it?
f
we’ll clean up and merge soon. that will push a new image.
q
nice, could you also build a new image with the pod template backend PRs included?
f
ah not familiar with that. i’ll defer to @thankful-minister-83577
t
yeah this is the thing i was talking about jeev
building a sandbox-bundled image with all the backend changes in instances where flytekit changes also need backend changes
i’ll build an image this afternoon - dealing with some bugs rn
q
awesome @thankful-minister-83577, then I can pick up the PR testing tomorrow
t
updated https://github.com/flyteorg/flyte/pull/3382 - jobs are queued though
will check back later
seems to have built, but in the middle of something so haven’t tested it
<http://ghcr.io/flyteorg/flyte-sandbox-bundled:sha-ae54328f88f8036cf1531bdea828ee43588aee6e|ghcr.io/flyteorg/flyte-sandbox-bundled:sha-ae54328f88f8036cf1531bdea828ee43588aee6e>
q
thanks, works!
🙌 2
347 Views
Open in Slack
PreviousNext
Powered by