https://flyte.org logo
Join Slack
Powered by
is it still possible to set the offloaded data sto...
# flytekit
f
is it still possible to set the offloaded data storage via environment variable and/or config or is this now only passed at registration time?
f
During registration or execution time. The launch form for example has advanced options
Please create an issue if you would like config
f
either. need a workaround for an issue where the feature is not exposed by CI/CD
so i guess what are my options for specifying outside of passing via 
-o
flag in registration script
f
What do you mean not exposed by ci cd?
f
our internal ci/cd is an api and they did not expose a way to set this value
f
ohh
f
as part of the registration step. We do have access to toggle the environment variables
f
should i talk to Guillaume about this?
f
no i'm already on it, just need a workaround for now
they're less familiar with this functionality since it's not heavily used eslewhere within spotify
f
hmm ok
but you can pass env vars?
f
yep 🙂
either in the docker container or from the api
this used to be how we set it, but i thought it was removed in favor of the cmd line arg
f
ya, so I think we can pass env vars for all flytectl arguments, cc @high-park-82026 is trying it out
btw, @famous-businessperson-24711 we were thinking of setting the raw-output-prefix at the project+domain level as a setting
you think that would help?
f
not in this current instance, but it's not a bad feature
h
What’s the flag you do want to pass to flytectl? 
--outputLocationPrefix
? You should be able to set:
Copy code
OUTPUTLOCATIONPREFIX=s3://<something> flytectl register...
Is what what you are looking for?
f
yes i think so!
let me try it
looks like it didn't work. will dig in later today
so i don't see this value anywhere in config
but what's this used for
@high-park-82026?
f
@famous-businessperson-24711 so the raw output data prefix exists, but these values are legacy and not used anymore
cc @thankful-minister-83577
f
As of one version? We have some projects that appear to work
t
one version?
so just looked, that setting is still used, but only because when i cleaned up the code, i didn’t delete all of 
flyte-cli
the 
RAW_OUTPUT_DATA_PREFIX
object is still used for 
flyte-cli register-files
which you’re correct has been subsumed by 
flytectl
but it’s still there in flyte-cli as well.
what’s your command rn dylan?
f
looks like flyte-cli. interestingly the env var override did not work though, trying the config file...
t
what env var?
f
FLYTE_AUTH_RAW_OUTPUT_DATA_PREFIX
t
and that isn’t working? can you copy paste the full 
flyte-cli
command?
f
flyte-cli register-files -p ${PROJECT} -d ${DOMAIN} -v ${IMAGE_TAG} ${INSECURE_OPTION} /tmp/_pb_output/*"
h
Yeah… I realized 
OUTPUTLOCATIONPREFIX
doesn’t work… flytectl has a bug where it doesn’t bind these envvars… 😞 I’ll see if this can be fixed easily…
t
sorry, distracted.
can you specify the flag? i was mistaken (obviously, the whole point of putting it into flyte-cli was so that it didn’t rely on the environment variable)
--output-location-prefix
f
Yea issue is that this isn’t exposed by our Ci/cd atm so looking for a workaround.
Side note can this be set at runtime in flytectl?
t
can talk a bit more after the oss sync.
but qq, i don’t think i understand.
you can make ci/cd run the 
flytectl
or 
flyte-cli
command… but you can’t pass an extra switch to it?
was this resolved btw dylan?
or still an issue?
166 Views
Open in Slack
PreviousNext
Powered by