https://flyte.org
Join Slack
is it still possible to set the offloaded data storage via environment variable and/or config or is ...
# flytekit
f
is it still possible to set the offloaded data storage via environment variable and/or config or is this now only passed at registration time?
f
During registration or execution time. The launch form for example has advanced options
Please create an issue if you would like config
f
either. need a workaround for an issue where the feature is not exposed by CI/CD
so i guess what are my options for specifying outside of passing via 
-o
flag in registration script
f
What do you mean not exposed by ci cd?
f
our internal ci/cd is an api and they did not expose a way to set this value
f
ohh
f
as part of the registration step. We do have access to toggle the environment variables
f
should i talk to Guillaume about this?
f
no i'm already on it, just need a workaround for now
they're less familiar with this functionality since it's not heavily used eslewhere within spotify
f
hmm ok
but you can pass env vars?
f
yep 🙂
either in the docker container or from the api
this used to be how we set it, but i thought it was removed in favor of the cmd line arg
f
ya, so I think we can pass env vars for all flytectl arguments, cc @high-park-82026 is trying it out
btw, @famous-businessperson-24711 we were thinking of setting the raw-output-prefix at the project+domain level as a setting
you think that would help?
f
not in this current instance, but it's not a bad feature
h
What’s the flag you do want to pass to flytectl? 
--outputLocationPrefix
? You should be able to set:
Copy code
OUTPUTLOCATIONPREFIX=s3://<something> flytectl register...
Is what what you are looking for?
f
yes i think so!
let me try it
looks like it didn't work. will dig in later today
so i don't see this value anywhere in config
but what's this used for
@high-park-82026?
f
@famous-businessperson-24711 so the raw output data prefix exists, but these values are legacy and not used anymore
cc @thankful-minister-83577
f
As of one version? We have some projects that appear to work
t
one version?
so just looked, that setting is still used, but only because when i cleaned up the code, i didn’t delete all of 
flyte-cli
the 
RAW_OUTPUT_DATA_PREFIX
object is still used for 
flyte-cli register-files
which you’re correct has been subsumed by 
flytectl
but it’s still there in flyte-cli as well.
what’s your command rn dylan?
f
looks like flyte-cli. interestingly the env var override did not work though, trying the config file...
t
what env var?
f
FLYTE_AUTH_RAW_OUTPUT_DATA_PREFIX
t
and that isn’t working? can you copy paste the full 
flyte-cli
command?
f
flyte-cli register-files -p ${PROJECT} -d ${DOMAIN} -v ${IMAGE_TAG} ${INSECURE_OPTION} /tmp/_pb_output/*"
h
Yeah… I realized 
OUTPUTLOCATIONPREFIX
doesn’t work… flytectl has a bug where it doesn’t bind these envvars… 😞 I’ll see if this can be fixed easily…
t
sorry, distracted.
can you specify the flag? i was mistaken (obviously, the whole point of putting it into flyte-cli was so that it didn’t rely on the environment variable)
--output-location-prefix
f
Yea issue is that this isn’t exposed by our Ci/cd atm so looking for a workaround.
Side note can this be set at runtime in flytectl?
t
can talk a bit more after the oss sync.
but qq, i don’t think i understand.
you can make ci/cd run the 
flytectl
or 
flyte-cli
command… but you can’t pass an extra switch to it?
was this resolved btw dylan?
or still an issue?
165 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by