https://flyte.org logo
Join Slack
Powered by
is it still possible to set the offloaded data sto...
# flytekit
d
is it still possible to set the offloaded data storage via environment variable and/or config or is this now only passed at registration time?
k
During registration or execution time. The launch form for example has advanced options
Please create an issue if you would like config
d
either. need a workaround for an issue where the feature is not exposed by CI/CD
so i guess what are my options for specifying outside of passing via 
-o
flag in registration script
k
What do you mean not exposed by ci cd?
d
our internal ci/cd is an api and they did not expose a way to set this value
k
ohh
d
as part of the registration step. We do have access to toggle the environment variables
k
should i talk to Guillaume about this?
d
no i'm already on it, just need a workaround for now
they're less familiar with this functionality since it's not heavily used eslewhere within spotify
k
hmm ok
but you can pass env vars?
d
yep 🙂
either in the docker container or from the api
this used to be how we set it, but i thought it was removed in favor of the cmd line arg
k
ya, so I think we can pass env vars for all flytectl arguments, cc @Haytham Abuelfutuh is trying it out
btw, @Dylan Wilder we were thinking of setting the raw-output-prefix at the project+domain level as a setting
you think that would help?
d
not in this current instance, but it's not a bad feature
h
What’s the flag you do want to pass to flytectl? 
--outputLocationPrefix
? You should be able to set:
Copy code
OUTPUTLOCATIONPREFIX=s3://<something> flytectl register...
Is what what you are looking for?
d
yes i think so!
let me try it
looks like it didn't work. will dig in later today
so i don't see this value anywhere in config
but what's this used for
@Haytham Abuelfutuh?
k
@Dylan Wilder so the raw output data prefix exists, but these values are legacy and not used anymore
cc @Yee
d
As of one version? We have some projects that appear to work
y
one version?
so just looked, that setting is still used, but only because when i cleaned up the code, i didn’t delete all of 
flyte-cli
the 
RAW_OUTPUT_DATA_PREFIX
object is still used for 
flyte-cli register-files
which you’re correct has been subsumed by 
flytectl
but it’s still there in flyte-cli as well.
what’s your command rn dylan?
d
looks like flyte-cli. interestingly the env var override did not work though, trying the config file...
y
what env var?
d
FLYTE_AUTH_RAW_OUTPUT_DATA_PREFIX
y
and that isn’t working? can you copy paste the full 
flyte-cli
command?
d
flyte-cli register-files -p ${PROJECT} -d ${DOMAIN} -v ${IMAGE_TAG} ${INSECURE_OPTION} /tmp/_pb_output/*"
h
Yeah… I realized 
OUTPUTLOCATIONPREFIX
doesn’t work… flytectl has a bug where it doesn’t bind these envvars… 😞 I’ll see if this can be fixed easily…
y
sorry, distracted.
can you specify the flag? i was mistaken (obviously, the whole point of putting it into flyte-cli was so that it didn’t rely on the environment variable)
--output-location-prefix
d
Yea issue is that this isn’t exposed by our Ci/cd atm so looking for a workaround.
Side note can this be set at runtime in flytectl?
y
can talk a bit more after the oss sync.
but qq, i don’t think i understand.
you can make ci/cd run the 
flytectl
or 
flyte-cli
command… but you can’t pass an extra switch to it?
was this resolved btw dylan?
or still an issue?
164 Views
Open in Slack
PreviousNext
Powered by