https://flyte.org logo
d

Dylan Wilder

02/03/2022, 2:41 PM
is it still possible to set the offloaded data storage via environment variable and/or config or is this now only passed at registration time?
k

Ketan (kumare3)

02/03/2022, 3:02 PM
During registration or execution time. The launch form for example has advanced options
Please create an issue if you would like config
d

Dylan Wilder

02/03/2022, 4:17 PM
either. need a workaround for an issue where the feature is not exposed by CI/CD
so i guess what are my options for specifying outside of passing via
-o
flag in registration script
k

Ketan (kumare3)

02/03/2022, 4:18 PM
What do you mean not exposed by ci cd?
d

Dylan Wilder

02/03/2022, 6:37 PM
our internal ci/cd is an api and they did not expose a way to set this value
k

Ketan (kumare3)

02/03/2022, 6:37 PM
ohh
d

Dylan Wilder

02/03/2022, 6:37 PM
as part of the registration step. We do have access to toggle the environment variables
k

Ketan (kumare3)

02/03/2022, 6:37 PM
should i talk to Guillaume about this?
d

Dylan Wilder

02/03/2022, 6:38 PM
no i'm already on it, just need a workaround for now
they're less familiar with this functionality since it's not heavily used eslewhere within spotify
k

Ketan (kumare3)

02/03/2022, 6:38 PM
hmm ok
but you can pass env vars?
d

Dylan Wilder

02/03/2022, 6:39 PM
yep 🙂
either in the docker container or from the api
this used to be how we set it, but i thought it was removed in favor of the cmd line arg
k

Ketan (kumare3)

02/03/2022, 6:40 PM
ya, so I think we can pass env vars for all flytectl arguments, cc @Haytham Abuelfutuh is trying it out
btw, @Dylan Wilder we were thinking of setting the raw-output-prefix at the project+domain level as a setting
you think that would help?
d

Dylan Wilder

02/03/2022, 6:42 PM
not in this current instance, but it's not a bad feature
h

Haytham Abuelfutuh

02/03/2022, 6:51 PM
What’s the flag you do want to pass to flytectl?
--outputLocationPrefix
? You should be able to set:
Copy code
OUTPUTLOCATIONPREFIX=s3://<something> flytectl register...
Is what what you are looking for?
d

Dylan Wilder

02/03/2022, 6:59 PM
yes i think so!
let me try it
looks like it didn't work. will dig in later today
so i don't see this value anywhere in config
but what's this used for
@Haytham Abuelfutuh?
k

Ketan (kumare3)

02/07/2022, 11:57 PM
@Dylan Wilder so the raw output data prefix exists, but these values are legacy and not used anymore
cc @Yee
d

Dylan Wilder

02/07/2022, 11:57 PM
As of one version? We have some projects that appear to work
y

Yee

02/08/2022, 12:00 AM
one version?
so just looked, that setting is still used, but only because when i cleaned up the code, i didn’t delete all of
flyte-cli
the
RAW_OUTPUT_DATA_PREFIX
object is still used for
flyte-cli register-files
which you’re correct has been subsumed by
flytectl
but it’s still there in flyte-cli as well.
what’s your command rn dylan?
d

Dylan Wilder

02/08/2022, 12:04 AM
looks like flyte-cli. interestingly the env var override did not work though, trying the config file...
y

Yee

02/08/2022, 12:04 AM
what env var?
d

Dylan Wilder

02/08/2022, 12:06 AM
FLYTE_AUTH_RAW_OUTPUT_DATA_PREFIX
y

Yee

02/08/2022, 12:09 AM
and that isn’t working? can you copy paste the full
flyte-cli
command?
d

Dylan Wilder

02/08/2022, 12:31 AM
flyte-cli register-files -p ${PROJECT} -d ${DOMAIN} -v ${IMAGE_TAG} ${INSECURE_OPTION} /tmp/_pb_output/*"
h

Haytham Abuelfutuh

02/08/2022, 12:31 AM
Yeah… I realized
OUTPUTLOCATIONPREFIX
doesn’t work… flytectl has a bug where it doesn’t bind these envvars… 😞 I’ll see if this can be fixed easily…
y

Yee

02/08/2022, 12:50 AM
sorry, distracted.
can you specify the flag? i was mistaken (obviously, the whole point of putting it into flyte-cli was so that it didn’t rely on the environment variable)
--output-location-prefix
d

Dylan Wilder

02/08/2022, 2:43 PM
Yea issue is that this isn’t exposed by our Ci/cd atm so looking for a workaround.
Side note can this be set at runtime in flytectl?
y

Yee

02/08/2022, 4:54 PM
can talk a bit more after the oss sync.
but qq, i don’t think i understand.
you can make ci/cd run the
flytectl
or
flyte-cli
command… but you can’t pass an extra switch to it?
was this resolved btw dylan?
or still an issue?