Channels
#flytekit
Title
d
Dylan Wilder
02/03/2022, 2:41 PMis it still possible to set the offloaded data storage via environment variable and/or config or is this now only passed at registration time?
k
Ketan (kumare3)
02/03/2022, 3:02 PM
During registration or execution time. The launch form for example has advanced options
Please create an issue if you would like config
d
Dylan Wilder
02/03/2022, 4:17 PMeither. need a workaround for an issue where the feature is not exposed by CI/CD
so i guess what are my options for specifying outside of passing via
-ok
Ketan (kumare3)
02/03/2022, 4:18 PM
What do you mean not exposed by ci cd?
d
Dylan Wilder
02/03/2022, 6:37 PMour internal ci/cd is an api and they did not expose a way to set this value
k
Ketan (kumare3)
02/03/2022, 6:37 PM
ohh
d
Dylan Wilder
02/03/2022, 6:37 PMas part of the registration step. We do have access to toggle the environment variables
k
Ketan (kumare3)
02/03/2022, 6:37 PM
should i talk to Guillaume about this?
d
Dylan Wilder
02/03/2022, 6:38 PMno i'm already on it, just need a workaround for now
they're less familiar with this functionality since it's not heavily used eslewhere within spotify
k
Ketan (kumare3)
02/03/2022, 6:38 PM
hmm ok
but you can pass env vars?
d
Dylan Wilder
02/03/2022, 6:39 PMyep 🙂
either in the docker container or from the api
this used to be how we set it, but i thought it was removed in favor of the cmd line arg
k
Ketan (kumare3)
02/03/2022, 6:40 PM
ya, so I think we can pass env vars for all flytectl arguments, cc @Haytham Abuelfutuh is trying it out
btw, @Dylan Wilder we were thinking of setting the raw-output-prefix at the project+domain level as a setting
you think that would help?
d
Dylan Wilder
02/03/2022, 6:42 PMnot in this current instance, but it's not a bad feature
h
Haytham Abuelfutuh
02/03/2022, 6:51 PM
What’s the flag you do want to pass to flytectl?
--outputLocationPrefix Copy code
OUTPUTLOCATIONPREFIX=s3://<something> flytectl register...d
Dylan Wilder
02/03/2022, 6:59 PMyes i think so!
let me try it
looks like it didn't work. will dig in later today
so i don't see this value anywhere in config
but what's this used for
@Haytham Abuelfutuh?
k
Ketan (kumare3)
02/07/2022, 11:57 PM
@Dylan Wilder so the raw output data prefix exists, but these values are legacy and not used anymore
cc @Yee
d
Dylan Wilder
02/07/2022, 11:57 PMAs of one version? We have some projects that appear to work
y
Yee
02/08/2022, 12:00 AM
one version?
so just looked, that setting is still used, but only because when i cleaned up the code, i didn’t delete all of
flyte-clithe
RAW_OUTPUT_DATA_PREFIXflyte-cli register-filesflytectlwhat’s your command rn dylan?
d
Dylan Wilder
02/08/2022, 12:04 AMlooks like flyte-cli. interestingly the env var override did not work though, trying the config file...
y
Yee
02/08/2022, 12:04 AM
what env var?
d
Dylan Wilder
02/08/2022, 12:06 AMFLYTE_AUTH_RAW_OUTPUT_DATA_PREFIX
y
Yee
02/08/2022, 12:09 AM
and that isn’t working? can you copy paste the full
flyte-clid
Dylan Wilder
02/08/2022, 12:31 AMflyte-cli register-files -p ${PROJECT} -d ${DOMAIN} -v ${IMAGE_TAG} ${INSECURE_OPTION} /tmp/_pb_output/*"
h
Haytham Abuelfutuh
02/08/2022, 12:31 AM
Yeah… I realized
OUTPUTLOCATIONPREFIXy
Yee
02/08/2022, 12:50 AM
sorry, distracted.
can you specify the flag? i was mistaken (obviously, the whole point of putting it into flyte-cli was so that it didn’t rely on the environment variable)
--output-location-prefixd
Dylan Wilder
02/08/2022, 2:43 PMYea issue is that this isn’t exposed by our Ci/cd atm so looking for a workaround.
Side note can this be set at runtime in flytectl?
y
Yee
02/08/2022, 4:54 PM
can talk a bit more after the oss sync.
but qq, i don’t think i understand.
you can make ci/cd run the
flytectlflyte-cliwas this resolved btw dylan?
or still an issue?