Thread
#flytekit
    Dylan Wilder

    Dylan Wilder

    7 months ago
    is it still possible to set the offloaded data storage via environment variable and/or config or is this now only passed at registration time?
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    During registration or execution time. The launch form for example has advanced options
    Please create an issue if you would like config
    Dylan Wilder

    Dylan Wilder

    7 months ago
    either. need a workaround for an issue where the feature is not exposed by CI/CD
    so i guess what are my options for specifying outside of passing via
    -o
    flag in registration script
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    What do you mean not exposed by ci cd?
    Dylan Wilder

    Dylan Wilder

    7 months ago
    our internal ci/cd is an api and they did not expose a way to set this value
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    ohh
    Dylan Wilder

    Dylan Wilder

    7 months ago
    as part of the registration step. We do have access to toggle the environment variables
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    should i talk to Guillaume about this?
    Dylan Wilder

    Dylan Wilder

    7 months ago
    no i'm already on it, just need a workaround for now
    they're less familiar with this functionality since it's not heavily used eslewhere within spotify
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    hmm ok
    but you can pass env vars?
    Dylan Wilder

    Dylan Wilder

    7 months ago
    yep 🙂
    either in the docker container or from the api
    this used to be how we set it, but i thought it was removed in favor of the cmd line arg
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    ya, so I think we can pass env vars for all flytectl arguments, cc @Haytham Abuelfutuh is trying it out
    btw, @Dylan Wilder we were thinking of setting the raw-output-prefix at the project+domain level as a setting
    you think that would help?
    Dylan Wilder

    Dylan Wilder

    7 months ago
    not in this current instance, but it's not a bad feature
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    7 months ago
    What’s the flag you do want to pass to flytectl?
    --outputLocationPrefix
    ? You should be able to set:
    OUTPUTLOCATIONPREFIX=s3://<something> flytectl register...
    Is what what you are looking for?
    Dylan Wilder

    Dylan Wilder

    7 months ago
    yes i think so!
    let me try it
    looks like it didn't work. will dig in later today
    so i don't see this value anywhere in config
    but what's this used for
    @Haytham Abuelfutuh?
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    @Dylan Wilder so the raw output data prefix exists, but these values are legacy and not used anymore
    cc @Yee
    Dylan Wilder

    Dylan Wilder

    7 months ago
    As of one version? We have some projects that appear to work
    Yee

    Yee

    7 months ago
    one version?
    so just looked, that setting is still used, but only because when i cleaned up the code, i didn’t delete all of
    flyte-cli
    the
    RAW_OUTPUT_DATA_PREFIX
    object is still used for
    flyte-cli register-files
    which you’re correct has been subsumed by
    flytectl
    but it’s still there in flyte-cli as well.
    what’s your command rn dylan?
    Dylan Wilder

    Dylan Wilder

    7 months ago
    looks like flyte-cli. interestingly the env var override did not work though, trying the config file...
    Yee

    Yee

    7 months ago
    what env var?
    Dylan Wilder

    Dylan Wilder

    7 months ago
    FLYTE_AUTH_RAW_OUTPUT_DATA_PREFIX
    Yee

    Yee

    7 months ago
    and that isn’t working? can you copy paste the full
    flyte-cli
    command?
    Dylan Wilder

    Dylan Wilder

    7 months ago
    flyte-cli register-files -p ${PROJECT} -d ${DOMAIN} -v ${IMAGE_TAG} ${INSECURE_OPTION} /tmp/_pb_output/*"
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    7 months ago
    Yeah… I realized
    OUTPUTLOCATIONPREFIX
    doesn’t work… flytectl has a bug where it doesn’t bind these envvars… 😞 I’ll see if this can be fixed easily…
    Yee

    Yee

    7 months ago
    sorry, distracted.
    can you specify the flag? i was mistaken (obviously, the whole point of putting it into flyte-cli was so that it didn’t rely on the environment variable)
    --output-location-prefix
    Dylan Wilder

    Dylan Wilder

    7 months ago
    Yea issue is that this isn’t exposed by our Ci/cd atm so looking for a workaround.
    Side note can this be set at runtime in flytectl?
    Yee

    Yee

    7 months ago
    can talk a bit more after the oss sync.
    but qq, i don’t think i understand.
    you can make ci/cd run the
    flytectl
    or
    flyte-cli
    command… but you can’t pass an extra switch to it?
    was this resolved btw dylan?
    or still an issue?