Apologies if I'm missing something in the docs, but I'm trying to do a MVP with a custom docker container task. What I've tried: 1.

flytectl demo start

2. Build the container with the tag

localhost:30000/custom_container:v1

3. exported the config

export FLYTECTL_CONFIG=/home/user/.flyte/config-sandbox.yaml

4. Tried to run the workflow

pyflyte run --remote workflows/mvp.py mvp --input_s3_path="<s3://data/path>"

The error I get is that it cannot pull the image. Am a I missing something very basic here?

hi Dan! just to double check, but in step (2) you’re doing

docker push localhost:30000/custom_container:v1

correct?

maybe that's my idiot mistake...

and you’ll have to reference the image in

pyflyte run

as well, with the

--image

flag

the image tag is going to make it the default image, correct?

right, using

pyflyte run --remote --image localhost:30000/custom_container:v1 ...

makes the workflow run with that image

perfect. Yes, I was trying that and also doing it at the task level.

whereas

docker pushlocalhost:30000/…

should make it available in the local Flyte cluster docker registry

I was forgetting to push. Somehow I missed the fact that it was a registry and that I was going to need to push it.

thanks Dan! glad it’s been (fairly) smooth so far 😉 feel free to keep this thread open if you run into anything else

Now I'm happily running into errors that I expected from my end 🙂

we’re happy to help out if you do! 🧱

ok, maybe another simple question. Is there a way to remove the

_FSEC_

prefix from the environment variables? It seems like there should be a way to set it for this config (https://github.com/flyteorg/flytekit/blob/9d313429c577a919ec0ad4cd397a5db356a1df0d/flytekit/configuration/internal.py#L141-L159) object but I'm not sure where it should be set. Is that in config-sandbox.yaml?

are you trying to access secrets within a task?

yes

hmm, can you try

FLYTE_SECRET_ENV_PREFIX=""

? (docs here)

Where would I put that?

you can

export

it before running pyflyte run

Gave that a shot. No luck. I realized I'd still have to do something on our end because of the group. FWIW, I totally understand why the variables are named the way they are and think it makes a lot of sense.

@Eduardo Apolinario (eapolinario) @Kevin Su there should be a way to override the

FLYTE_SECRET_ENV_PREFIX

configuration right?

unfortunately, this is not possible today.

is there any recourse or workaround for this? @Dan Corbiani is this a blocker for you, or are there workarounds on your side you can implement?

totally not a blocker. I can make this work on our side.

also, I may not have full context here, but Flyte does come with a way to inject secrets into your tasks: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/use_secrets.html#how-to-use-secrets-injection-in-a-task Which you can access via a flytekit-specific API:

@task(
    secret_requests=[
        Secret(key=USERNAME_SECRET, group=SECRET_GROUP),
        Secret(key=PASSWORD_SECRET, group=SECRET_GROUP),
    ]
)
def user_info_task() -> Tuple[str, str]:
    secret_username = flytekit.current_context().secrets.get(
        SECRET_GROUP, USERNAME_SECRET
    )
    secret_pwd = flytekit.current_context().secrets.get(SECRET_GROUP, PASSWORD_SECRET)
    # use secrets here

Saw I didn't reply to this. That was what we were using. We were running into some issues since certain tools in our workflow require environment variables to be set. We were able to pretty quickly get all of the pieces fit together and basically used something similar to what you posted.

Awesome @Dan Corbiani! glad you were able to get things working.