https://flyte.org logo
#ask-the-community
Title
# ask-the-community
d

Dan Corbiani

02/14/2023, 4:10 PM
Apologies if I'm missing something in the docs, but I'm trying to do a MVP with a custom docker container task. What I've tried: 1.
flytectl demo start
2. Build the container with the tag
localhost:30000/custom_container:v1
3. exported the config
export FLYTECTL_CONFIG=/home/user/.flyte/config-sandbox.yaml
4. Tried to run the workflow
pyflyte run --remote workflows/mvp.py mvp --input_s3_path="<s3://data/path>"
The error I get is that it cannot pull the image. Am a I missing something very basic here?
n

Niels Bantilan

02/14/2023, 4:15 PM
hi Dan! just to double check, but in step (2) you’re doing
docker push localhost:30000/custom_container:v1
correct?
d

Dan Corbiani

02/14/2023, 4:15 PM
maybe that's my idiot mistake...
give me a sec to try that
n

Niels Bantilan

02/14/2023, 4:15 PM
and you’ll have to reference the image in
pyflyte run
as well, with the
--image
flag
d

Dan Corbiani

02/14/2023, 4:16 PM
the image tag is going to make it the default image, correct?
n

Niels Bantilan

02/14/2023, 4:17 PM
right, using
pyflyte run --remote --image localhost:30000/custom_container:v1 ...
makes the workflow run with that image
d

Dan Corbiani

02/14/2023, 4:18 PM
perfect. Yes, I was trying that and also doing it at the task level.
n

Niels Bantilan

02/14/2023, 4:18 PM
whereas
docker pushlocalhost:30000/…
should make it available in the local Flyte cluster docker registry
d

Dan Corbiani

02/14/2023, 4:18 PM
I was forgetting to push. Somehow I missed the fact that it was a registry and that I was going to need to push it.
bravo for setting this up so nicely. This is super slick.
n

Niels Bantilan

02/14/2023, 4:19 PM
thanks Dan! glad it’s been (fairly) smooth so far 😉 feel free to keep this thread open if you run into anything else
d

Dan Corbiani

02/14/2023, 4:20 PM
Now I'm happily running into errors that I expected from my end 🙂
Honestly, this is exceptionally smooth so far. Waiting to find the brick wall that we've run into with other frameworks.
n

Niels Bantilan

02/14/2023, 4:29 PM
we’re happy to help out if you do! 🧱
d

Dan Corbiani

02/14/2023, 5:00 PM
ok, maybe another simple question. Is there a way to remove the
_FSEC_
prefix from the environment variables? It seems like there should be a way to set it for this config (https://github.com/flyteorg/flytekit/blob/9d313429c577a919ec0ad4cd397a5db356a1df0d/flytekit/configuration/internal.py#L141-L159) object but I'm not sure where it should be set. Is that in config-sandbox.yaml?
n

Niels Bantilan

02/14/2023, 5:07 PM
are you trying to access secrets within a task?
d

Dan Corbiani

02/14/2023, 5:07 PM
yes
I've gotten it to work but the secret name is prefixed with
_fsec_
That throws off our configuration tool that is looking for them without the prefix. I can change it within the code but I wasn't sure if there was an easy way to modify it within flyte.
n

Niels Bantilan

02/14/2023, 5:12 PM
hmm, can you try
FLYTE_SECRET_ENV_PREFIX=""
? (docs here)
d

Dan Corbiani

02/14/2023, 6:12 PM
Where would I put that?
n

Niels Bantilan

02/14/2023, 7:12 PM
you can
export
it before running pyflyte run
d

Dan Corbiani

02/15/2023, 3:36 PM
Gave that a shot. No luck. I realized I'd still have to do something on our end because of the group. FWIW, I totally understand why the variables are named the way they are and think it makes a lot of sense.
n

Niels Bantilan

02/15/2023, 4:12 PM
@Eduardo Apolinario (eapolinario) @Kevin Su there should be a way to override the
FLYTE_SECRET_ENV_PREFIX
configuration right?
e

Eduardo Apolinario (eapolinario)

02/15/2023, 7:32 PM
unfortunately, this is not possible today.
Another user reached out some time ago asking about that, here's the issue: https://github.com/flyteorg/flyte/issues/3053
n

Niels Bantilan

02/15/2023, 8:05 PM
is there any recourse or workaround for this? @Dan Corbiani is this a blocker for you, or are there workarounds on your side you can implement?
d

Dan Corbiani

02/15/2023, 8:47 PM
totally not a blocker. I can make this work on our side.
n

Niels Bantilan

02/15/2023, 9:09 PM
also, I may not have full context here, but Flyte does come with a way to inject secrets into your tasks: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/use_secrets.html#how-to-use-secrets-injection-in-a-task Which you can access via a flytekit-specific API:
Copy code
@task(
    secret_requests=[
        Secret(key=USERNAME_SECRET, group=SECRET_GROUP),
        Secret(key=PASSWORD_SECRET, group=SECRET_GROUP),
    ]
)
def user_info_task() -> Tuple[str, str]:
    secret_username = flytekit.current_context().secrets.get(
        SECRET_GROUP, USERNAME_SECRET
    )
    secret_pwd = flytekit.current_context().secrets.get(SECRET_GROUP, PASSWORD_SECRET)
    # use secrets here
d

Dan Corbiani

02/22/2023, 8:55 PM
Saw I didn't reply to this. That was what we were using. We were running into some issues since certain tools in our workflow require environment variables to be set. We were able to pretty quickly get all of the pieces fit together and basically used something similar to what you posted.
n

Niels Bantilan

02/27/2023, 4:57 PM
Awesome @Dan Corbiani! glad you were able to get things working.
6 Views