https://flyte.org logo
Join SlackCommunities
Powered by
Hi everyone, I’m encountering an issue when trying...
# flyte-support
h
Hi everyone, I’m encountering an issue when trying to register the following Flyte workflow from the examples: https://docs.flyte.org/en/latest/flytesnacks/examples/mnist_classifier/pytorch_single_node_multi_gpu.html This workflow was working fine until yesterday, but now I get the following error:
Copy code
Error: rpc error: code = InvalidArgument desc = Type mismatch for Parameter hp in default_inputs has type simple:STRUCT  metadata:{fields:{key:"additionalProperties"  value:{bool_value:false}}  fields:{key:"properties"  value:{struct_value:{fields:{key:"backend"  value:{struct_value:{fields:{key:"default"  value:{string_value:"gloo"}}  fields:{key:"type"  value:{string_value:"string"}}}}}  fields:{key:"batch_size"  value:{struct_value:{fields:{key:"default"  value:{number_value:64}}  fields:{key:"type"  value:{string_value:"integer"}}}}}  fields:{key:"epochs"  value:{struct_value:{fields:{key:"default"  value:{number_value:10}}  fields:{key:"type"  value:{string_value:"integer"}}}}}  fields:{key:"learning_rate"  value:{struct_value:{fields:{key:"default"  value:{number_value:0.01}}  fields:{key:"type"  value:{string_value:"number"}}}}}  fields:{key:"log_interval"  value:{struct_value:{fields:{key:"default"  value:{number_value:10}}  fields:{key:"type"  value:{string_value:"integer"}}}}}  fields:{key:"seed"  value:{struct_value:{fields:{key:"default"  value:{number_value:1}}  fields:{key:"type"  value:{string_value:"integer"}}}}}  fields:{key:"sgd_momentum"  value:{struct_value:{fields:{key:"default"  value:{number_value:0.5}}  fields:{key:"type"  value:{string_value:"number"}}}}}  fields:{key:"test_batch_size"  value:{struct_value:{fields:{key:"default"  value:{number_value:1000}}  fields:{key:"type"  value:{string_value:"integer"}}}}}}}}  fields:{key:"title"  value:{string_value:"Hyperparameters"}}  fields:{key:"type"  value:{string_value:"object"}}}  annotation:{annotations:{fields:{key:"cache-key-metadata"  value:{struct_value:{fields:{key:"serialization-format"  value:{string_value:"msgpack"}}}}}}}  structure:{dataclass_type:{key:"backend"  value:{simple:STRING}}  dataclass_type:{key:"batch_size"  value:{simple:INTEGER}}  dataclass_type:{key:"epochs"  value:{simple:INTEGER}}  dataclass_type:{key:"learning_rate"  value:{simple:FLOAT}}  dataclass_type:{key:"log_interval"  value:{simple:INTEGER}}  dataclass_type:{key:"seed"  value:{simple:INTEGER}}  dataclass_type:{key:"sgd_momentum"  value:{simple:FLOAT}}  dataclass_type:{key:"test_batch_size"  value:{simple:INTEGER}}}, expected simple:BINARY
It seems related to the 
Hyperparameters
class being treated as a complex structured type. Has anything changed recently, or is there something I need to update in my code or environment? Any pointers would be greatly appreciated! Thank you!
t
feels like there’s a mismatch in versions. dataclasses were changed substantially in 1.14 to support pydantic (if you don’t use dataclasses/pydantic/untyped dicts then it doesn’t matter), but in order to support it you need 1.14 on backend, and all tasks downstream of a task that uses 1.14.
looking at the error it looks like something is trying to use 1.13 and something else is on 1.14
c
@thankful-minister-83577 I expect older workflows to work after upgrading Flyte. We've upgraded our backend to flyte-binary-1.14.2. It fails when trying to deserialize a dict input. The workflow-image is flytekit1.12 and backend is 1.14 so it delivers a msgpack. How can we instruct the backend to identify older version of flytekit?
t
this shouldn’t happen - it’s a requirement that the backend be updated if flytekit is updated but not the other way around. 1.12 should continue to work. can you paste any more information about what’s happening? structure of the workflow would help along with any task signatures.
is caching involved at all?
how is the dict input being provided? through the UI or another means?
c
Thanks for the response @thankful-minister-83577 I'll start by providing the error message, and try to share a simplified workflow version a little later. No caching involved. The inputs that fail to convert are of type 
dict[Any, Any]
The error message:
Copy code
[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/home/.local/lib/python3.11/site-packages/flytekit/exceptions/scopes.py", line 183, in system_entry_point
        return wrapped(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/.local/lib/python3.11/site-packages/flytekit/core/base_task.py", line 701, in dispatch_execute
        raise type(exc)(msg) from exc

Message:

    TypeTransformerFailedError: Failed to convert inputs of task 'services.flyte_crawler.workflows.OOO_OOO_wf.OOO_OOO_scrapy':
  Error converting input 'settings' at position 2:
  Cannot convert from <FlyteLiteral(Literal) scalar { binary { value: "\200" tag: "msgpack" } }> to dict[typing.Any, typing.Any]

SYSTEM ERROR! Contact platform administrators.
The task is 
dynamic
It's basically something like (simplified but not compiled):
Copy code
@dynamic
def scrapy_task(
    user_arguments: dict[Any, Any],
    settings: dict[Any, Any],
) -> None:
  ...


@workflow
def scrapy_task_wf(
    user_arguments: dict[Any, Any],
    settings: dict[Any, Any],
) -> None:
    scrapy_task(
        user_arguments=user_arguments,
        settings=settings,
    )
The workflow is triggered using Flytekit 1.14 (which now when explaining it to you, might be the issue)
t
ah yes
there’s a flag you can add
one sec
c
I don't really want to maintain what workflows were upgraded or not
t
true.
but it’s 
FLYTE_USE_OLD_DC_FORMAT
if you just want to verify. 
export FLYTE_USE_OLD_DC_FORMAT=true
if you need it
c
@thankful-minister-83577 is this supposed to be exported in the environment where it's triggered?
will it work if I'm using code to trigger? ('remote')
t
should work in both cases yes.
c
@thankful-minister-83577 thanks for your help! it's a little annoying though... because the only is to specify an environment variable, in the code it's 
os.environ[...
. But I suppose it'll be only temoprary
t
yeah i’m sorry, we know it’s annoying. we hope the benefits of it outweigh the pain
c
@thankful-minister-83577 what I suggested above doesn't even really work... consider a workflow registered using flytekit 1.12 to an environment of 1.14. Subsequently the workflow is triggered using flytekit 1.14 with the environment variable set to "true". I still get an error:
Copy code
Failed to trigger workflow for pipeline 4: USER:BadInputToAPI: error=None, cause=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "invalid settings input wrong type. Expected union_type:{variants:{simple:STRUCT structure:{tag:"Python Dictionary"}} variants:{simple:NONE structure:{tag:"none"}}}, but got union_type:{variants:{simple:STRUCT annotation:{annotations:{fields:{key:"cache-key-metadata" value:{struct_value:{fields:{key:"serialization-format" value:{string_value:"msgpack"}}}}}}} structure:{tag:"Typed Dict"}}}"
So bottom line - looks we are going to have to go through a significant migration
probably include downtime
t
could you paste the signature of thing you’re trying to run?
@damp-lion-88352 when you have a chance, could you help see if you can repro this? i can work with you
c
Copy code
@task(
    container_image="<http://cr.flyte.org/flyteorg/flytekit:py3.11-1.12.3|cr.flyte.org/flyteorg/flytekit:py3.11-1.12.3>",
)
def say_hello(settings: Optional[dict[Any, Any]] = None) -> str:
the repro is fairly simple - just add a dict parameter (I used the "hello world" workflow from the flytesnacks repo) register using pyflyte 1.12. using flytekit 1.14 and a flyte remote, try to trigger the workflow, passing something in the dict. doesn't work whether the envvar is set or not
28 Views
Open in Slack
PreviousNext
Powered by