#4654 [Core feature] Allow passing additional metadata in inputs/outputs of tasks/workflows Issue created by EngHabu ### Motivation: Why do you think this is important? There are a few instances we faced in the past where we wanted to pass additional contextual information about the inputs/outputs of a task that are not themselves inputs/outputs (The most recent example is flyte decks). Unfortunately the way we had to pass that is by convention. The engine (i.e. Propeller) assumes a task writes decks, checks for the file existence and bails if it's not there. Conventions are not very strong :-) and limit the optimizations the system can make. Another instance of this is executing, generating and exporting input validation report from Great Expectations or Pandera. This is not really an output but rather metadata about the input/output dataframes. ### Goal: What should the final outcome look like, ideally? This is a complex change and its complexity stems from a few aspects: 1. The system assumes Inputs and outputs are always LiteralMaps... which is an assumption we are breaking 2. Backward compatibility, at any point in time users expect to use an old flytekit with a new backend and maybe an even older frontend and some 4th version of flyte propeller. Making the system backward and forward compatible is crucial for a smooth rollout. Here is a proposed breakdown of the work particularly around the component boundaries: • Make the change in

flyteidl

and all backend code. This code should default to reading the new

InputData

and

OutputData

and fallback to

LiteralMap

if unmarshaling fails, this will ensure old

propeller

can continue to work with a newer

admin

(but not vis versa) • When

flytepropeller

writes

inputs.pb

for a task, it should gate that behavior based on the version of flytekit. If it's not a

flytekit

runtime or it's an old version, it should default to writing the old format until other SDKs are updated. • Make the change in

flytekit

to do the same. • Make the change in

flyteconsole

to do the same. The rollout plan is: 1. Update flyteadmin to latest 2. Update flyteconsole, copilot, propeller, flytekit to latest in any order As long as flyteadmin is updated first, the system assumptions will hold. ### Describe alternatives you've considered Continue to use conventions to pass data... not good ### Propose: Link/Inline OR Additional context No response ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte