https://flyte.org logo
#ask-the-community
Title
# ask-the-community
m

Martin Hwasser

10/21/2022, 12:29 PM
Hi, I have a strange error where I am creating several LaunchPlans with different names using
flytekit.LaunchPlan.get_or_create
. However, when I try to register this archive with the launchplans, it fails on a particular one with the name:
a-cst-old
, giving the error:
Error: rpc error: code = InvalidArgument desc = missing name
It’s just this name and the fixed input (string) that’s changed, nothing else, if I prefix this name with something else like, it works. And on a related note, is there a way to delete, not just archive, all launchplans?
FWIW I have attached the protobuf here:
s

Shivay Lamba

10/21/2022, 3:44 PM
But @Kevin Su or @Prafulla Mahindrakar could add to it
p

Prafulla Mahindrakar

10/26/2022, 4:46 PM
Hi @Martin Hwasser thats a very strange issue . Will need to check more on it . Also regarding deletion, currently we dont have a delete functionality in flyte and every object in flyte can be archived . This is by design i think . You can always choose to delete the archived ones from the DB .
So strange comes due to unmarshalling this proto works for both TaskSpec and Launchplan object. ``````
Following test demonstrates it. I am assuming you are using flytectl to register where you see this error.
Copy code
func TestUnmarshalProtoLaunchPlan(t *testing.T) {
	pbBytes, err := os.ReadFile("/Users/praful/Downloads/17_a-cst-old_3.pb")
	lp := &admin.LaunchPlan{}
	err = proto.Unmarshal(pbBytes, lp)
	assert.Nil(t, err)
	assert.Equal(t, "a-cst-old", lp.Id.Name)
}

func TestUnmarshalProtoTaskSpec(t *testing.T) {
	pbBytes, err := os.ReadFile("/Users/praful/Downloads/17_a-cst-old_3.pb")
	lp := &admin.TaskSpec{}
	err = proto.Unmarshal(pbBytes, lp)
	assert.Nil(t, err)
	assert.Equal(t, "a-cst-old", lp.Template.Id.Name)
}
@Martin Hwasser may be you can share the updated proto where this works for you and we can run both test on it. I bet it fails the 2nd test and hence it works for you.
flytectl register relies on the fact the object will be one of the following types and not multiple • admin.LaunchPlanadmin.WorkflowSpecadmin.TaskSpec In your case the proto is able to be unmarshalled without error into Launchplan and TaskSpec type and taskSpec type gets precedence in flytectl code and thats the one thats gets sent to admin for registration and fails since the Name field in the unmarshalled taskSpec proto is empty . If you unrmarshal it with Launchplan it has the correct Id.Name field
Cc: @Haytham Abuelfutuh
h

Haytham Abuelfutuh

10/28/2022, 5:50 PM
I downloaded an example from flytesnacks releases. And I do not see a way to distinguish between types based on the file. I believe the protocol based on such convention is not good enough… I can think of two ways to improve the package -> register behavior: 1. Change the naming of these files to include a prefix informing flytectl/pyflyte register about how to parse these (e.g. `1_tsk_blast`… `2_wf_blast`… etc.).. This change will need to happen in all SDKs (python, java, scala)… the register behavior will need to be backward compatible… 2. Create an envelope message for the data serialized in these files… something like below. And update the registration logic to always only unmarshal into that envelope type.
Copy code
message RegistrationEntity {
  oneof item {
    TaskTemplate task = 1;
    WorkflowTemplate workflow = 2;
...
  }
}
There is no way around backward compatibility ugliness here, we will have to fallback to existing logic. Both options require changes in flytekit. I do think the second option a tad better since it doesn’t rely on string comparisons and doesn’t assume the existence of “files” to begin with (this logic works just fine with REST APIs and programmatic APIs)… If we don’t have one already, we should file an issue on this and list these (and other?) options considered and maybe move the discussion there.. CC @Yee @Ketan (kumare3) @Eduardo Apolinario (eapolinario)
p

Prafulla Mahindrakar

11/01/2022, 6:39 AM
2 Views