Hey @sparse-addition-80036 nice presentation... bioinformatics-oriented question... how have you been using FlyteFiles for indexed files? Do you have tabix indexes etc. on your bed of vcf files? This has been something my company has been thinking about/prototyping (using 2 flyte files is a bit annoying because the index doesn’t end up “next to” the file its indexing so in our shell tasks we have to cp the files + its index into a new temp folder, but what if we had something like FlyteDirectory that wrapped a file family with an expected number of files)

interesting

I haven't though enough about how *t*his should work. Its tough to do the static checking because for some of these "file families", they are treated by C code/binaries as though they were one file... e.g, doing file_a/file_b doesn't make sense, rather the binary takes a filestem without the suffix

got it thank you.

hey Calvin, it seems like

FlyteDirectory

would be appropriate to use here… how are these “file families” distinct from directories?

Yeah good question, maybe FlyteDirectory, or an extension of it, is the right way to go

Sometimes you want to download just the index so you can check whether something exists in the large file it indexes before you download it

interesting… yeah I think if you can write down requirements like this in an issue it would help us figure out how to extend FlyteDirectories https://github.com/flyteorg/flyte/issues/new?assignees=&labels=enhancement%2Cuntriaged&template=feature_request.yaml&title=%5BCore+feature%5D+

Hey @shy-holiday-15500 , we had a similar need when dealing with indexed files. The way we ended up solving it was with a custom type that can be extended to handle files with any number of associated indexes.

class FlyteFileWithIndex(FlyteFile, metaclass=TypeTransformerMeta):
    index_extensions: typing.List[str] = []
    index_requirement: typing.Literal["all", "any"] = "any"

The type transformer looks at the list of index extensions, checks if “any” or “all” of the index files exists based on the specified index_requirement (“any” is useful for things like BAM that can have either “.bai” or “.bam.bai” suffix, “all” is useful for strict matching). If index(es) exists, it will download both the file and its associated index files in the same directory, otherwise it will error. So for example, this is what a type would look like to handle BAM files, which can be defined inside the workflow or in a library that is used by the workflow:

class BamFile(FlyteFileWithIndex):
    index_extensions = [".bai", ".bam.bai"]
    index_requirement = "any"

Nice! This is exactly what I was thinking about

do you think you can help port this to an issue @shy-holiday-15500?

Yes!

tyty