#2910 [Core feature] Support for File Families/Files With Indices Indices Issue created by CalvinLeather Motivation: Why do you think this is important? In bioinformatics work especially, file families, or files with indices, are common. This occurs when multiple files with the same stem, but different suffixes, jointly contain a dataset. This occurs either because a single csv-like has a index allowing for efficient lookups (e.g., a .vcf file and a tabix-based .vcf index), or when a library stores a file in multiple parts, often because matrix/tabular data is stored seperately from metadata about rows and columns (for example, a plink .bed triple containing genetic information about a number of samples/people, where the .bed file stores the matrix of data, .fam stores information about samples in the data e.g. IDs of the people aka the columns, and the .bim stores informations about the sites/variants of DNA aka the rows. Providing better support for these files will enhance the UX for people writing bioinformatics-pipelines with Flyte. Goal: What should the final outcome look like, ideally? The following should be true: • When using this input pattern in a shell task, the various components of the family are all put into the same path. This is required, as, for example, vcftools expects the index of a .vcf file to have the same stem (i.e., to be next to it with the same filename, e.g.

~/dir/test.vcf

and

~/dir/test.vcf.tbi

). • A task that takes a file family as input should take 1 input value, not N. In the user's mental model of computation to be done, the file family of N files is one entity, not N. Allowing the inputs/outputs to map to this model will improve the UX. • Helper utilities to enforce presence of files in the family at runtime (e.g., if you create one of these objects, and you're missing one the components, you can have an error raised) • It should be easy to extend these classes with additional methods/transformers to allow easy read out of the file families into memory. Typically this is done with another library (e.g., vcf + .vcf.tbi is loaded with pyvcf, a python library backing C-based htslib, a high performance library for loading/manipulating vcf files and their indices). Describe alternatives you've considered • FlyteDirectory • Extend FlyteFile (suggested by Greg Gydush in slack https://flyte-org.slack.com/archives/CP2HDHKE1/p1663719669816229?thread_ts=1663692816.505609&cid=CP2HDHKE1) • Compose multiple FlyteFiles into a composite Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte