The researchers introduce a novel slice discovery approach called Domino in their paper “DOMINO: DISCOVERING SYSTEMATIC ERRORS WITH CROSS-MODAL EMBEDDINGS.” The method uses a new family of cross-modal representation learning algorithms that produce semantically meaningful representations by combining images and text in the same latent space. Their experiment results show how cross-modal representations improve slice coherence while allowing Domino to produce plain language descriptions for recognized slices.

Slice discovery involves searching unstructured input material (such as photos, movies, and audio) for semantically significant subgroups where a model fails. Slice Discovery Methods (SDMs) compute a set of slicing functions that partition the dataset into slices given a labeled validation dataset and a trained classifier. An ideal SDM should identify slices, including cases when the model underperforms or has a high error rate. Furthermore, the slices should be identified by examples that are coherent or closely correlate with a human-understandable idea.

Domino follows a three-step process as follows:

Embed: It uses a cross-modal encoder to embed the validation images alongside the text in a shared embedding space.
Slice: It uses an error-aware mixture model to discover locations in the embedding space with a high concentration of mistakes.
Describe: Domino creates natural language descriptions of the slices to help practitioners understand the commonalities among the cases in each slice. It accomplishes this by surfacing the text closest to the slice in the embedding space using the cross-modal embeddings generated in Step 1.

https://www.marktechpost.com/2022/04/14/latest-research-from-stanford-introduces-domino-a-python-tool-for-identifying-and-describing-underperforming-slices-in-machine-learning-models/

#domino #evaluationcontrol