Light quark matter (i.e. udQM) comprised of up and down quarks might be stable with respect to nuclear matter and strange quark matter. In this work we investigate the stability of nuggets made of such types of matter, which shows that udQM nuggets at certain sizes are more stable than others if a large symmetry energy is adopted. In such cases, there may be compact dwarfs comprised entirely of udQM nuggets and electrons, i.e., udQM dwarfs. We then examine the structure of such udQM dwarfs and consider their possibility of being covered by normal matter crusts. It is found that udQM dwarfs typically have smaller radii and those covered by normal matter can explain the recently observed white dwarfs with unusually small radii.