Possibly.
The shinobi were part of the people considered outside and below the castes, including entertainers (which may or may not be a euphemism), the Ainu, convicted criminals, and unclean professions. There also used to be people above the castes, like the shogun and the emperor and their families, but that stopped with the fall of the empire.
Rogue samurai seem to be mostly mercenaries, rather than working for a daimyo as part of a samurai family. Authors seem to think farmers, artisans, merchants, and basically anyone who isn't part of the samurai class should not be fighting.
The main cause of the breakdown seems to be the general instability following the fall of the empire, and the rise of shinobi and warlords destabilizing the military structure, and the shinobi and warlords not caring about castes. It's unclear how far back the informal structure dates, though it was only formally encoded a few centuries ago.
Not everyone fell into a caste, and the merchant, artisan, and farmer castes were semi-fluid.
Warlords mostly seem to have arisen as a result of farmer or merchant rebellions, though there's been a few rogue samurai and shinobi who've established themselves. They vary tremendously in their politics - some are harsh to their people, some are permissive, and the permissive ones seem to be more popular. There's at least one woman who graduated from highwayman to warlord, and seems to be something of a folk hero. It seems like hiring low-level shinobi and mercenary forces in enough numbers to overwhelm the daimyo's few samurai is relatively cheap.