Terry Anderson and P. J. Hill "An American Experiment in Anarcho-Capitalism: The Not so Wild, Wild West"
(The authors have just brought out a book on the subject:
The Not So Wild, Wild West: Property Rights on the Frontier
Tom Bell, "Polycentric Law."
It might also be worth exploring the literature on western mining camps, which established their own legal system, and on the vigilantes, in particular in California. I haven't yet found any good online sources on either.