The state seeks billions in penalties and a halt to operations, joining other states in a legal battle over regulation of prediction markets.