Regulator cites AML failures, missing SARs and weak controls in historic broker-dealer penalty against Canadian firm’s US business.