Security analysis and modified attack algorithms for a nonlinear optical cryptosystem based on DRPE

•Thes security of a nonlinear cryptosystem based on DRPE is analyzed•Shortcomings of the method are discussed and new constrains are highlighted•Modified chosen-plaintext attack and known-plaintext attack is proposed to crack the nonlinear cryptosystem•The numerical simulation results have been discussed for the proposed attack algorithms