New Orleans actually was the NYC of the South in the 19th century; at the time of the Civil War, it was about 2.5x more populous than any other Southern city (source) excluding Baltimore and Washington, and was about 4x more populous than any other Confederate city. Excluding Baltimore and Washington, New Orleans was still the most populous city in the South in 1940. And subjectively, my understanding is that New Orleans was generally considered to be the South's "most cosmopolitan" city for much of its history.
So while issues unique to New Orleans certainly play a role, a lot of the answer comes down to differences between the North and South in the 19th and 20th centuries.
This all strongly suggests heat and disease was the limitation. Today in the AC/antibacterial/vaccine era, Houston is next door in the same climate and it's going to be the 3rd largest city in the country (and 2nd largest east of the Rockies) within the next decade.
