57 Tory MPs have said today they will vote against a George Osborne post-referendum budget.
As a budget is effectively a confidence vote, that pretty much guarantees the government will fall in the event of a no vote.
So a palace coup, or another general election? I'm not sure the general public would be able to cope with yet another election; but it would be completely unpredictable at this point.
I am not sure that the old conventions still apply.
In the pre-fixed term Parliament era the convention was that a government defeated on a major financial measure should either resign or advise a dissolution of Parliament. If it resigned then the conventional response was for the monarch to invite the Leader of the Opposition to form a government. In practice the new Prime Minister, being supported by a minority of the House of Commons, would then ask for a dissolution.
The convention was weakened in the UK in the 1970s, when the Callaghan minority government continued after losing votes on amendments to financial legislation.
Under the Fixed-term Parliaments Act 2011 the conventions appear to have been largely replaced by statutory provisions. The House of Commons briefing note about the Act summarises the early election provisions.
It seems to me that the Prime Minister, after defeat on a budget, could secure passage of a confidence motion and continue in office (if he thought it was worth being in office with mutinous backbenchers limiting what he could do).