It started at March 4 because it Washington that long to get to New York by horse, after being officially informed. By the late 1800's, the technology was such that anyone elected president could easily get sworn by January 20, even in wartime.
Washington was not inaugarated until June of 1789.
The Constitutional Convention was acting extra-legally in proposing the Constitution (the Articles of Confederation were still in force, and the states were parties to it). To bypass this impediment, the convention sent the proposed Constitution to the Congress (meeting in New York City) and requested that they pass it on to the
people of the states for their consideration in convention, and requested that Congress monintor the ratification count, and establish the calendar for the Constitution to come into effect.
Once the 9th ratification occured, Congress began considering a schedule. But they wanted to wait for Virginia and New York and Virginia to ratify, and they also could not decide on a capital. Eventually, they came to an agreement and set out the following schedule:
1st Wednesday in January, Presidential electors chosen.
1st Wednesday in February, electors meet in their respective states and cast their vote.
1st Wednesday in March, Congress meets.
The 1st Wednesday in March, 1789 was March 4. It took some time for quorums to be established in the House and Senate, so they could take official action, including their role of counting the electoral votes (and if necessary choosing the President and/or Vice President). Eventually, they reached a quorum, and counted the electoral votes, and notified Washington and Adams of their election. Adams got to New York several weeks before Washington, and was sworn in as Vice President (he worried before Washington's inauguration whether he should be introduced to President (of the US) Washington as the President (of the Senate) Adams or as Vice President (of the US) Adams).
Later, Congress decided when the next terms would begin (and the initial terms would begin). They decided that since the Constitution specified terms of 2, 4, and 6 years, it should be interpreted literally, and from then on Congressional and Presidential terms began coincidentally on March 4.
March 4 was not a particularly good meeting date since the roads would be muddy, and if a session lasted very long it would interfere with planting, and there was greater risk of yellow fever and malaria, especially after the capital was moved to the South. So the practice was established with meeting in December (the Constitution specifies that Congress meet each year, and specifies December if they can not decide on a meeting time, but later Congresses chose December).
So the actual sessions of Congress were held at the end of years of the term. The first session was typically held from December of odd years into the late spring/early summer of the the even years, while the second session was held from December of even years to March 3 of odd years. When a new President was inaugarated, the Senate would meet in a special session beginning on March 4 in order to confirm the new President's cabinet officers. From mid-19th century onward, there was usually a Christmas recess which would occur a couple of weeks after the session began.
Because of this schedule, some states did not elect representatives until the middle of the odd years, already into the congressional term. On occasion, some states were not represented in special sessions because they hadn't elected their representatives.
Only after the Civil War was a common day for congressional elections chosen. The practice of November elections in even years was largely based on Northern custom. Southern states had more typically held elections in the spring of odd years.
With a November election date, the entire second session of Congress was conducted by representatives who may not have been elected for the subsequent term, and in the case of sessions that began in December had been elected more than a year prior to their actually beginning service.
In the early 20th Century, Congress often used a 3-session calendar. The first session would begin in spring of the odd years (usually in April and May) and run into later summer early fall. The second session would begin in December of the odd year and run into the summer of the even year. The third session would run from December of the even year up to March 3 of the odd year (end of term). The added session was actually the first, but the new 1st and 2nd sessions were closer to the schedule we are familiar with, and the 3rd session was more apparently seen as a lame duck session.
After the first presidential election, it was the outgoing Congress that counted the electoral votes, and in 1801 and 1825, who chose the President. Since it was desirable for there to be a President elected before their term began, and the terms of Congress and the President were coincident, the outgoing Congress counted the electoral votes and if necessary chose the President.
The 20th Amendment permitted Congress to begin their first session in the winter, soon after the election and avoid meeting over Christmas, and also provided an opportunity for the new Congress to count the electoral votes and choose the President or Vice President if necessary. In addition, it eliminated the need for a Senate special session to consider presidential nominees.
