In Iraq and Afghanistan, and any conflict where there isn't a declared "war", what is the line between legal and illegal killing on the part of soldiers?
The same as in a declared war. The First Geneva Convention of 1949 says:
In addition to the provisions which shall be implemented in peacetime, the present Convention shall apply to all cases of declared war or of any other armed conflict which may arise between two or more of the High Contracting Parties, even if the state of war is not recognized by one of them.
The Convention shall also apply to all cases of partial or total occupation of the territory of a High Contracting Party, even if the said occupation meets with no armed resistance.
Although one of the Powers in conflict may not be a party to the present Convention, the Powers who are parties thereto shall remain bound by it in their mutual relations. They shall furthermore be bound by the Convention in relation to the said Power, if the latter accepts and applies the provisions thereof.Which is why the Japanese had people hanged for war crimes, despite not signing the 1911 Convention (which had this too, I believe).
The UCMJ (Uniform Code of Military Justice) says this on the subject.
918. ART. 118. MURDER
Any person subject to this chapter whom without justification or excuse, unlawfully kills a human being, when he- -
(1) has a premeditated design to kill;
(2) intends to kill or inflict great bodily harm;
(3) is engaged in an act which is inherently dangerous to others and evinces a wanton disregard of human life; or
(4) is engaged in the perpetration or attempted perpetration of burglary, sodomy, rape, robbery, or aggravated arson;
is guilty of murder, and shall suffer such punishment as a court-martial may direct, except that if found guilty under clause (1) or (4), he shall suffer death or imprisonment for life as a court-martial may direct.
919. ART. 119. MANSLAUGHTER
(a) Any person subject to this chapter who, with an intent to kill or inflict great bodily harm, unlawfully kills a human being in the heat of sudden passion caused by adequate provocation is guilty of voluntary manslaughter and shall be punished as a court- martial may direct.
(b) Any person subject to this chapter who, without an intent to kill or inflict great bodily harm, unlawfully kills a human being--
(1) by culpable negligence; or
(2) while perpetrating or attempting to perpetrate an offense, other than those named in clause (4) of section 918 of this title (article 118), directly affecting the person;
is guilty of involuntary manslaughter and shall be punished as a court-martial may direct.
That doesn't make any distinction and applies in all circumstances to military personnel (and enemy POWs)
