I think OP was talking about winning the most seats (regardless of whether they form a government or not) but getting less votes than another party. In order for this to happen, you basically HAVE to have FPTP, or at least mixed member PR with some of the members elected via FPTP.
Not really. Malapportionment can produce that type of results in PR systems. I can provide you several examples of regional elections held in Spain (PR-D'Hondt in all cases).
Basque Country, 1986Basque Nationalis Party (PNV) gets 23.71% of the vote and wins 17 seats; Basque Socialists (PSE-EE) come second getting 22.05% and winning 19 seats.
Reason: Every one of the three provinces returns 25 seats to the Basque Parliament regardless of population. The population of Biscay is nearly 4 times higher than the population of Álava, and it happens that is in Biscay where the PNV is traditionally stronger.
PNV and PSE formed a coalition government with a nationalist premier.
Catalonia 1999 and 20031999: Pasqual Maragall, of the Socialist's Party of Catalonia (PSC), challenges patriarch Jordi Pujol, of the Convergence and Union (CiU) nationalist federation.
The alliance led by PSC gets 37.85% of the popular vote and wins 52 seats; the CiU gets 37.7% and wins 56 seats.
Jordi Pujol remains as premier propped up by the People's Party.
2003: Pasqual Maragall runs again for the PSC, whereas Jordi Pujol is replaced by Artur Mas in CiU.
PSC gets 31.16% of the vote and wins 42 seats; CiU gets 30.94% and wins 46.
PSC, ERC and ICV formed a coalition government and Pasqual Maragall is elected premier.
Reason: Inner Catalan provinces (Girona, Lleida and Tarragona) are overrepresented in the Parliament of Catalonia. The Barcelona province has nearly 3/4 of the population, but only returns 63% of the seats (85/135). PSC was traditionally stronger than CiU in Barcelona, whereas the inner provinces use to lean nationalist.
However, the more surrealistic examples of results distorted by malapportionment can be found in the Canarian regional elections. I'll put a sample, although in this case the result was a draw in terms of parliamentary seats.
Canarian parliamentary election, 2011:People's Party (PP) gets 31.84% winning 21 seats; the Canary Coalition (CC) gets 24.89% and wins 21 seats.
CC and PSOE deal a coalition government.
Reason: the allocation of seats in the Canary Islands Parliament follows a principle called "triple parity", with the result of a huge disproportion in the 'cost' of a seat between the different electoral districts (in this case islands instead of provinces). In other words, a seat in the most populated island represents about 20 times more votes than a seat in the less populated.
