Tjyylli

Madman theory revolves around the idea of appearing more "crazy" than one actually is.

It is actually a signaling game in its simplest form. At least nowadays such games are exemplified with a "crazy" (non-accommodating) player type among other scenarios.

I wonder however: who coined the term "madman theory"? It appears the term gained popularity referring to Nixon's approach to foreign policy, but it was probably a label he didn't coin himself. It sounds more like his critics would have come up with it.

Google ngram finds some uses before Nixon's time as president, apparently:

I don't know if these early uses were with the same meaning though.

This seems to be coming from Haldeman:

While Nixon sought support from the silent majority, diplomats were at work on both North and South Viet- namese. Kissinger continued his talks with Le Duc Tho and indicated that his boss might be mentally unhinged. Nixon himself carefully cultivated the fears that he might go to irrational lengths to get his way. Chief of Staff Haldeman later dubbed this procedure the “madman strategy”: if the North Vietnamese doubted that Nixon felt bound by conventional restraints, they would sign before risking destruction. While Kissinger played upon these fears, Haig returned to Thieu in Saigon with some brutally frank messages. (p303)

Robert D. Schulzinger: "U.S. Diplomacy Since 1900", Oxford University Press: Oxford New York, ⁵2002.

Or

Kimball delved into Nixon’s “madman strategy,” a term that came from chief of staff H.R. Haldeman’s recollection that Nixon thought he could intimidate North Vietnam by making them believe he would escalate the war relentlessly.
_{(Schulzinger "Nixon & Kissinger", p369)}

The latter primarily consisted in the continuation of current ground operations, the escalation of air operations, and what Nixon secretly called “the madman theory” – his version of brinkmanship.
_{(Kimball "Vietnam War", p385)}

Another element in the Nixon-Kissinger strategy for Vietnam, the madman theory, was long neglected in Nixonian historiography. The first public mention of this stratagem of coercion was in White House chief-of-staff H. R. Haldeman’s memoir, The Ends of Power (Haldeman 1978: 82–3). With a few exceptions (Hersh 1983; Isaacson 1992), however, works published before 1998 often ignored, doubted, or rejected Haldeman’s testimony (Greenstein 1983; Hoff 1994; Immerman 1991). Some historians and journalists suggested that Haldeman’s assisting author, Joseph DiMona, was responsible for inserting the story about the madman theory in the memoir and that Haldeman later retracted his association with it. But Haldeman affirmed Nixon’s embrace of the madman theory in a 1990 interview with Walter Isaacson (1992; see also Summers 2000). Haldeman’s comments, the memoirs of other Nixon aides, Nixon’s own semi-private statements, and declassified documents and tapes serve to confirm Nixon’s faith in and implementation of the madman theory and of Kissinger’s collaboration in the project (Kimball 1998, 2004).
_{(Schulzinger, p 386)}

Melvin Small: "A Companion to Richard M. Nixon", Wiley-Blackwell: Chichester, 2011.

This is of course only focussing on Nixon's strategy as that of a madman.
Previous examples of the term not only foreshadow the Vietnam/Cold War use:

Brownson's review, 1845:

We are not disposed to undertake the refutation of this theory, which may be termed the demoniacal, or madman's theory, for none but a madman will attempt to reason a mad man out of his crotchets.

The New York Times Magazine, 1968 (tiny snippet view talking about Asian/American politics; alternative view)

Saturday Review 1963:

A fourth type of accidental war is what has sometimes been called the "madman" theory.

The re-adaptation of this political behaviour as deliberate tactic seems to be even earlier attested:

Paul Alan Lawrence Smith: "Politics and Social Life: An Introduction to Political Behavior", Houghton Mifflin, 1963.

Even more curious might be that after all, this type of game-theory application is only fully appreciated if you read it 'in the Russian original':

In the late fifties, Schelling, Morgenstern, Kissinger, and Ellsberg (as well as others) were part of a circle of strategic thinkers centered at Harvard and RAND who were concerned mainly with the problem of the instability of the nuclear arms race and nuclear deterrence.5 They recognized, analyzed, understood, and criticized the irrationality theory. Although they believed in the efficacy of force in foreign policy, in the late fifties none of these civilian strategists, as far as I can discern, actually advocated a madman strategy of ambiguity, irrationality, uncertainty, unpredictability, or excessive force and ruthlessness as later practiced by Nixon. (Kissinger converted to the madman theory while working for Nixon; Schelling had a falling out with Kissinger over his and Nixon’s conduct of the Vietnam War.)[…]

Even if I am wrong and these “wizards of Armageddon” had indeed advocated madman strategies, and even though they may have been the first game theorists and civilian deterrent strategists to discuss the madman theory, they had not invented or originated it, as some writers have claimed or implied.

I have argued elsewhere that the madman theory is as old as “civilization,” if not humanity itself, and that the madman theory’s inventors and practitioners were political-military decision makers:

The principle of instilling fear by threatening excessive force is as ancient as statecraft, war, and terror. Demanding the return of a hostage, the second millennium bce Hittite King Mursli, for example, issued this warning . . . to the hostage-taker: “I will come and destroy you along with your land.” Over three thousand years later, the principle became an essential component of [strategic bombing and] “atomic diplomacy.” While serving as vice president at the dawn of the nuclear age—the period in which Nixon had come of age as a policymaker and strategist—he learned about the “uncertainty principle,” which was one of the principles that lay at the heart of the atomic “brinkmanship” or “massive retaliation” strategy of President Dwight D. Eisenhower and Secretary of State John Foster Dulles.7

Nixon had also learned from Nikita Khrushchev, who, Nixon claimed, was “the most brilliant world leader I have ever met,” because he nurtured a reputation for rashness and unpredictability that “scared the hell out of people.” 8

Jeffrey Kimball: "Did Thomas C. Schelling Invent the Madman Theory?", History News Network, George Washington University, (2019?)

The above gives a gist of what might be called a zeitgeist.

Who invented the strategy, who named it first, who publicised it so widely that now everyone knows its meaning?

When Nixon coined the name Madman Theory for his version of the principle of threatening and signaling the use of excessive force, he did not have to look far for the title. The words mad, madness, and madman in reference to nuclear policy and strategy were commonplace in the world of the 1950s and 1960s. Critics of a nuclear deterrent strategy based on massive retaliation or flexible response often used the word mad or similar language to describe the international arms regime. These included moviemakers, novelists, and scientists, as well as grassroots activists in the disarmament movement who opposed the nuclear arms race, nuclear testing, and massive retaliation for impoverishing economies, causing cancerous radioactive fallout, and risking global doom.17

Ironically and paradoxically, the practitioners of nuclear deterrence also invoked madness in reference to the system. Eisenhower referred to a US-USSR nuclear war as “insane” in correspondence with Winston Churchill in 1956.18 Admiral Arleigh Burke, chief of naval operations under President Eisenhower and an advocate of “minimal deterrence” based on submarine-launched ballistic missiles, said in 1957 that the United States had to be prepared “to counter any madmen who would resort” to all-out nuclear war.19 Robert McNamara, President Kennedy’s secretary of state, used the word in 1967 in describing the “mad momentum intrinsic to the development of all new nuclear weaponry.”20 On the other side of the Cold War divide, Soviet general secretary Khrushchev regularly claimed that “some madmen in the camp of the imperialists threaten the world with their atomic and hydrogen bombs.… Only a madman would want war.” Yet at other times, he could issue mad warnings of his own: “Should any madman launch an attack on our state or on other Socialist states we would be able literally to wipe the country or countries which attack us off the face of the earth.”21
Reinforcing the meme of madness during the 1960s was “assured destruction” (AD), a strategic doctrine that Secretary McNamara formulated in late 1963 to describe “an actual and credible second-strike capability” (a retaliatory ability to destroy at least 50 percent of the Soviets’ industrial capacity, 30 percent of their population, and 150 of their cities).22 As the term became public, many observers regarded assured destruction as yet another name for mutual deterrence.23 Skeptics, such as journalist Tom Wicker, voiced their objections to the “Strangelovish world of megatonnage and ‘assured destruction’” and the surrealistic, rationally irrational logic of “the terrible balance that renders both Soviet and American nuclear power useless for anything but mutual destruction.”24 By at least 1968, McNamara was publicly referring to assured destruction as “mutual assured destruction,”25 although it was apparently not until May 1971 that Donald G. Brennan, former president of the Hudson Institute and an advocate of missile defense, coined the satiric acronym MAD in New York Times op-eds.

Jeffrey P Kimball & William Burr: "Nixon's Nuclear Specter: The Secret Alert of 1969, Madman Diplomacy, and the Vietnam War", Modern War Studies, University Press of Kansas: Lawrence, 2015.

That would make the theory/strategy itself 'older than dirt', floating around in the 50s with increased attention from cold war strategists. Then Nixon seemed to have adopted that theory in this phrasing to describe his own intentions in the late 60s internally – while others still used this phrase interchangeably to describe real madmen's behaviours and outcomes. But only after Haldeman's memoirs were published the general public took over the current meaning of 'most probably make-believe madmanhood' in international, political relations.

That makes for quite a few cooks –– or 'fathers', 'coining' the term: Schelling, Morgenstern, Kissinger, Ellsberg, Nixon and Haldeman should be regarded as responsible for the current main meaning and understanding of "madman theory".

There's a bit more interesting theorizing in Hersh's book The Price of Power how the theory may have come to the attention of Nixon.

It is highly likely that Kissinger was responsible for Nixon's adoption of the phrase the "the madman theory". In 1959, Ellsberg had present two lectures to Kissinger's Harvard seminar on the conscious political use of irrational military threats. Ellsberg called the theory "The Political Uses of Madness". In essence he described a problem in bargaining theory: what to do when the available threat is so extreme or costly as to make it seem unlikely that a sane and reasonable person would carry it out. On the other hand, since the threat is so extreme, Ellsberg noted, it does not have to be very likely to be used in order to be effective, particularly if the nation posing the threat is making only moderate negotiating demands. Ellsberg postulated that one way of making the threat somehow credible--which, in this case, was all that would be necessary--would be for the person making the threat to appear not to be fully rational. In effect, and application this was implicit in Kissinger's Nuclear Weapons and Foreign Policy, which advocated among other things, a "strategy of ambiguity" in connection with United States use of of tactical nuclear weapons. In his Harvard lecture, Ellsberg cites as his example Hitler's conscious use of his reputation as a madman to win victoires without firing a shot in the Rhineland, Austria and Munich before World War II. As his model of Hitler indicated, Ellsberg regarded such a strategy as reckless and utterly dangerous for the world, but a possibility to be anticipated from opponents because, in connection with a nuclear threat, it might work.

Wikipedia says however that

In Nixon's Vietnam War, Kimball argues that Nixon arrived at the strategy independently, as a result of practical experience and observation of Dwight D. Eisenhower's handling of the Korean War.

YMMV. There's more than one plausible route how Nixon came up with the term and/or the idea.

P.S.: One more interesting thing: Lang Lang's answer (quoting Kimball) says Nixon also learned from Khrushchev. But, interestingly, Khrushchev is also briefly discussed as an example in the final page of Ellsberg's 1959 lecture:

For if Hitler was mad, his theory was not. "The difference between Munich and now," Khrushchev said recently (March, 1959) "is that I am not Adolph Hitler." (He didn't feel a need to distinguish himself from Chamberlain.)