Ultra (cryptography)

This article is about WWII intelligence material codenamed 'Ultra'. For other usages, see Ultra (disambiguation)

Ultra (sometimes capitalised ULTRA) was the name used by the British for intelligence resulting from decrypts of German communications in World War II (WWII), primarily those protected by the various Enigma machines. Most of this traffic was military, but the Nazi party SD organization and the Abwehr also used Enigmas of one variant or another, as did the railways.

Until the name Ultra was finally adopted there were several names for intelligence from this source, including Boniface. The corresponding name used by the Americans for analogous intelligence from Japanese decrypts in WWII was Magic.

Sources and history

Encrypted messages

The Ultra material largely came from the German cypher traffic. These messages were generated by (several variants of) an electro-mechanical rotor machine, the Enigma, widely thought to be unbreakable in practice in the 1920s when a variant of the commercial Model D was first used by the German Navy. The German Army, Navy, Air Force, Nazi party, Gestapo, and diplomats all used Enigma machines, but there were several variants (eg, the Abwehr used a four-rotor machine without a plugboard, and Naval Enigma used different key management than the Army or Air Force making its traffic far more difficult to cryptanalyze). Each variant required different cryptanalytic treatment. The commercial versions were not so secure. Dilly Knox, of GC&CS, is said to have broken it during the 1920s.

Breaking the code

Main article: Cryptanalysis of the Enigma

There are several conflicting stories of how the Allies made the initial breaks into Enigma (see Enigma for more). The most fundamental break into the Enigmas was made in Poland in 1932 by Marian Rejewski, who for the first time used theoretical mathematics to break into the Engima system. Together with his colleagues at the Biuro Syszfrow, he went on to develop practical methods of decrypting some of the Enigma traffic. The Poles designed and built working replicas of the Enigmas as well as additional equipment which helped in finding the keys needed for decryption (the cyclometer, the bomba, perforated sheets). By 1938 much German Enigma traffic was being routinely deciphered by the Poles, but a change in German operations (greater rotor choice), and the impending war led the Poles to share their work on Enigma with France and England. This happened during the famous meeting in Pyry near Warsaw in 1939. Since neither had made any practical progress on breaking Enigma traffic, this was a major boost. On the other hand, physical seizure of crypto material was very significant for some Enigma variants, particularly German Naval Enigma.

With this massive Polish assistance, the British and French began to work on German Enigma traffic themselves. French work, and some of the Polish cryptographers, ended up at PC Bruno, which was closed when the Gestapo seemed to be moving in. Early in 1939 Britain's secret service had installed its Government Code and Cipher School (GC&CS) at Bletchley Park, 50 miles (80 km) north of London, to work on enemy message traffic. They also set up a large interception network to collect enciphered traffic for the code breakers at Bletchley. Eventually, there was a very large organization controlling the distribution of the resulting – secret – decrypted information, which was eventually called Ultra. Strict rules were established to restrict the number of people who knew about the existence of Ultra in the hope of ensuring that nothing (eg, leaks, actions) would alert the Axis powers that the Allies were reading their message traffic. Early in the war the product from Bletchley Park was codenamed 'Boniface' to give the impression to the uninitiated that the source was a secret agent. Such was the secrecy surrounding reports from 'Boniface' that 'his' reports were taken directly to Winston Churchill in a locked box to which the Prime Minister personally held the key. The information so produced was eventually termed 'Ultra'.

The group at Bletchley Park working on breaking messages was a mix of crossword enthusiasts, chess mavens, mathematicians, and pioneer computer scientists. This last group overlapped some of the others and included Alan Turing, one of the fathers of modern computing, and Max Newman the lead designer of the world's first electronic programmable digital computer, the Colossus. By 1943, large portions of the intercepted signals (more than 2,000 daily at the maximum) were routinely read, including some from Hitler himself. Such information enabled the Allies to routinely develop an accurate picture of enemy plans and orders of battle, and when used sensibly were of great value in forming the basis of Allied strategic and tactical war plans.

Methods of attack

British attacks on the Enigma(s) were similar in concept to the original Polish methods, but based on different specifics. First, the German Army had changed their practices in 1939 (more rotors, different 'message setting', etc), so the Polish techniques no longer worked directly. Second, the German Navy had always used more secure practices, and no one had broken any of their traffic.

One new attack relied on the fact that the reflector (a patented feature of the Enigma machines) guaranteed that no letter could be enciphered as itself, so an A could not be sent as an A. Another technique counted on various common German phrases, like "Heil Hitler" or "please respond", which were found to likely be in this or that plaintext; successful guesses as to the plaintext were known at Bletchley as cribs. With a probable plaintext fragment and the knowledge that no letter could be enciphered as itself, it wasn't uncommon that a corresponding ciphertext fragment could be identified. This often provided a large hint as to the message settings, much in the same way the message setting codes had done for the Poles before the War started.

German operators themselves also gave the cryptanalysts immense help on a number of occasions. In one instance an operator was asked to send a test message, so he simply hit the T key repeatedly and sent it. A British analyst received a long message without a single T in it from the interceptor stations, and immediately realised what had happened. In other cases, Enigma operators would constantly use the same settings for their message codes, often their own initials or those of their girlfriends (one was apparently named Cillie, so Bletchley Park named such hints 'cillies'). Analysts were set to finding these messages in the sea of intercepted traffic every day, which winnowed out enough possibilities to allow Bletchley to use the original Polish techniques (more or less) to find the initial settings for the day from those that remained. Other German operators used "form letters" for daily reports, notably weather reports, so the same crib could be used every day.

Had the Germans ever replaced every rotor at the same time, it is possible that the British would not have been able to break back into the system. And had German operational practice been better, things would have been much more difficult. However, both because of the expense and because of the difficulty of getting all those new rotors to all the necessary ships and units, it was never done. Instead the Germans simply added new rotors to the mix every so often, allowing the settings of the newest ones to be deciphered after a short period.

Use of Ultra

Usable Ultra information came too late to be of great help during the Battle of Britain.

The Allies were seriously concerned to conceal from the Axis command that they had made any breaks into any of the Enigma traffic. This was taken to the extreme that, for instance, though they had intercepted and knew the whereabouts of U-boats lying in wait in mid-Atlantic, they were not generally hunted unless a 'cover story' could be arranged -- often a search plane was 'fortunate enough' to sight the U-boat, thus explaining an attack. Ultra information was used to attack and sink many of the Afrika Korps supply ships travelling to North Africa, but as in the North Atlantic, every time it was used, some 'innocent' explanation had to be provided; scout planes would often be sent on otherwise unnecessary missions to ensure they were seen by the German military. The British were, it is said, more careful about this than the Americans, and the difference in caution on this point was a source of friction between them.

The distribution of Ultra information to the Allied commanders and units in the field involved considerable risk of discovery by the Germans, and great care was taken to control both the information and knowledge of how it was obtained. Liaison officers were appointed for each field command to manage and control dissemination.

In the summer of 1940, British cryptanalysts, who were successfully breaking the German Air Force Enigma cypher variants, were able to give Churchill information about the issuance of maps of England and Ireland to the Sealion invasion forces.

From the beginning, the Naval version of Enigma used a greater variety of rotors than did the Army or Air Force versions, as well as various operational methods that made it much more secure than other Enigma variants. There was no hint at all to the initial settings for the machines, and there was little probable plaintext to use either. Different, and far more difficult methods had to be used to break into Naval Enigma traffic, and with the U-boats running freely in the Atlantic after the Fall of France, a more direct approach recommended itself.

On 7 May 1941 the Royal Navy deliberately captured a German weather ship, together with cipher equipment and codes, and 2 days later U-110 was captured, together with an Enigma machine, code book, operation manual and other information enabling the submarine message traffic to be broken until the end of June. And they did it again shortly afterwards.

In addition to U-110, Naval Enigma machines or settings books were captured from a total of 7 U-boats and 8 German surface ships, including U-boats U-505 (1944) and U-559 (1942), as well as from a number of German weather-reporting boats, from some converted trawlers, a small vessel (the Krebs) captured during a raid in the Lofoten Islands off Norway, and so on. Several other more imaginative techniques were dreamed up, including Ian Fleming's James Bondian suggestion to "crash" captured German bombers into the sea near German shipping, hoping to be "rescued" by the crew, which would then be taken captive by the Commandos hiding in the plane and the crypto material captured intact.

In other cases the Allies forced the Germans to provide them with a crib. To do this they would drop mines (or take some other action), and then listen for messages thus provoked. In the case of mining this or that channel, they expected the word "Minen" would be in some of them. This technique was called gardening at Bletchley.

Even these brief periods were enough to have dramatic effects on the progress of the War. Charting the amount of traffic decoded against the British shipping losses for that month shows a strong pattern of increased loss when Naval Enigma was blacked out, and vice versa. But, by 1943, so much traffic had been decrypted that the code breakers had an excellent understanding of the messages coming from various locations and times. For instance, a brief message sent from the west at 6am was likely to be sent by a weather reporting boat in the Atlantic, and that meant the message would almost certainly contain these cribs, and similarly for other traffic. From this point on, Naval Enigma messages were being read constantly, even after changes to the ground settings.

However, the new tricks only reduced the number of possible settings for a message. The number remaining was still huge, and due to the new rotors the Germans had added from time to time, that number was much larger than the Poles had been left with. In order to solve this problem the Allies, especially the US, "went industrial", and produced much larger versions of the Polish bomba that could test thousands of possible key settings very rapidly indeed.

Some Germans had suspicions that all was not right with Enigma. Karl Dönitz received reports of "impossible" encounters between U-boats and enemy vessels which made him suspect some compromise of his communications. In one instance, three U-boats met at a tiny island in the Caribbean, and a British destroyer promptly showed up. They all escaped and reported what had happened. Doenitz immediately asked for a review of Enigma's security. The analysis suggested that the signals problem, if there was one, wasn't due to the Enigma itself. Dönitz had the settings book changed anyway, blacking out Bletchley Park for a period. However the evidence was never enough to truly convince him that Naval Enigma was being read by the Allies. The more so, since his counterintelligence B-Dienst group, who had partially broken Royal Navy traffic (including its convoy codes during the early part of the War), supplied enough information to support the idea that the Allies were unable to read Naval Enigma. Coincidentally, German success in this respect almost exactly matched in time an Allied blackout from Naval Enigma.

In 1941 British intelligence learned that the German Navy was about to introduce M4, a new version of Enigma with 4 wheels rather than 3. Fortunately, for the Allies, in December, a U boat mistakenly transmitted a message using the four rotor machine before it was due to be implemented. Realising the error, they re-transmitted the same message using the 3 rotor Enigma, giving the British sufficient clues to break the new machine very shortly after it became operational on February 1 1942. The U-boat network which used the four rotor machine was known as Triton, codenamed Shark by the Allies. Its traffic was routinely readable.

It is commonly claimed that the breaks into Naval enigma resulted in the war being a year shorter, but given its effects on the Battle of the Atlantic (1940) alone, that might be an underestimate.

A break of some messages (not in German Enigma, however) led to the defeat of the Italian Navy at Capa Matapan, and was preceded by another 'fortunate' search plane sighting. British Admiral Cunningham also did some fancy footwork at a hotel in Egypt to prevent Axis agents from taking note of his movements and deducing that some major operation was planned. Ultra information was of considerable assistance to the British at El Alamein in Western Egypt in the long running battle with the Afrika Korps under Rommel. Intelligence from signals between Adolf Hitler and General Günther von Kluge was of considerable help during the campaign in France just after the Allied landings on D-Day, particularly in regard to estimations of when German reserves would be committed to battle.

By 1945 almost all German Enigma traffic (Wehrmacht, Navy, Luftwaffe, Abwehr, SD, etc.) could be decoded within a day or two, yet the Germans remained confident of its security. Had they been better informed, they simply could have, and surely would have, changed systems, forcing Allied code-breakers to start over. The Germans considered Enigma traffic so secure that they openly discussed their plans and movements, handing the Allies a huge amount of very useful information. However, Ultra information was also misused or ignored at times. For instance, Rommel's intentions just prior to the Battle of the Kasserine Pass in North Africa had been suggested by Ultra, but this was not taken into account by the Americans. Likewise, Ultra traffic suggested an attack in the Ardennes in 1944, but the Battle of the Bulge was a surprise to the Allies because the information was disregarded.

After the War, the American TICOM project teams found and detained a considerable number of German crypto personnel. Among the things they learned was that German cryptographers, at least, understood very well that Enigma messages might be read; they knew Enigma was not unbreakable. They just found it impossible to imagine anyone going to the immense effort required. (See Bamford's Body of Secrets in regard to the TICOM missions immediately after the War.)

A most intriguing alleged, and still open, use of Ultra information may have been in the Lucy spy ring. This was an extremely well informed, and rapidly responsive, ring which was able to get information 'directly from the German General Staff Headquarters' -- often on specific request. The allegation is that it was, in major part, a way for the British to get Ultra information to the Soviets in a way which appeared to have come from highly placed espionage, and not from cryptanalysis of German radio traffic. The Lucy ring was operated by, apparently, one man, Rudolf Roessler, and was initially treated with considerable suspicion by the Soviets when it began to operate. The information it provided was accurate and timely, and Soviet agents in Switzerland (including Alexander Rado, the director) eventually took it quite seriously.

Magic and Purple in Europe

In the Pacific Theatre, the Japanese cypher machine called "Purple" was unrelated to the Enigmas, but was used for the highest level Japanese diplomatic traffic. It was also cracked, though, by the US Army's Signal Intelligence Service. Some of the Purple traffic was very useful elsewhere during the War, for instance the very detailed reports by the Japanese Ambassador to Germany which were sent to Japan using the Purple machine. Some of these included reviews of German strategy and intentions, some were reports of direct inspections (in one case of the Normandy beach defenses!) by the Ambassador, and some were reports of long interviews with Hitler.

The Japanese are said to have obtained an Enigma machine as early as 1937, although whether they were given it by their German allies, or bought a commercial version which, except for the plugboard and the actual rotor wirings, was essentially the German Army / Air Force machine, is disputed.

Public disclosure of Ultra

For 29 years after the war the existence of Ultra remained an official British (and American) secret. Since it was British and, later, American work which had been most significant, this meant that the significance of Enigma decrypts in the War remained unknown. Discussion by either the Polish or French of Enigma breaks early on would have been uninformed for most of the War. Nevertheless, the first public disclosure of breaks into Enigma traffic was made by French Intelligence officer Gustave Bertrand in the book he published in 1973. With that release, pressure to discuss the rest of the Enigma/Ultra story (mostly a British and later American one) mounted.

The ban was not lifted until 1974, the year that a key participant on the distribution side of the Ultra project, Frederick William Winterbotham, published The Ultra Secret. Wintherbotham's book is very interesting, but is in error on many points. He worked on the operation to distribute Ultra to end consumers and, based on the evidence of his book, did not understand much about cryptography. Peter Calvocorressi's book (Top Secret Ultra) is better written and more responsible. He was involved in Bletchley Park's intelligence analysis of decrypted traffic, working between the codebreakers and Winterbotham's distribution operation.

The official history of British intelligence operations during WWII was published in five volumes from 1979 to 1988. It was mostly written by Sir Harry Hinsley, with one volume by Michael Howard. There is also a one volume collection of recollections, edited by Hinsley.

After the War; public disclosure

That Enigma had been broken during the War remained secret until the late 1960s. The important contributions to the War effort of a great many people remained unknown, and they were unable to share in the glory of what is likely one of the chief reasons the Allies won the war as quickly as they did.

After the war ended, the British and Americans sold surplus Enigmas and Enigma-like machines to many countries around the world, who remained convinced of the security of this remarkable cipher machine. Their traffic was not so secure as they believed, which is, of course, one reason the British and Americans made the machines available. Switzerland even developed its own version of the Enigma, the NEMA and used it for decades (at least into the late 70s).

Some information did get out however. In 1967, David Kahn published The Codebreakers, which described the capture of a Naval Enigma machine from U-505 in 1945. He went on to mention, somewhat in passing, that Enigma messages were already being read by that time, requiring 'machines that filled several buildings'. By 1970, newer computer-based ciphers were becoming popular as the world increasingly turned to computerised communications, and the usefulness of Enigma copies (and rotor machines generally) rapidly decreased. It was decided at this point to "let the cat out of the bag", and reports about some of Bletchley Park's operations were permitted in 1974.

The National Security Agency retired the last of its rotor-based encryption systems in the 1980s.

Difficulties with some disclosures

Many accounts of the Enigma story, and of other World War II crypto happenings, have been published. Several are unreliable in many respects about WWII cryptography. This can be traced to several causes:

First, not all of the authors were in a position to know. Several books have been published by those on the Ultra distribution side at Bletchley Park, but work there was very seriously compartmentalised, making it difficult to credit some episodes when they are due only to such a source. The story about Churchill deliberately not interfering with a Luftwaffe bombing of Coventry which was known through Enigma decrypts is one such. Peter Calvocoressi's book, Top Secret Ultra contains a sounder account of the episode.

Second, the cryptanalytic work was tricky and quite technical. It requires someone with a considerable understanding of cryptanalysis, and of Enigma, to adequately comprehend -- or explain -- how either worked.

Third, documents have been 'lost' in secret archives. Those not actually lost have taken decades to be released to the public, and some are, presumably, still to be released. In any case, none of them were originally written, nor made available later, with historical clarity in mind; considerable perspective is required to make reasonable use of them.

Fourth, governments have chosen to keep secret or release information to serve their own purposes, not historical accuracy or completeness.

Fifth, several authors have had agendas which took precedence over accuracy in their reports. At least one incident is known of whole-cloth fabrication regarding British cryptanalytic progress on a particular World War II Japanese Navy crypto system. The account was claimed to have been written from the unpublished memoirs of an Australian cryptanalyst, but substantive parts of the published version appear to have been simply invented.

Sixth, many writers have not done their research. The exact fate of the Enigma spy "Asch" was not publicly known till Hugh Sebag-Montefiore tracked his daughter down, circa 1999. Her account can be found in his book.

As with other history, but more than for most, the history of cryptography, especially its recent history should be read carefully, due to its complexity and to possibly confusing or misleading agendas.

External links

Literature on ULTRA