Coldspur

[A warning. This is a very demanding report – definitely ‘Advanced’. At almost 28,000 words it runs about twice the size of my regular bulletins. I believe it will be of interest only to dedicated intelligence aficionados and to readers who have been exposed to some degree to the VENONA project, and who may have been as puzzled as I was in trying to make sense of it all.]

Contents:

Introduction

The Organization of the Transcripts

Published Books

Academic Articles

Intermediate Summary

Factors in Decryption

Soviet Reactions

Why No Progress?

Conclusions

*          *          *          *          *          *          *          *          *          *          *          *          *         

Introduction:

The time dimensions, and major milestones, of the VENONA project are confusing, the official accounts are unreliable, and the archives – American, British and Russian – (mainly) unyielding. The project itself was a massive USA/UK undertaking to decrypt Soviet diplomatic traffic at the end of World War II, and it led to the unmasking of several Soviet spies. Yet the publications that describe the project are often misleading and contradictory, citing other works that do not offer adequate references. I detect a mixture of deliberate distortion of the facts by some insiders, in an attempt to confuse, and a tendency by some historians who have written about it to engage in bluff and guesswork. The lack of accuracy and precision evident in the various accounts is particularly distressing since they address a discipline that demands those qualities. The VENONA messages themselves are hopelessly scattered over a variety of different platforms, requiring herculean efforts by any analyst to weave them into a coherent shape. The overall picture is thus obscure. Even seventy-five years later, for some reason the methods used in decrypting KGB and GRU traffic between Western Embassies and Moscow remain highly confidential for GCHQ and for the NSA (the National Security Agency, which replaced the ASA in 1952). My frustration at not being able to find a comprehensive and coherent account of the whole VENONA project, complemented by the deterioration in the availability of on-line transcriptions, led me to undertake this study.

In this report, I set out to investigate the following questions:

• How are the various transcripts of VENONA traffic organized? (or disorganized?)
• How do the published accounts of the VENONA project contribute to an authentic story?
• What special factors contributed to, or hindered, the decryption of Soviet traffic?
• What were the Soviet reactions? Why were they so sluggish?
• Why has no further progress on VENONA messages been made since 1980?
• Why does there continue to be such secrecy over the programme?

I shall not pretend that I can provide answers to all these questions, as I do not have access to all the relevant research, and much of the evidence desired may not exist, or it may have been permanently buried.

First, a reminder of what VENONA was. (See also my report from December 2025, of which this paragraph is an excerpt.)

VENONA was the decryption project initiated by US Army Signal Security Agency and GC&CS analysts, starting in the nineteen-forties, and continuing occasionally until 1980. It exploited careless procedures by the management of the GRU and the KGB (a useful generic term for the various guises in which Soviet foreign counter-intelligence took shape from the nineteen-thirties until the nineteen-eighties *), notably the re-use of One-Time Pads (OTPs). Such devices, which translate already encoded messages into numeric strings practically impossible to decipher when used properly, by their definition should never be re-used if message security is to be maintained. Violations of procedure enabled the ASA and GC&CS, however, to break hundreds of messages, enabling the FBI and MI5 to identify many Soviet agents, including Klaus Fuchs and Donald Maclean. The coded names of that pair appeared in message traffic, and their identity was confirmed by factual details concerning their lives and movements.

[* In common with many other writers and historians, I have occasionally generalized here the multiple forms that Soviet foreign counterintelligence and espionage took from 1917 (Cheka, GPU, OGPU, NKVD, NKGB, GUGB, MGB, KI, MVD) to ‘KGB’, which was in fact founded only after Stalin’s death, in 1954. In quoting from other authors, and in direct analysis of what they wrote, I have left references as they stand.]

An important aspect of the project was the method of collection of Soviet messages. In many cases (e.g. in the United States and in Australia) where long-distance wireless communication was difficult or impossible, telegrams were written up and submitted to commercial cable companies for onward transmission. Legally, copies of all traffic were kept, and handed over to the local authorities, with the identical process occurring with incoming messages. The traffic might thereafter be routed at some stage through a wireless network, but the messages were still considered ‘cables’. In Europe (e.g. in the United Kingdom and in Sweden) wireless communications to and from the Soviet Embassy were intercepted and stored for possible later analysis. I thus refer to the generic sets of communication between stations as ‘traffic’, irrespective of the medium, to the logical connection between two stations as a ‘channel’, and to a synchronous communication by wireless as a ‘link’, indicating the direction by listing the initiating station first, as in (for example) ‘Canberra-Moscow’ or ‘Moscow-Canberra’. The term ‘channel’ is also frequently used to differentiate the systems, or lines of business, maintained by Soviet legations and embassies.

As I compiled this study, I maintained a particular interest in the Australia KGB traffic, i.e. that between Canberra and Moscow, for the following reasons:

1. It is one of the most substantial sets, reputedly 329 pages reflecting traffic from 1943 to 1948.
2. It was one of the last to be released, in Release 5.3 of 1996.
3. It has attracted much less attention from the historians than the US and British traffic.
4. It is unique in that it continued until 1948, as a result of the Soviets’ failure to replace their faulty one-time pads for this circuit.
5. The incompetence of the Soviets ironically contributed to the mythology of ‘Black Friday’.
6. It contains much important traffic, the decryption of which contributed to successes elsewhere.
7. Its contents are scattered among various sources, and much seems to have been withheld.
8. It has important connections to the defection of the Petrovs, and their knowledge of cipher techniques.

1. How are the various transcripts of VENONA traffic organized? (or disorganized?)

The VENONA traffic has multiple dimensions, namely the line of business, the geography, and the chronology of its decryption. The cryptanalysts at Arlington Hall in Virginia, the former girls’ school where the ASA (Army Security Agency) was headquartered when it in 1945 replaced the SSA (Signal Security Agency), fairly quickly worked out that there were five distinct systems being deployed in the intercepted traffic: i) trade, ii) diplomatic, iii) KGB (generic for Soviet foreign intelligence), iv) GRU (Soviet military intelligence), and v) GRU naval. The bulk of the messages successfully analyzed were from system 3, with 4 falling some way behind. The countries involved ranged from the USA to Australia, from Canada to Colombia, from Great Britain to Sweden, and many more. Over 3000 individual messages were recorded as undergoing (at least partial) decryption – a small amount of the total traffic exchanged by Soviet stations. As Harvey Klehr writes: “The earliest cables dated from 1941 and the latest to 1950 [sic! – see below for contrary indications].  Most were from the period 1943 to 1945.  The project started in 1943, decoded its first cable in 1946, and continued until NSA shut down the project in 1980 when it judged the remaining cables vulnerable to decryption, almost all from the early 1940s, were too old to be of any current intelligence interest.  While cables from Soviet stations in sixteen nations were deciphered, the great majority were between Moscow and its stations in the United States.”

One would expect that a comprehensive register of all messages that had been decrypted to reveal something of use had been compiled, and that it would be easily available. The traditional repository has been owned and delivered by the NSA. It maintained what I remember as a very useful site, available on-line, but it is now defunct. Nevertheless, several other on-line organizations dedicated to VENONA (such as the Wilson Center at Stanford University) still maintain the obsolete NSA url, as, of course, do many publications that appeared up until a few years ago. Instead, the NSA now offers a reformatted guide and registry, at https://www.nsa.gov/Helpful-Links/NSA-FOIA/Declassification-Transparency-Initiatives/Historical-Releases/Venona/  that are, to all intents and purposes, useless. The site  introduces its Index to document images by stating: “The first of six public releases of translated VENONA messages was made in July 1995 and included 49 messages about the Soviets’ efforts to gain information on the U.S. atomic bomb research and the Manhattan Project. Over the course of five more releases, all of the approximately 3,000 VENONA translations were made public.”

The structure that follows indicates that 100 pages of items, with each page containing fifteen documents, are viewable, equating to 1,500 documents rather than the 3,000 advertised. Yet the documents have been sorted alphabetically from all sources, driven by the metadata describing what the relevant cable is about, with the character ‘”’ appearing first. Page 100 contains the metadata ‘Moscow’: thus the registry is missing half of the entries, with titles headed ‘N’ to ‘Z’ absent. The Search facility does not work. There is no ability to select a certain channel (e.g. New York-Moscow) from the listings. One can only plod through every individual item looking for pieces of relevance. In late December 2025 I submitted an on-line query to NSA requesting an explanation of why it has formatted the transcripts this way, and seeking advice on how to use the material more effectively. I never received an acknowledgment, let alone a reply.

The Wilson Center concentrates on KGB traffic, and enhances it offerings with extracts from the Vassiliev Notebooks, and the Concordance that accompanies them. (See https://www.wilsoncenter.org/article/venona-project-and-vassiliev-notebooks-index-and-concordance )  The site was developed by John Earl Haynes, and appears to have been last updated in 2013. Hence it also offers the obsolete NSA url when inviting readers to track down messages by date. It very sensibly organizes the traffic in separate PDFs that each represent a communication channel, and the line of business (e.g. New York KGB, London GRU). It thus offers a Canberra KGB PDF, but it is obviously incomplete, since the cables presented constitute only a fraction of the two hundred or so that other sources state exist. There are none from the critical year of 1948, for example: no explanation is offered, but it may simply be because the Students at the Mercyhurst College for Intelligence Studies (who transcribed them) ran out of time on their project. They are presented in rough chronological order. I have not counted the number of messages from other channels.

Through contacting Mr. Haynes, I was able to obtain a set of all the Canberra-Moscow traffic that he possessed (and I thank him for sending the files to me). It contains several items from 1948, but all are from Moscow to Canberra. I do not know why no cables in the other direction appear: were they not decryptable for some reason? Have they been censored? Unfortunately, the set has also been rather haphazardly assembled. Over two hundred messages are listed, but many are re-issues of earlier messages that have undergone further decryption. While numbered, they are not in chronological order. The dates on the cables produced are frequently wrong (primarily the year given).

My former on-line colleague, the late Denis Lenihan, produced a very useful catalogue and analysis of the Australian traffic, available at https://www.academia.edu/36543821/AUSTRALIAS_KGB_NETWORK_1944_1950_A_NOTE_ON_AUSTRALIAN_VENONA_pages_pdf?email_work_card=view-paper (probably requiring a subscription). Yet the messages he lists appear to be based on the defunct NSA site, and they do not correspond to the Haynes archive described above. Moreover, Lenihan’s listings are selective: he states that he has chosen them based on their relationship to espionage, but that means he overlooks some critical items, such as the June 2, 1948, cable about codes. He does offer, however, some very useful insights about the GCHQ records, having inspected them at Kew, and he makes a few sharp observations, such as pointing out that no Canberra to Moscow traffic after May 1946 was decrypted at all – a puzzling phenomenon that deserves some closer analysis.

The Internet Archive, at The VENONA Files : Free Texts : Free Download, Borrow and Streaming : Internet Archive boasts 3,262 results, and offers images of them all. (The higher number is due to the fact that the contents are not exclusively messages, but include various annotations and commentary.) It offers ‘Search’ capabilities only through metadata, so identifying relevant documents requires a knowledge of how the decrypted message is described in the header. One cannot search for a particular channel, for example. Yet the items can be sorted by year, and requesting ‘1948’ (when only Canberra was still open) shows all 28 items (not all of which are messages) summarized in the year-by-year totals on the left. Based on information from other sources, however, this set does not appear to be complete.

Another site that contains millions of government documents released primarily by requests through the Freedom of Information Act (FOIA) is the Black Vault Archive. The VENONA documents are described at https://www.theblackvault.com/documents/nsa/venona/index.html, where William P. Crowell’s original proud announcement is reproduced. Yet the page that claims to guide viewers to the scanned images (https://www.theblackvault.com/documents/nsa/venona/venona_docs.html ) is restricted to Release Six of the documents, and contains only null pointers. Table 9 would in principle be very interesting, but it turns out to be valueless.

Britain’s GCHQ probably holds a considerable number of transcripts. In its introduction to the HW 15 series, it makes some confident and ambitious claims about the collection (see https://discovery.nationalarchives.gov.uk/details/r/C9294) Yet the follow-up is not very precise (and its authorized history, by John Ferris, almost completely ignores the topic of VENONA). The series is tantalizing, but it is undigitized, and it is not clear from the summary descriptions whether all the transcripts are contained in the archive. For example, HW 15/1 and 15/2 claim to describe at item level all the KGB traffic between Canberra and Moscow from 1943 to 1948.  Yet the description indicates that the last message in the traffic was from January 1948, while other sources show the June 1948 message from Moscow to Canberra. The items are not indexed, and it is impossible to verify anything without inspecting the records in person.  

Another valuable source can be books on the subject (see below), but they are not a failsafe guide to locating where the originals of messages can be found. For instance, Breaking The Codes (1998) by Desmond Hall and David Horner, describes an array of the Canberra-Moscow traffic, and includes two page-size reproductions of critical cables from 1946 and 1948, but their Notes and Sources do not provide any information on where they may be found, apart from a pointer to the (obsolete) NSA url. I suspect that secret Australian government archives contain the full couple of hundred items that are frequently mentioned here, and elsewhere.

Lastly, a few comments on classification of items. The conventional way for the records to be stored and identified was to use the serial numbers and dates provided by the Soviet representatives. At the start of each year, a fresh counter was set for each direction of traffic. For example, the Wilson Center shows twenty-one items for Canberra and Moscow from the years 1943 to 1947 (with only one being from Canberra to Moscow). In 1943, the first entry deciphered for Moscow to Canberra is numbered 124, dated August 21, and the latest, 233, is dated December 2. The first for 1944 is numbered 182, on July 27, and the latest 214, August 31. 1945 starts earlier, on February 16, with number 29, with the last, 253, appearing on November 6. The sole Canberra to Moscow cable is numbered 116, on March 16, 1946. Hall and Horner offer cable number 123 for March 19 of that year, and 324 from September 1.

Thus it is fairly straightforward to derive the probable volumes of traffic over a given channel, and thus to provide an estimate of what percentage of the total number of cables was decrypted, if only partially. It would be useful if commentators and writers provided both serial number and date for any message they refer to: Nigel West does not do so, while Haynes and Klehr are very methodical in their Notes, but, since no comprehensive archive appears to exit for researchers to follow up on, and the NSA repository is so flawed, it does not matter much.

• How do the published accounts of the VENONA project contribute to an authentic story?

1. Books:

I identify sixteen publications that are either dedicated to VENONA, allocate a chapter to it, or provide broad insights across their texts. I highlight Lamphere’s rather audacious contribution that antedated the formal VENONA announcements, as well as Peter Wright’s quite astonishing disclosure from 1987. Several other works refer to VENONA, and a few have valuable information on it: I shall cite them occasionally in later sections. I also analyze a few relevant articles that have appeared in the academic press. First of all, however, I list a few books published before the official VENONA announcement that gave broad hints about the project, and I summarize their contributions.

Wilderness of Mirrors by David C. Martin * (1980)

            Martin introduced the idea of a group of American cryptanalysts mounting an attack against the Russian cipher system ‘midway through World War II’, and ‘using as their basic weapon the charred remnants of a Soviet code book that had been salvaged from a battlefield in Finland’. Note the timing, and the reference to a battlefield, not a consulate at Petsamo, and the fact of the codebook’s being available during the war. Yet perhaps Martin confused two stories, since he mixes up ‘charred’ (Petsamo) with ‘bullet-holes’ (battlefield). Martin described how the cryptanalysts discovered the duplication of OTPs, but declared that, because of the partial nature of the codebook’s reassembly, the first big breakthrough did not occur until 1949.

[* I have formerly criticized Wilderness of Mirrors for its incorrect representation of William Harvey’s sudden inspiration concerning Kim Philby, when the information had been planted on him by Dick White and Arthur Martin. I was gratified to make contact with Mr. Martin several weeks ago, and I pointed him to my research. He very graciously acknowledged what the archives have since shown, and he accepted my criticism with dignity. He was, after all, a pioneer working with sketchy information, and I now salute him.]

A Matter of Trust by Nigel West (1982)

            West presented the interception and decryption of Soviet wireless traffic (U-TRAFFIC) as primarily a British success, as early as 1945, with the Americans ‘duplicating’ that effort with Operation BRIDE at Fort Meade, in Maryland. Yet the account becomes more garbled: West suggested that the Americans made their first advances in the field in Australia, at the end of the war. “An American intercept station at Shoal Bay near Darwin”, he writes, “had by luck cut into some Soviet diplomatic wireless traffic.” The results were so promising that Courtney [sic] Young was sent out, and Australia subsequently set up ASIO. Of course, it was routine cable traffic had been picked up, and what West had been told was a romantic fiction.

The Shadow Warriors by Bradley F. Smith (1983)

            Smith’s contribution lay entirely in the matter of the purchase by Bill Donovan’s OSS (the Office of Strategic Services, the forerunner of the CIA) of the Soviets’ codebooks and other cipher material from the Finns in November 1945, and he gave a useful break-down of the events, ending up with the handover of the material to Foreign Minster Gromyko on February 15, 1945. Smith made no reference to the material’s being copied before return, but took a dovish approach, justifying the action in the context of US-Soviet relations at the time, even though it ‘virtually guaranteed that the Russians would change their codes’.  “The really irresponsible gamble had been taken by Donovan when he purchased the codes in the first place”, writes Smith.

Too Secret Too Long by Chapman Pincher (1984)

Pincher covered the evolution of Operation BRIDE/DRUG/VENONA in Chapter 18 of Too Secret Too Long, relying mostly on ‘confidential information’, while also crediting David Martin for the anecdote about the recovery of the Petsamo code-book.  He added that the codebreakers were ‘greatly assisted in this mammoth endeavour’ by that acquisition, and he provided a useful explanation of the Soviet mistake of re-using OTPs. He sprinkled references to BRIDE generously around his text, as he explained how the decryption effort led to the unmasking of Fuchs and Macean, and triggered further suspicions on characters such as Mitchell and Hollis.

Australia’s Spies and Their Secrets by David McKnight (1994)

McKnight’s book provided an excellent description of the fallout from the investigation into security leakages in Australia, but he was naturally hesitant and suspicious about the codebreaking exercise itself. He spoke to several retired ASIO officers, but they were reluctant to divulge anything more than sketchy details. McKnight gained access to an internal ASIO history that made reference to experiences gained from the ULTRA project. He had read Lamphere and Wright, but assessed that ‘these books give jumbled and contradictory accounts’ – very true. Thus he did not know whom to trust, and for some reason stated that it was the FBI who had been breaking the transmissions. He assumed that, when Gouzenko defected, all Russian ciphers and codes were changed, but that the 1945 window ‘provided the codebreakers with enough work to last at least two decades’.

Now I turn to the major works:

1.  The FBI-KGB War by Robert Lamphere (1986)

The retired FBI agent, Robert Lamphere, was the first to break detailed news about VENONA, in 1986, when his book The FBI-KGB War was published. Lamphere was obviously not allowed to mention the word ‘VENONA’ (it does not appear in the index), despite the fact that Pincher had already revealed it, but Chapter Six, The Break, gives a good explanation of how he helped Meredith Gardner on the project. He admits to having been constrained in what he could write: “The National Security Agency  . . . does not want me to reveal certain aspects”. Yet he went on to say that he thought he could tell enough for anyone reading his account to comprehend the magnitude of the breakthrough.

In 1947, when he was assigned to the project, Lamphere was given fragments of Moscow-New York messages from 1944-45 to inspect, and was reminded of what Gouzenko, the cipher-clerk who defected in Canada, had told his interrogators. His first important observation is that Gouzenko, even though he was employed by the GRU (military intelligence), knew enough of the KGB encipherment system to inform his interrogators that it was similar to the GRU’s OTP system. Lamphere then gives a clear description of how the encryption process worked, explaining the role of a codebook (or dictionary) to translate common terms into a numeric value (although he describes each codebook entry as having five-digit numbers, not four, as some other accounts declare). Lamphere was introduced to Gardner only in 1948, however, and he wondered how his colleague had managed to make any progress on the KGB codebook. He was shown the partially burned codebook that Gardner kept in his office, and, even though it was out of date, Lamphere stated that it had nevertheless been ‘immensely helpful’ in helping him slowly construct the current codebook. ‘Out of date’ is a characteristically imprecise phrase, however: was it obsolete because the current codebook (1948) had replaced it, or because it was a version that had been superseded by that used during the period under review (1943-1946), or was it obsolete because it derived from a period that antedated 1943?

Here Lamphere reports the frequently mangled story about the Finns’ recovery of the charred codebook ‘during World War II’ (nothing more specific), and how William Donovan, the head of OSS, in November 1944 purchased 1500 pages of code and cipher material, a copy being provided ‘immediately’ to the ASA (the Army Security Agency, forerunner of the NSA). Secretary of State Stettinius insisted that the codebook be returned to the Russians, but not before Donovan had it copied. Of course, the Soviets knew what must have happened, and changed their codes in May 1945, thus putting a stop to any further decryption. Lamphere does not make any distinction between changes of codebooks and OTPs, however, and sadly mixes his terminology between ‘ciphers’ and ‘codebooks’. Yet he concludes this section with the statement: “ . . . in 1948 Gardner had a codebook that the KGB had used in Finland in 1944. It wasn’t the current codebook, but it was similar, and above all it provided him with a start.”

This is a contradiction, however, and Lamphere appears to be confusing the year in which the codebook was handed over (1944) with the year in which was retrieved (1941). If the codebook had been used in 1944, it was current for the purposes of the traffic that they were inspecting. It may have been replaced in 1945, alongside the provision of new OTPs, when Arlington Hall could not break into any fresh messages: we are not told. On the other hand, in 1945, the Soviets may have continued with the codebook in place, in the belief that its new OTPs would make the issue of exposed codebooks irrelevant. Thus the codebook in question may have beencurrent for 1948: by then, only Canberra-Moscow traffic was still being decrypted, and that success would suggest the codebook had endured. Lamphere was mightily confused. Yet what he writes constitutes a strong statement that Gardner indeed had access to an important tool for grappling with what turned out to be a more intransigent challenge.

Lamphere ascribes Gardner’s success in making a crack into the KGB system to the availability of all the wartime traffic (copied from the cable companies’ records) and the availability of a codebook, ‘although not the right one’. (Given that wartime traffic was split across two codebooks, this is not an authoritative statement.) He then helped his colleague by tracking down the plain text of some materials that the Soviets had transcribed and enciphered in 1944. How he knew that much is, however, not clear. Lamphere helps Gardner out, mainly through the efforts of a translator, Mr. Boguslav, who provided the FBI field office in New York with a host of material, mostly in Russian, except for a few translations into English. Rather provocatively and mysteriously, Lamphere writes: “This material had been photographed by New York agents in the course of an investigation into Soviet operations in New York in 1944.” He offers no other explanation: these must have been the night-time raids on the Soviet consulate that came to light later.

This anecdote seems so utterly illogical that I attempt to parse it here:

1. Gardner happened to ask Lamphere whether there was a possibility of obtaining the plain text of materials enciphered by the Soviets in 1944;
2. Gardner presumably meant acquiring the original American texts, since he would have had no imaginings that secret Soviet sources would be available;
3. Gardner had enough information on the materials to offer a description that made sense to Lamphere’s colleagues;
4. Lamphere’s friend Boguslav came up with the exact texts required, having himself translated them from Russian into English;
5. The material had been photographed by FBI agents while investigating Soviet operations in New York at the exact time of the transmissions in which Gardner had shown interest;
6. The FBI, after this highly illegal and audacious raid, the authorization of which is obscure, then sat on the material for four years;
7. When presented with the results, Gardner was excited, but apparently never showed any surprise at the coincidental nature of the recovery.

This seems to me like a very unconvincing cover story.

In any event, according to the author, these items contributed greatly to Gardner’s being able to fill out his codebook, and Lamphere gives examples of some other source documents that were able to help Gardner add further items to it: telegrams from Churchill to Truman, scientific reports on uranium 235, and other reports on the Manhattan project.  In summary Lamphere’s contribution was a breakthrough, but frequently elliptical, often wrong, sometimes confused, and occasionally brazenly phony. Maybe that misrepresentation was what the NSA wanted to encourage. I shall investigate such anomalies later.

• Spycatcher by Peter Wright (1987)

Peter Wright quite blatantly outs VENONA and all its mysteries. (There is an irony in the publication of Spycatcher, since the book was originally banned in the UK, and copies frequently imported there from the USA, whose intelligence officers cannot have been delighted with what Wright revealed about their hitherto secret project.) The author lays out the project name, how the coding systems worked, and describes the stresses that led to the duplication of OTPs. Next he presents the shocking revelation that Gardner made use of the ‘charred remains of a Russian codebook found on a battlefield in Finland’, indicating that the remnants included the very valuable entries for ‘Spell’ and ‘Endspell’. He mistakenly attributes the discovery of repeated use of the same OTP page to Gardner, as well.

One of Wright’s groundbreaking pieces of analysis is the detail with which he covers how the British and Americans exploited the breaks they made with something called a ‘window index’ (see p 181). He also presents the issue of codebooks in a new light: “Sometimes [??] they changed, and whereas the Ambassadorial, GRU, and trade channels used a straightforward listed codebook, rather like a dictionary, so that the codebreakers could guess from the group where in the codebook it appeared, the KGB used a special multivolume random codebook which made decrypting matched KGB channels a mindbending task.” This observation anticipates what Budiansky later wrote [see below] concerning one-part and two-part codebooks, but I believe the way Wright presents it is unique, as opposed to the generic change that occurred when POBEDA was replaced across all channels, as other accounts describe. His comment suggesting that the GRU used a one-part codebook is provocative.

Wright has much more to say about VENONA, and the outcomes from the decryptions. Chapter 13 of Spycatcher is well worth reading again: the author was at the centre of the efforts to convert the fragments passed on by GCHQ into the identification of real Soviet spies. His text is mostly very accurate, although the chronology in his coverage of the Australian scene is awry. But he must have given the NSA a fright, and his comments about the codebook exploited by Gardner will have alarming consequences when I later analyze that issue in more depth.

• Introductory History of VENONA and Guide to the Translations by Robert Louis Benson (1995)

The Americans set out to control the narrative when Robert Louis Benson, who had been a history professor at California University, in 1995 published an ‘official’ introduction to the saga under the auspices of the NSA. It is a very short pamphlet, of historical interest since it preceded the much more formal and bulky item that Benson authored with Michael Warner, issued the following year [see below], but one that has largely been overlooked. And I can understand why its existence may have been suppressed: within its ten thin pages exists an implicit rebuff to what Lamphere wrote, what I would call a counter-mythology based on very shaky ground. I cite a significant paragraph:

            In spite of what has been written in a number of books and articles, Arlington Hall made the VENONA breakthroughs purely through sweat-of-the-brow analysis. There was no cryptographic assistance for Lieutenant Richard Hallock, Cecil Phillips, or Meredith Gardner  and their colleagues from lost, discovered, or battlefield-recovered Soviet codebooks during the years in which the main analytic breakthroughs were made (through 1952). It was not until 1953 that a photocopy of a partially burned codebook (recovered by U.S. Military Intelligence in 1945) was discovered to be related to the VENONA cryptographic systems after another cryptanalytic breakthrough. The successful decryption of the VENONA messages was a triumph of analysis by a small group of intelligent and dedicated women and men working long hours in their cramped offices at Arlington Hall.

This seems to me an epic example of what I have called ‘Hinsleyesque denial’ after the celebrated British intelligence historian – attempting to deny the truth of previously asserted facts without identifying where they appeared, or providing evidence that would gainsay them, while at the same time piquing the reader’s interest into the source of such stories. ‘Sweat of the brow’ suggests no benefits of data processing equipment. The attempt is so clumsy, and so counter-intuitive, and flies in the face of so many facts, that I shall return to analyzing it later in this piece.

The pamphlet has many other weaknesses. It claims to be an Introductory History to Venona, but restricts itself to the USA, mentioning New York and Washington, and including a brief reference to San Francisco, while utterly ignoring the worldwide dimensions. It states that, by 1948 (that late!), the British had joined the project, yet other sources (e.g. Romerstein and Breindel, and even Benson and Warner in the volume below!) inform us that co-operation started in 1945. Benson attributes the ‘excellent cooperation’ to the efforts of Lamphere and Gardner, attempting to lock in the year as 1948 while also giving an inadvertent nod to the FBI man he wished to bring down a peg. He identifies Philby as a source of leakage (but not Currie or Weisband), dating his revelations to 1949-1951, when the notorious ‘Black Friday’ had already occurred. All in all, it is a careless and ill-designed tract that attempts to swell the skills and achievements of the Americans at Arlington Hall without giving any proper justice to the broader dimensions of the project.

• History of Venona by Robert Louis Benson and Cecil James Phillips (1995)

I do not believe this volume has been published in the form given above. The first reference to it that I have found appears in Spying Through a Glass Darkly, by David Alvarez and Eduard Mark [see below]. In Footnote 30 to Chapter 2, the authors list it as being published by the NSA in 1995, but then ominously add: “The authors are indebted to Lou Benson for arranging the declassification of portions of this multivolume in-house history of the so-called Venona project.” That is all highly irregular, of course. The work was an in-house history, and still classified, yet one of its authors took it upon himself to arrange for some of its material to be released – presumably for general public consumption – but especially for a ‘trusted’ author. So where can the declassified sections be found?

I found another clue in the work of Matthew Aid. On a 2009 page of the US National Security Archives (see https://nsarchive2.gwu.edu/NSAEBB/NSAEBB278/index.htm) can be found a statement that Aid, on that day (June 19) ‘posted a collection of declassified documents obtained for his new book The Secret Sentry on the Archive’s Web site’. ‘Document 1’ consists of extracts from the same History of Venona, which Aid equates to another item titled Top Secret Umbra. And indeed, that link leads to a document familiar to me with that title – something that appears to be an early version of the History, heavily redacted (with obscured words that make the redactions look absurd, in 2025) for release in 2004. Does it comprise the whole work? I think not. Alvarez and Mark referred to a ‘multivolume ‘history’, and Top Secret Umbra stops with ‘Volume 3, The Bill Smith Era, November 1943-1946’. Moreover, if Alvarez and Mark had to arrange for a special declassification in preparing their 2016 publication, the result of their efforts would surely have been information not in the public domain. Yet their further references, e.g. Footnotes 51 and 52, point accurately at paragraphs in the UMBRA document. Had they overlooked, or misidentified UMBRA, and did Benson mislead them? But then why would Aid, while he claimed to have succeeded in having the History of Venona declassified, restrict his disclosure to an old and heavily redacted UMBRA document instead?  It does not make sense. In any event, I summarize the key points made in Top Secret Umbra. (The full text can be seen at https://archive.org/stream/history_of_venona-nsa/01_djvu.txt.)

The style of the work is informal and chatty. It covers a lot of detail on the decryption processes not found elsewhere, but there are few radically fresh insights. Benson (the primary author) early on makes the claim that ‘each Russian entity had its own codebook’, and he asserts that the process of identifying cipher pads shared between Trade and Diplomatic traffic was the best method of making progress, irrespective of codebook variants. He states that the so-called ‘Black Friday’ had nothing to do with VENONA (something with which I am in agreement). The second myth he wants to debunk is that the breakthrough came about because the OSS had obtained Russian codebooks. “The OSS did not in any way contribute to the Venona break;”, he writes, continuing:  “ . . . the fundamental cryptanalytical discoveries and the decryptions through 1952 were not aided by our side having any KGB or GRU code book from any source. It was an analytical success.” I believe that is a Jesuitical statement. While denying that the OSS did not contribute, it does not specifically exclude the fact that GCHQ supplied the material in question, and the second part of the sentence digs a hole from which Benson and Phillips could not easily escape, as I shall show later. Benson then reiterates the non-involvement of OSS in sharp terms, lest his readers miss the point.

There are other nuggets. Benson carefully points out that Captain Abraham Sinkov’s report [see below], after his February 1941 visit to GC&CS, specifically does not mention Soviet diplomatic or intelligence service systems, instead describing features of military systems. He does refer to Tiltman’s pre-Petsamo recoveries, describing them as ‘military codebooks and other cryptographic material’, and he carefully informs us that the codebooks Tiltman retrieved ‘represent a different trove than the so-called Petsamo material’. It would have been useful if he could have written more about that, since it would appear he has studied both. Yet in the same paragraph, he writes that the Petsamo material, ‘which included a KGB codebook, instructions for using additive, tables and an emergency cipher system . . . reached the UK-US in 1945-46’.

Benson states that, during the war, there was a strong resistance to sharing any aspects of the VENONA project with the British, but does not explain why. He presents a challenging anecdote (relying on what Lou Madison, GCHQ archivist told him in 1992!) about MI6’s and MI5’s cooperation in 1942 on ‘working Russian targets’, and that the exercise by RSS, the Radio Security Service, ‘discovered extensive Russian illicit radio links, apparently GRU, KGB and Comintern’. This is surely wrong: routine mop-up operations against unauthorized wireless transmissions were undertaken by RSS, but apart from the familiar output from the Soviet Embassy, little of a sinister nature was found.

The work presents some useful information on Lauchlin Currie, who probably exaggerated what he heard about work on Soviet ciphers to suggest to his handlers in 1944 that the US ‘had broken the diplomatic code’, something that another Soviet agent Elizabeth Bentley subsequently heard about. Currie slipperily tried to wriggle out of his betrayals when he was interviewed by the FBI in 1947. Benson speculates that the indicator changes that the Soviets implemented on May 1, 1944 (evidence of which comes from a famous VENONA message) may have been triggered by Currie’s action, but is not certain that it was so.

Benson’s last fling is to belittle the importance of the Petsamo material. It is worthwhile reproducing how he described the main treasures, including the ‘partially burned codebook of the First Chief Directorate of the KGB’:

This codebook (KOD POBJEDA), and its indicator system later came to be known as [redacted]. The Petsamo trove also included KOD 26, a true Dip (consular) codebook, and at least one GRU codebook, as well as rules for using one-time pads (the additive) to encipher groups from the codebooks, and instructions for using an emergency cipher system in case of compromise of the regular systems . . . . Some traffic, plaintext and cipher text was [sic] also taken at Petsamo.

He goes on to write: “When the British took over this source in 1946, they too got copies of the Petsamo material and passed more copied on to Arlington Hall.” Yet Benson continues: However, the real story is this: in 1945, TICOM had already obtained all of this and more in their sweep through the German Sigint centers, the team seizing German photocopies of the material originally taken by the Finns (or maybe the Germans themselves) at Petsamo.

Now all that may be fine. Disparage the resented OSS. Admit that the British were the suppliers of the Petsamo material. Yet claim it was all rather irrelevant. (How Benson knew enough about both troves is a bit of a stretch, but maybe he just trusted what he was told. There were 73 steel file cabinets shipped, enclosing 300,000 pages of material.) He next relates that some of the TICOM material, shipped to GC&CS for study, was microfilmed and sent on to Arlington Hall rather quickly. How generous of the Brits! Was that effort really necessary? Did they not trust the US Army to get it to the right place? Benson then offers some useful detail about some of the Petsamo documents, writes about Gouzenko and Bentley, but never mentions that the TICOM material did not become of use until 1952, a fact that he very clearly publicizes in his other works. Why is that, I wonder?

Moreover, there is one last coda. Benson needs to demolish one more legend – that of the charred codebook recovered on the battlefield, an event that never happened, in his estimation. So what about the codebook displayed by Gardner, which Lamphere remarked on? He brings in Meredith Gardner himself:

Meredith Gardner, who was the first person to recognize the KGB nature of [redacted] later told Bob Lamphere of the FBI that the codebook that he (Gardner) used to make the breakthrough had been found on a battlefield and had a bullethole in it. Meredith later told me that he was referring to a mark that looked like a bullethole but certainly wasn’t. We are getting ahead of the story, but the book that Meredith was using was the aforementioned KOD14, which he studied to learn KGB codebook vocabulary and just to see what a Russian codebook looked like. It was not a Venona system, and did not lead to the first Venona break, which was accomplished by bookbreaking without the benefit of the relevant book [redacted]. Pages 86 and 87 of the KOD 14 book (which I’ve only seen in photocopy) do indeed show a round, but irregular, black mark – probably an ink blot.

So that’s all right, then, is it? It sounds to me as if Gardner had to be trained to get his story right. Why did he simply not show Benson the codebook in question? I am sure he would have hung on to it as a souvenir. Yet Benson had informed us earlier in his history that KOD 14, part of the Petsamo haul, ‘was an important find, used by the NKVD rear service security troops’, and that it had also been seized by the Finns during military operations on the Karelian front. In what sense was KOD 14 not a ‘Venona system’? Why would Gardner have wanted to have selected this particular item, and why was he allowed to extract it and display it in his office, so soon after its delivery, I wonder? And confusing ink blots and bulletholes? How lacking in perceptiveness could these cryptanalysts be sometimes.

• American Cryptology During the Cold War 1945-1989 by Thomas R. Johnson (1995)

This ‘Official Four-Volume History of the NSA’ predictably foreshadows the NSA/CIA production of the following year. While omitting some important aspects it does add a few nuggets in Chapter 4, ‘The Soviet Problem’. It credits the Army’s successful attack on Japanese diplomatic communications for giving SIS (the Signals Intelligence Service, renamed the Signals Security Agency, or SSA, in 1943) ‘some handholds’ [unspecified] into Soviet systems’. On the BRUSA agreement, it generously acknowledges that ‘the British provided much of the cryptanalytic expertise, the Americans most of the processing capability’ and adds that ‘TICOM debriefings of German cryptologists also gave the partners useful information about Soviet systems’.

After giving a careful description of what Lieutenant Richard Hallock achieved with his ‘depth-testing’ exercises (i.e. looking for possible duplicated pages), with the first  reward coming in November 1944, after more than a year’s effort, Johnson relates how Gardner uncovered the ‘spell-endspell’ values that allowed the Soviet cryptographers to enter non-Russian words, or terms that did not appear in the codebooks. He records how the defecting cipher-clerk Gouzenko was able to explain how codebooks were put together, and how additives were used for final encryption. Next, he expands on Lamphere’s rather cautious remark concerning photographing Soviet material: “A second source of information was a 1944 FBI burglary of AMTORG, during which agents carried off stacks of unenciphered messages with their cipher text equivalents”, but Johnson declares that the FBI turned over ‘this bonanza’ to Gardner only in 1948. This is a reinforcement of the Lamphere message, and equally hard to believe. Why would the FBI engage in such a hazardous enterprise and then sit on the fruits of its labour for four years?

Johnson covers the STELLA POLARIS saga in some depth. His version of the story is that, in 1941 at Petsamo, the Finns, even though some of the material they recovered was burned beyond use, managed to recover ‘certain of the codebooks  . . . more or less intact’. (I do not understand how anything might be recovered ‘more intact’ than in its original state, but I’ll let that pass.) He then adds that these codebooks were the same ones which, ‘in the mid-1940s, Meredith Gardner was working on’. I shall have to revisit this claim later, as the evidence is contrary [see Lamphere above]. Rather enigmatically, he writes that ‘the charred codebook fragments’ were turned over to the Finnish COMINT service, headed by one Colonel Hallamaa. (Just the charred fragments, that were of no use, or the intact items as well? Again, it does not make sense.) Yet these remnants in 1944 found their way to Sweden, where they were copied, and such copies landed up in the hands of the Swedish, German and Japanese COMINT organizations. Eventually the British laid their hands on copies, and, at the same time, in 1945, the Americans’ OSS began working with Hallamaa, and received their quota, ‘although not, perhaps, a complete set’. (Who would know what a ‘complete set’ was?)

The book then explains that this OSS-Hallamaa set was not the same as the set that the TICOM team acquired shortly after the end of the war, the latter being that which made its way to Meredith Gardner. (Of course it may have been the same material, simply being the copy that landed up with the Germans. The point is not followed up.) And then the pot is further stirred: “Shortly thereafter AFSA began obtaining Petsamo materials from the British under the codename Source 267 and may, at some point, have received copies from OSS/CIG, but these were no more than duplicates of materials they already had,” he writes. How an official historian can be so vague about the first half of the equation, but so authoritative over the second, is a conundrum. In addition, AFSA (the Armed Forces Security Agency) was not created until May 1949: Johnson probably intended to write ‘ASA’ not ‘AFSA’ in this passage.

After a lengthy digression on the coverage of spies who were revealed by Venona, Johnson emphasizes that most of the decrypted traffic came from ASA’s 1944-45 files (without explaining why: that will become clear later), and that it was not decrypted until the late 1940s and early 1950s. The section tantalizingly ends with a paragraph headed by the title ‘Black Friday’ in bold. Unfortunately, this whole section has been censored, as have many other parts of this history. Why these matters were so sensitive in 1995 is also puzzling: my copy of the book is the original 1995 edition, and I do not believe it has been updated since.

• VENONA: Soviet Espionage and the American Response 1939-1957 edited by Robert Benson and Michael Warner (1996)

The US agencies undertook a more serious and substantial initiative quickly thereafter, the next publication being VENONA: Soviet Espionage and the American Response 1939-1957, issued jointly by the CIA and the NSA in 1996. (It is available on-line at https://www.cia.gov/resources/csi/books-monographs/venona/). The year 1939 would appear to be premature for the starting-point of an history of VENONA, but the editors, Robert Louis Benson and Michael Warner, took as their impetus a conversation that Loy Henderson had with defector Walter Krivitsky in March 1939, an experience that highlighted the threats from Soviet espionage. 1957 seems to have been chosen as the terminus because of the imprisonment of the KGB illegal Rudolf Abel, and a Supreme Court decision that ruled that the US government had enforced the Smith act too broadly, thus making ‘the Act almost useless in prosecuting Communists’. A more imaginative approach would have been to explain what happened between 1957 and July 1995, when the CIA released the first group of the NSA’s translations to the public, as Deputy Director of the NSA, William P. Crowell announced in his Foreword.

The thirty-page Preface is in indispensable guide to the evolution of the project, although it finesses some of the more sensitive aspects of the process by which the cryptanalysts achieved their successes, and tends to some self-aggrandizement about the skills and breakthroughs that they made. As its title suggests, it has a very American focus. It mentions British involvement, but the Preface says nothing about the multiple systems of traffic that were deployed between Moscow and other countries. It refers to the capture of scorched codebooks and cryptographic materials from the Soviet consulate in Petsamo in June 1941, but it interprets the episode only in terms of the Russians’ being alerted soon afterwards to the Germans’ trying to exploit the codebook. It records in a Footnote that it was the same as that which the US Army recovered in Germany in April 1945, but says nothing as to what assistance it gave to Arlington Hall. It does present the important fact that, on May 1, 1944, ‘KGB code clerks began using a new message starting-point indicator for telegrams’, and the book includes an image of the completely deciphered message (No. 26) that broadcast the change. Yet the reader is left stranded: when was this message decrypted, and what subsequent efforts did it facilitate?

Part 1 consists of 190 pages, titled The American Response to Soviet Espionage and contains a variety of documents and memoranda concerning that response. Part 2 presents 99 individual cables, almost all of which are taken from traffic between Moscow and the USA (New York, San Francisco and Washington). Two derive from Moscow to London, and one from Moscow to Mexico City. A few GRU (Army & Naval) messages are included among the dominant KGB traffic. They are properly introduced with dates. The last message listed is from September 17, 1945, from London to Moscow, describing Philby’s reaction to the Gouzenko case.

This is an obviously inadequate compilation. It makes no express statement as to why no US-based traffic was able to be decrypted after 1945 (with its selection of cables strongly hinting as much). It makes (on a page contributed by Cecil Phillips, headed What Made VENONA Possible?) an oblique hint that some messages were still being decrypted in June 1948, without identifying the channel and circumstances. Phillips must have been the source of Benson’s comment on ‘breakthroughs’ in the previous publication, since he writes here: “Arlington Hall’s Venona breakthrough in 1943-46 was a purely analytical accomplishment, achieved without benefit of either Soviet code-books or plain-text copies of original messages”, perhaps subtly signalling that later breakthroughs did take advantage of such devices, but again weakly discrediting evidence from elsewhere, and minimizing the assistance of IBM equipment.

Phillips’s contribution is annoyingly elliptical. He writes: “The 1944-46 messages – which yielded the early translations and the bulk of all translations – were recovered over a period of years by Arlington Hall cryptanalysts and decoded from a ‘codebook’ that crypto-linguist Meredith Gardner reconstructed by using classic codebreaking techniques.” There is no explanation of what those techniques were: the assertion cries out for support and explanation. Yet on the same page Phillips writes: “Nevertheless, most of the duplicate pages were used between 1942 and 1944 – years of rapid expansion of Soviet diplomatic communications.” ‘Why was the majority of the traffic reconstructed from a period when the phenomenon of duplicate OTPs was quickly fading away?’, one might ask.

Overall this is a very parochial account, and designed not to give much away. Yet the Soviet threat was international, and the cross-country connections were profound. The Arlington Hall cryptanalysts were able to take advantage of the Canberra-Moscow traffic, for example, but, presumably out of a perverse notion of pride, the American authors were not able or willing to give full credit.

• The Venona Story by Robert Benson (1996)

The NSA issued a useful handbook, written by Robert Benson, in 1996. It is available at https://www.nsa.gov/portals/75/documents/about/cryptologic-heritage/historical-figures-publications/publications/coldwar/venona_story.pdf.  As a summary of the project, and a description of its main deliverables, it is an excellent guide. Yet it is a sanitized version, makes no mention (for example) of FBI break-ins, and strongly affirms Benson’s previous message that no use of acquired codebooks was made until the 1950s, with the exploitation of the TICOM material, emphasizing the ‘sweat-and-brow’ nature of the achievements.

• Breaking the Codes by Desmond Hall and David Horner (1998)

The next volume to appear arrived in 1998, Breaking the Codes, by Desmond Ball and David Horner, subtitled Australia’s KGB Network. It is a solid, resourceful work, but not properly discriminatory in its coverage of VENONA overall, those sections having been undertaken by Desmond Ball. Ball follows closely the sources above, quoting liberally, but he also cites as references Pincher, Wright, Borovik, Modin, West, Bower and Philby, without offering any explanation as to why they should be trusted. On the other hand, he does quote from some exclusive items of correspondence and interviews, such as that of Gardner with I. Livingstone in 1996, and the author with Renée Frank of the NSA a year later. Some of these may offer reliable insights. For instance, the exchange with Frank leads to a claim that “In June 1946, during a visit to GCHQ, Cecil Phillips gave his British counterparts a detailed briefing on the ‘techniques and progress’ of the Venona program”. Yet Ball does not comment on how that intelligence would tend to contradict what Benson claimed in his Introductory History, which Ball uses widely (and occasionally mis-cites).

Ball then moves briskly into the British side of the project, using some familiar sources (e.g. Benson/Warner, Aldrich) and his interviews, but also plucking from the Public Record Office some papers on VENONA from a 1996 release. Unfortunately, these are not identified precisely, and it is difficult to determine from the Discovery function at the National Archives what the correspondences are. Moreover, none of the VENONA files at HW/15 (the critical GCHQ material), or Meredith Gardner’s special reports at HW 15/58, have been digitized. This is a colossal disappointment, and makes it almost impossible for a remote researcher to investigate and verify Ball’s text properly.

The author makes some crisper claims about the level of collaboration. Writing of 1948, he declares:

The skills and techniques developed at Arlington Hall were passed to the British VENONA team, which in turn took over some of the VENONA activities, including further cryptanalytic work on the Moscow-Canberra traffic. A virtually complete set of the 1944-45 codebook and thousands [sic!] of duplicated OTP pages were soon collated.

And that is where his comment about ‘real-time’ appears. That might explain the Americans’ reluctance to say much about that circuit. Between 1948 and 1950 hundreds of KGB cables that had been encoded with the 1944-45 and 1945-48 codebooks * were decrypted by GCHQ and Arlington Hall, he adds, before lamenting the dual exposures that occurred by allowing Philby and Weisband access to what was going on. Philby’s briefing by Oldfield, before he left for Washington, happened after Black Friday, of course, but his knowledge thereafter alerted him to the search for HOMER (Maclean), REST (Fuchs) and eventually STANLEY (himself).

[* This statement implies that new codebooks were issued in 1945: I see no evidence for that assertion anywhere.]

One observation is puzzling. Ball writes that “A wholesale change in Soviet encryption systems was introduced by the KGB’s Central Cryptographic Service in 1950, including new codebooks, OTP books and encipherment practice.” He provides no source for this claim, and it would appear to have been redundant if Black Friday had in fact taken over a year beforehand.: I believe this assertion betrays a common confusion concerning a) the VENONA traffic, and b) internal Soviet police, army, industrial and prison camp systems, which had largely been conducted en clair up till then, and which the Americans had been closely monitoring. His final sentence in this section is to quote Andrew and Gordievsky for claiming that ‘the KGB cipher office responsible for the production and distribution of the duplicated OTP pages was reportedly later shot.’

Of course, the freshest insights come from a chapter dedicated to Soviet espionage in Australia. Ball relies on the details of the Fifth Venona Release, especially Gardner’s report on ‘Covernames in Diplomatic Traffic’, dated August 30, 1947, which appears on page 93 onwards in the Benson/Warner compilation. He makes the point that ‘despite its modest proportions, the Australian Venona operation was very lucrative for both cryptanalytic and counter-espionage points of view’ and gives as a prime example the British War Cabinet documents transmitted verbatim in March-April 1946. That discovery, and the tracing of the originals, allowed Gardner  and his team to recover a large proportion of the four-digit groups in the KI codebook (KI, the Committee of Information, then being the short-lived ‘KGB’ home.) That was something that the NSA has been reluctant to acknowledge.

Using some local evidence from Australia, to a small degree Evdokia Petrov’s testimony from 1954, and then through a broad sweep of the voluminous report identified as ‘Soviet State Security Service Foreign Intelligence Operational Techniques (‘Legal’ residency System)’, 18 May 1955, in CRS A6283/XR1’ in the government archives, Ball was able to provide a comprehensive analysis of the KLOD group working for the KGB. Of course, none of this would have been available if the flawed OTP pads had been replaced a couple of years earlier.

In summary, Ball was resourceful and inquisitive – up to a point. He was more of a chronicler than an investigative, analytical historian.

• Venona by Nigel West (1999)

I have stated before that I believe Venona is one of Nigel West’s better books. Its main benefit is that it provides a very valuable analysis of traffic between London and Moscow, and it goes a long way in identifying the participants whose coded names appear in the transcripts. It also provides a broader perspective to the USA-dominated canon, recognizing that VENONA had important international aspects. While the author predictably has a British focus, and analyzes very carefully the London-Moscow channels of the GRU and KGB, he beneficially also tackles Australia, and Sweden, and brings more detail to the enticing story of STELLA POLARIS. For instance, he sheds more light on the Petsamo incident:

The four codebooks were a diplomatic codebook designated Kod-26; the NKVD Pobeda (Victory) code; one for use by the GRU; and one for use by the naval GRU. Studied in conjunction with medium-grade military crypto items recovered from the battlefield, which included at least one NKVD Border Guards’ codebook, the material allowed the Finnish analysts to understand how the Soviets adapted military terminology in their systems, built code-tables, and relied on a very straightforward mathematical formula to encode emergency signals.

This is a very challenging claim, since it points to two critical codebooks (GRU and Naval GRU) being part of the Petsamo haul, and it probably exaggerates what the Finns were able to accomplish. The sources of West’s intelligence are not stated.

The above passage, however, illustrates a chief defect of West’s work, namely a customary lack of identified sources – egregious in this case, as West offers only two pages of sketchy and barely relevant Endnotes. Thus one has to tread carefully through his introductory chapter, which is clearly dependent on what several intelligence professionals chose to tell him. Did he trust all he was told? Did they disagree on some points? One cannot tell: unfortunately it reads as if a series of contributors had offered chunks that have not been seamlessly rewoven.

That first chapter, titled ‘Breakthrough’, is remarkable since it constitutes the first attempt to cover the initial processes, and the extended cycle, of decryption, in detail. Yet I write ‘attempts’, since I found the overall story incoherent. (When I first read this book, many years ago, I assumed that my incomprehension was due to my slow-wittedness. I now include that the author is at fault.)  The text reads very glibly, the superficially smooth narrative flow dispensing with clarity and logic as if all were understood and obvious. The author is also lax in giving his readers a precise chronology. For example, West refers to the GCHQ liaison officer, John Tiltman, contributing to the discovery of, and investigation into, duplicate OTPs, probably in 1943, thus bringing forward the timing of GB-USA co-operation. No other source credits Tiltman with this discovery. Yet the primary topic of interest, the method by which the NSA team managed to re-create codebooks, is clumsily handled, and essential processes are finessed. For instance, West, suggesting how a breakthrough occurred, writes that the analysis of more mundane and predictable ‘trade’ messages (which tended to have formulaic preliminaries) led to steady construction of embryonic codebooks. “Hallock and his colleagues made good progress on the trade texts, to the point that they were able to predict opening phrases and acquire tentative codebooks which were stored in the indexed key banks of IBM punchcards”, writes West (p 17). But no date given, no quantification, no examples. It could be an important insight, pointing to how Gardner was able to make such progress on the codebook, but the evidence and explanation are thin.

And how did these discoveries relate to diplomatic and intelligence traffic? Were the same codebooks used? The same mangled OTPs? West is elliptical. He had written (p 14): “Following Hallock’s discovery, a prolonged study of JADE by Cecil Phillips and Genevieve Feinstein revealed an astonishing fact: the JADE material had been enciphered precisely using the same OTPs as the trade traffic.” Yet JADE has never been defined; the discovery is not dated, and he never follows up to explain. Phillips did not join Arlington Hall until June 1943, and then West rambles on to claim, first, that “there were too many sixes’” in the first group of ZDJ [?, but evidently part of the JADE system] messages, and that that phenomenon could not be accidental, and then:

Further analysis by the sector’s lead cryptographer, Genevieve Feinstein, led to a full-scale comparison between the first groups in the JADE traffic and the trade data. They were found to be identical, and when Burton Phillips [any relation?] and Katie McDonald delved further they found several hundred duplicate keys in the New York JADE traffic and the Washington trade messages.

He then writes that ‘it had been believed that JADE and its predecessors, DDE and JDF, were consular in nature ….’ Who believed it? When? What are DDE and JDF? Does this show bluffing, or simple clumsiness? And next: “This mistaken belief led to greater resources being devoted to the Soviet diplomatic system, JDA, but [but?] in 1945 Katie McDonald  . . . . identified the codegroups for months, punctuation and numbers from 1 to 1000.” For which codebook? Again, were there portions shared among the different systems? One cannot tell. According to West, Gardner then apparently concentrated on the Soviet spellcode employed to encipher English words, and was thus able to start creating a Soviet codebook through such analysis. Eventually (no date given), he found a message from New York in the JADE series dated 18 May 1944 which was entirely in English. The text, about Roosevelt’s chances of electoral victory was ‘proof that JADE was diplomatic in nature, and might be broken to reveal evidence of espionage’.

I do not follow the reasoning, or the processes. I do not like this casual impression of expertise that does not submit easily to close analysis. Cautiously, I have to suggest that West did not clearly understand what he was told by his interlocutors (who may have wanted to mislead him, anyway), and that he merely tried to put a smooth spin on a tangled tale. And the chapter continues in its random way, switching back and forth in time, introducing irrelevant information, digressing into too much detail, such as in the description of the retrieval of the Petsamo codebook in Germany by the TICOM expedition, but not explaining probably what its value was, referring briefly to the controversial break-ins by the FBI at the Soviet Consulate in Washington [Or was it in New York? Or were there multiple break-ins?], covering only summarily the implications of the Gouzenko and Petrov defections, and ending up with a summary of the harm that Leo Long, William Weisband and Kim Philby did. It is a muddle. The whole chapter should have been rewritten.

1. Venona: Decoding Soviet Espionage in America by John Earl Haynes & Harvey Klehr (1999)

This volume is a first-rate guide to the way that the Soviet Union infiltrated US government institutions, delivered by a fierce examination of the transcripts available, and a fortuitous inspection of records of the American Communist Party held by The Russian Centre for the Preservation and Study of Documents of Recent History (known affectionately as RTsKhIDNI) in Moscow before it was closed to foreigners. It offers copious notes, useful Appendices of persons referred to in the texts, and provides an excellent background historical framework.

The authors offer a smoothly-written Chapter 2 (‘Breaking the Code’) to explain how VENONA messages were cracked. It relies on the traditional sources Benson & Warner (primarily), as well as the customary interviews with cryptanalysts. While more logical in their approach than West, however, Haynes and Klehr similarly finesse the difficult aspects of the investigation. For instance, they have Meredith Gardner being transferred to the project early in 1946, becoming the principal ‘book-breaker’. He also learned Russian that spring. And then the breakthrough work is collapsed into a few terse sentences:

            His linguistic work was helped by the now expanding body of plain code groups (what was left when the overlying one-time pad cipher was removed being extracted from Trade and KGB messages). By about mid-summer 1946, Gardner had recovered enough code groups and text to be sure that the messages involved Soviet espionage. By the end of 1946, he had broken out the text of a message that revealed Soviet spying. Eventually [when?] it would become clear [to whom?] that Gardner had reconstructed the codebook used by the KGB from November 1943 into early 1946. This work allowed [when?] the reading of hundreds of KGB messages from that period and provided some of the most complete text that the Venona Project produced.

The vagueness of the chronology is frustrating. Again, I do not understand the process. The authors imply that the Trade and Diplomatic variants shared the same codebook, but never actually say so. This is a very important point: while the boilerplate structure of many commercial messages may have allowed gradual creation of a codebook, unless the lexicon of commercial and diplomatic texts greatly overlapped, and the two channels shared a codebook, the exercise would not have helped Gardner much in his book-making endeavours for diplomatic traffic. Probably, a few formulaic passages led to some initial entries in the trade codebook being made. But how did they start rescuing more significant terms for the diplomatic codebook – and all in a matter of months, no less? Yes, ‘eventually’ the codebook would be reconstructed, but the phrase ‘it would become clear that Gardner had reconstructed the codebook’ must be false. He may have reconstructed part of it by the end of 1946: as the authors write, only one incriminating message had been retrieved by then. Haynes and Klehr then inform us that the KGB had used a different codebook for 1942 and for most of 1943, and they admit that NSA analysts made little progress on those messages for several years – until some breakthrough in 1953 by one Samuel Chew.

The authors then turn to the case of the 1941 charred codebooks retrieved from Petsamo. Their account has it that the Germans obtained the first copy from the Finns, and a US Army team salvaged it in May 1945. “The book assisted Gardner in reconstructing part of the codebook for the earlier years,” they write. But what does this mean? When did the Army hand it over? When did this further reconstruction occur? What ‘earlier years’ are these? Was the 1941 NKVD codebook exactly the same as that used in 1943, or did it simply have some common properties? The authors merely state that the reconstruction was not so complete as it was for the 1943-1946 period, which meant fewer decryptions were accomplished. ‘STELLA POLARIS’ does not appear in their Index.

They next explain that this codebook was different from the one obtained by the OSS in late 1944, and that it was eventually handed back to the Soviets on the orders of State Secretary Stettinius. For some reason, Haynes and Klehr believe a copy was not made before the return, but they do point out in an Endnote that ‘before the NSA released Venona in 1995 and provided accurate information about the history of the project, one rumor (reported in several books, including one by these authors) erroneously conflated the 1944 OSS Finnish material with that obtained by Army intelligence in 1945.’ (They never mention Tiltman’s experience with the same Petsamo material.)

And that is all. No mention of Canberra traffic, or cribs. Thus we are left with the paradoxical conclusion that the NSA was able to achieve far more decryption for the years 1943-1946, when they did not have access to a relevant codebook, than for the years 1942-1943, when they did. The authors summarize those facts as follows:

Venona uncovered, in whole or in part, roughly half (49 percent) of the messages sent in 1944 between the KGB New York Office and its Moscow headquarters, but only 15 percent of the messages from 1943 and a mere 1.8 percent of messages from 1942 (only twenty-three out of nearly thirteen hundred). Only 1.5 percent of the 1945 traffic between the KGB Washington office and Moscow was deciphered.

I believe these statistics cry out for some explanation. If the flawed OTPs were distributed in 1942, why was the success rate for 1943 so much lower than for 1944? Was the fall-off for the whole of 1945 due exclusively to new OTPs being distributed? When did the latter come into use? The process by which those valuable codegroups for the 1943-1946 traffic were resolved is left opaque. The authors’ focus is, admittedly, America, but it would have been useful for them to present the corresponding figures for other countries.

1. The Haunted Wood by Allen Weinstein & Alexander Vassiliev (1999)

Ther authors were able to exploit the thaw in relations between the United States and the Russian Federation to inspect, between 1994 and 1946, the archival records of the Foreign Intelligence Service (the SVR). Payments were made. While the book provides a rich confirmation of Soviet espionage from the Moscow side, one of its main contributions is to identify a memorandum describing Weisband’s work, and how the information he provided led the USSR’s ‘state security organs’ to carry out a number of defensive measures that frustrated the Americans’ ability to decipher and analyze its secret messages. The memorandum is not dated, but probably was written around August 1948.

1. The Venona Secrets: The Definitive Exposé of Soviet Espionage in America by Herbert Romerstein and Eric Breindel (2000)

This book has nothing to say about the mechanisms of decryption. It is a guide to the historical background, and an analysis of the spy-rings.

1.  GCHQ by Richard Aldrich (2010)

Aldrich dedicates his Chapter 4 to Venona. He relates the familiar background narrative in an engaging way, but has Gardner joining the project in 1944 – two years earlier than as advertised by Haynes and Klehr. Phillips joined him on Soviet traffic in May 1944, where he ‘quickly realised the scale of duplication, and made a number of progressions that led to wider breaks in the cypher system used by the KGB’. No useful insights there. The December 1946 message listing scientists on the Manhattan Project is featured. And then Aldrich turns mysterious: “Early accounts of Venona suggested that the first breaks were achieved as a result of the recovery of a partly burned Soviet codebook found in Finland and sold to America’s wartime intelligence agency. Stories have long circulated about how American diplomats insisted that protocol required that it be returned to the Soviets.”

This is typical Aldrichian elision, and is not good historiography, with a surplus of rumours. What are those ‘early accounts’? He does not say: no Endnote explains. He provides no dating for the codebook ‘found in Finland’ (or discovered in Germany). ‘Stories have circulated’ – where? How does Aldrich treat them? It appears to be of no importance to him. He continues: “In fact, up until 1952, the progress made on Venona was probably driven by the pure sweat of mathematics, and represented a remarkable intellectual achievement.” Probably? No use of computers, or cribs? Where did those ‘early accounts’ come from, then, and since it would appear to offer a useful explanation as to how progress had been made so well? Without offering any evidence to the contrary, Aldrich just abandons those early stories. Yet he then introduces the second charred codebook, retrieved by TICOM, informing us that it was only in 1953 that the link was made between the KGB system they were working on and the codebook they had owned since 1945. It beggars belief. All these smart people, and they could not work out what was under their eyes? Aldrich offers no judgment.

Aldrich reinforces the notion of earlier GB/USA co-operation, claiming that Britain learned of the project as early as August 1945, and that John Tiltman, head of the Cryptographic Group at Eastcote was kept informed of progress. Full cooperation came a little later, when Phillips spent six months at Eastcote, working with Philip Howse. Aldrich comes up with the useful insight that the recognition that the Australian traffic was KGB, rather than low-level consular material, did not occur until December 1947, following a visit from Gardner. He then moves on to Weisband’s role, attributing to him (quoting The Haunted Wood) the revelation that the Soviets changed all their systems on the ‘Black Friday’ of October 29, 1948. Not only were new security procedures implemented for the KGB; medium-grade communications between Army, Navy, Air Force and Police units were encrypted for the first time. But those systems were not VENONA.

The author has more to say about the ramifications, especially for Australia, but the above is all he writes about decryption itself.

1. Code Warriors by Stephen Budiansky (2016)

Budiansky, who is an expert in cryptology, dedicates two chapters of his book to what he calls ‘the Russia problem’. As a sample of how comfortable he is in describing precisely some of the technical processes, I present the following:

            By October 1943 Arlington Hall had up and running a semi-automated decryption processing line for Japanese army traffic that punched incoming teleprinter messages onto paper tape, converted the paper tape to IBM cards, matched the resulting decks of punch cards with other sets of cards pinched with the corresponding sequence of cipher key, subtracted one from the other to reveal the underlying codegroups and punched those on a third set of cards, and then used a library of cards containing codegroups whose dictionary meanings had been recovered to print out the complete decoded message.

The first conclusion from this assessment is that a sophisticated electromechanical system was at work, building on the experiences with bombes at Bletchley Park and in Washington. It was not all ‘intellectual sweat’. A system was in place to prepare for VENONA traffic.

The author points to early cooperation with the British. William F. Friedman had recommended as early as July 29, 1943, that they be given the Soviet diplomatic traffic to handle, and a Captain Stevens, the British liaison officer at Arlington, had the previous year picked up hints that the Americans were keen to re-address the Russian problem. Budiansky points out that the Finns had made progress on Soviet OTP diplomatic ciphers, and that they were sharing these results with the Japanese. ‘A series of lengthy cables offered a wealth of basic technical details about the Russian systems’, he writes (perhaps not the most precise sentence in his book – no doubt Helsinki-Tokyo traffic was being intercepted, but it would have been useful to provide deeper information on what such messages revealed.) An important technique learned was the method by which encipherers informed their counterparts which OTP page they were using for a transmission. Rather clumsily, the ‘indicator group’ was taken from the first set of numbers of the deployed OTP: that helped Arlington Hall considerably after they discovered the duplicated pads.

He also reveals that a direct radio teleprinter link was established between the Pentagon and Moscow, in order to improve quality transmission over the flimsier radio link that the cable companies operated across the North Pole. The Russians were not aware that the set-up allowed a teleprinter at Arlington to copy the Soviet traffic automatically. “For several years it would prove the most important source of enciphered Russian traffic available to the American codebreakers”, he writes. Regrettably he does not explain why that was so.

Budiansky then explains how the inspection of the Japanese messages helped Arlington to deconstruct the codes. It led them to confirm the two-step process of a codebook that held values that were normally four digits long. “To each of the code groups in a message to be transmitted, a second set of digits, drawn in sequence from a book or pad [sic!] containing random numerical groups of ‘additive’ (or ‘additive key’) was then added,”, he writes. Yet even Budiansky, whose prose is normally crisp and clear, does not explicitly unravel in one place the mysteries behind the various systems, and how they were simplified by the Arlington cryptanalysts.

I noted above the casual reference to a ‘book or pad’, which I believe is an important factor. One traditional method of performing encoding was for the transmitter and the receiver both to own an obscure book, the pages of which could be used to identify keywords to enable encipherment. (Foote used such a system in Switzerland, and the KGB/GRU used such a system as back-up, as Gouzenko revealed.) This was a low-maintenance system, as it required no updating or distribution of fresh materials. Yet it had exposures, as identical ‘additive key’ might well be used by chance in different encryption exercises, and the identity of the book might be betrayed somehow. Using a One-Time-Pad was a much tighter concept: indeed, when OTPs are used correctly, they are practically impossible to break. On the other hand, they require constant maintenance and refreshment. Proper random-number generation exercises have to be carried out. The Soviets were not alone. Randy Rezabek, exploiting TICOM archives, states that the US Army Signal Intelligence Service was able to exploit the Germans’ failure with OTPs when they used a machine that created repeating patterns after a long series. Then the pads have to be printed, bound, and distributed, a strenuous project, especially in time of war. The Soviet Union was very constrained in enabling ships to depart from Murmansk in 1942 and 1943 with that precious cargo. And that is where the KGB fell down.

Another critical aspect is the shape of the codebook. I referred earlier to the paradox that Arlington Hall was able to make better progress on the 1943-45 traffic, when they had no codebook support, than it was ever able to on the 1939-43 transmissions, when they (eventually) did. In Appendix A, Budiansky explains that codebooks came in one of two forms – a one-part code, and a two-part code. This is a very important point. The first was simpler, allowing encipherer and decipherer to use the same document, which would consist of a list of words and their equivalent numerical values. Thus it might appear as follows:

A                     0001

Aardvark         0002

Abacus            0003, etc.

In other words, no randomization, with both lists sorted, and an easy look-up process at both ends. On the other hand, a two-part code would be more complicated, and look something like this:

A                     5487

Aardvark         1946

Abacus            3305, etc., etc.

with the corollary (say)

0001                Embassy

0002                Meeting

0003                Tomorrow, etc., etc.

Such a system requires two books to be created, so that the decipherer can look up from a less significant string of numbers what the term behind it is. Again, more work has to be done when creating a new codebook. But the vital aspect for the cryptanalyst was that, with a knowledge of the Russian source language, and using imagination over the context of the messages, it was easier for a book breaker to fill in missing entries in a one-part code, since they appeared in alphabetical order. The POBEDA codebook was much more intractable.

As Budiansky informs us:

The Jade codebook used with the 1944 and 1945 NKGB one-time-pad messages (also known as Code 2A by Arlington Hall) was a one-part code, and was recovered entirely through Meredith Gardner’s book breaking without ever seeing the original. Code 1B, the NKGB codebook that the Russians called Kod Pobeda and which was used from 1939 to November 1943, was a two-part code, and the recovery of a copy of most of the original book by TICOM Team 3 played a significant part in the effort at NSA beginning in the mid-1950s to break most of the 1943 messages.

From this I deduce first, that the charred 1941 codebook retrieved was indeed valid for the whole period of the war up until 1943, when the new codebook was introduced. Under pressure, the Soviets introduced this weaker new codebook at a time when security should have been tightened – perhaps a counterintuitive measure when one considers how technology is supposed to improve with the next release. It perhaps serves to support Cecil Phillip’s renowned statement that the breakthrough of 1943 to 1946 was ‘a purely analytical accomplishment, achieved without the benefit of either Soviet codebooks or plaintext copies of original messages’. And Budiansky sheds much light on how that breakthrough was achieved. His reference to ‘1B’ suggests that an initial ‘1A’ system must have been deployed beforehand – presumably before 1939 – but he frustratingly writes nothing about it.

Another conclusion that I think is implicitly rather than explicitly made is that the trade and diplomatic traffic shared both codebooks and OTPs, and that it was a critical factor in allowing Gardner to transfer knowledge gained from the inspection of more formulaic trade messages to the study of the diplomatic and intelligence messages, which had fewer standardized pointers. (Recall my sceptical comments above.) Arlington Hall also learned much from Gouzenko after his defection in September 1945. Unfortunately, Gouzenko had handled only GRU traffic, which used a different codebook from the ZET trade and ZDJ diplomatic systems, but, as Budiansky writes, ‘some of the details he was able to supply about the construction of the GRU codebook probably applied to those other systems as well’. Gouzenko told his inquisitors about the one-part code that the GRU used (thus confirming what West wrote), as well as the system used to translate non-Russian vocabulary by means of a special code group that would indicate a ‘begin spell’ routine. The repetition of such a stream was, in itself, another important aid.

And that is how such rapid strides were able to be made by Gardner in 1946, all provoked by the chance discovery by Richard Hallock in 1943 that some OTP pages had been duplicated. There is more that could be said about the techniques used in establishing the existence of repetitions (‘depths’), and I recommend readers to go back to Code Warriors if they are interested in learning more. Budiansky’s book overall performs well the job that all previous attempts to explain VENONA have failed to do.

1. Spying Through a Glass Darkly: American Espionage against the Soviet Union, 1945-1946 by David Alvarez and Eduard Mark (2016)

I acquired this book because I imagined that the two serious historians might have been able to exploit recently released archival material to clarify the story. In Chapter 2 (‘A Mystery in an Enigma’) they carry out a careful inspection of the VENONA events, but it turns out that they rely almost exclusively on the familiar sources, namely Benson and Phillips, Aid (see below), Smith, Aldrich, Andrew (with Mitrokhin or Gordievsky), sprinkled with some correspondence between government officials that adds little to the grand picture. Indeed, the authors lament, several times, the fact that crucial records have not been released. As an example:

Much about the STELLA POLARIS affair remains obscure, including the exact contents of the purchase, their distribution within the American intelligence community, and their contribution to developing an intelligence picture of the Soviet Union in 1945.

Indeed. And maybe deliberately so.

While Alvarez and Mark give a thorough account of the hesitations and deceptions that were embodied in the negotiations between OSS, the Finns, and the State Department, they appear somewhat confused over the contents and disposition of the STELLA POLARIS material. They report that the NSA’s records contain ‘a long list of Soviet codes, ciphers, and other cryptographic materials under the heading “STELLA POLARIS/Source 267”’, but they add that it may well represent copies of the OSS collection, even though the official historians denied receiving anything from the OSS. Yet they then explain that ‘Source 267’was in fact GCHQ, handing over in the spring of 1946 all that it had acquired from the Finns. They also note that the OSS station in Stockholm did indeed copy the materials before sending them to Washington. Their conclusion at this juncture, however, runs as follows: “If, as seems almost certain, Arlington Hall did not receive the STELLA POLARIS records until after the war, the acquisition would have largely duplicated material already in the code breakers’ hands”.

I do not know what to make of this. The authors cannot be referring to Source 267, whose transfer occurred in 1946. It looks as if they are referring to the TICOM acquisition, since the raids of May and June 1945 just slipped in before the end of the war (although Victory in Europe had already been declared), and they cite Benson and Warner for asserting that the TICOM acquisition made the following STELLA POLARIS trove superfluous. Yet Benson himself stated that the TICOM material was not able to be exploited until the early 1950s. Alvarez and Mark again express their frustration, writing that ‘the continued classification of postwar American and British communications intelligence records makes it difficult to determine the impact of the foreign materials after the war’. Yet, while they provide a deep analysis of how the TICOM exercise helped with the decryption of Soviet domestic traffic (i.e. non-VENONA, such as in Operation CAVIAR), they never even register that the information on diplomatic codes was not exploited until the early 1950s. Thus they never pose the question: “Why, if the diplomatic codes were part of the package, and were of use in 1953, were they not taken advantage of in 1946?” It is all very bizarre.

1. Agent Link: The Spy Erased From History, by Raymond J. Batvinis (2024)

This profile by Batvinis (a retired FBI agent, and notorious chairman of the FBI’s inquisition into Hollis as Soviet spy) is a discursive work that recapitulates stories of Soviet espionage that go far beyond Weisband himself. It contains some useful insights, but also much vague description that could surely have been tightened up by now. References to VENONA are scattered round the central chapters of the book, but they are frequently annoyingly imprecise. Moreover, his publication contains no Index, which is unpardonable, in my opinion.

For example, he has a couple of pages on the STELLA POLARIS incident, but never describes whereof the purchased consignment of material consisted. He beguilingly reports an ‘unprecedented level of collaboration underway with Bletchley Park’ as early as 1943, but he does not spell out why he contradicts the official accounts. And then an important, but unsourced, observation appears, namely that the Japanese had picked clues from the externals of Soviet traffic: “The Russians used the first and second digit of the first cipher group of the text in a message to signal the length of the message while the fourth and fifth digit [sic] identified the specific additive page used to encrypt the message.” He adds some useful facts about the explosive growth of IBM equipment, including custom-built machinery, between 1943 and 1945. He rewrites extracts from Benson’s dossier of memoranda in a more natural English way (and, along the way, introduces new personalities, such as Genevieve Grotjan (presumably née Feinstein), although his references to page numbers of Benson’s work do not correspond to the published version.

Batvinis spends several paragraphs on the TICOM windfall of the summer of 1945, but he never compares the source material to the STELLA POLARIS trove. He writes:

What Allied troops found was the complete German foreign Ministry Service Signals Archive, which included a number of Russian codes and ciphers. There were important records that Russian consulate officials in Helsinki hastily attempted to burn as they were fleeing when the Finnish government sided with the Axis in 1941. The collection, turned over to the Germans, also produced a KGB codebook called Kod Pobeda, a one-time pad series scheduled for use between 1939 and 1943, replaced when a new system called JADE was introduced.

This is just sloppy: ‘Kod Pobeda’ was a codebook, not ‘a one-time pad series’. Moreover, who turned it over to the Germans? When? And why Helsinki, not Petsamo? Was this material equivalent to the Hallamaa set? And ‘scheduled’ for use between 1939 and 1943? Why would an end-point have been decided in advance? One expects greater precision from a professional historian. He does, however, add some useful details about intelligence gained from interrogating German POWs.

Oral histories are part of Batvinis’s sources. I had not noticed these being used much beforehand, but he provides some extracts from Frank Rowlett’s oral history (see https://www.governmentattic.org/50docs/NSA2ohiFrank_Rowlett1983_1985.pdf for the probable source). Rowlett was the post-war head of ASA’s Operations Division, and had been the lead officer who interrogated Gouzenko. Yet Batvinis’s choices are enigmatic: I sense that the following, said to describe the code clerk’s method of working (derived from Gouzenko) is important, but cannot work out exactly what it means:

By examining the “external characteristics”, a trained eye could discern clue such as intentional scratching out of an indicator and replacing it with something else. Careful attention to columns of numbers and letters for even tiny alterations could signal important leads into the actual length of the column, or, more critically the length of the key, both of which would have significant implications for a code breaker’s chance of success. It would pay off handsomely, not at the time, but in the not-too-distant future.

How these conclusions could be reached through the analysis of encoded messages is beyond me.

Batvinis’s description of how Gardner made his breakthrough with the codebook in 1946 is not convincing. Using what Gardner later told an interviewer (but again unsourced), Batvinis reports that Gardner, having identified prepositions and conjunctions, was able through ‘basic instinct and informed guesses’ to determine what word would logically follow a certain word, and thereby slowly began building a vocabulary of Russian nouns and verbs. This is far too much of a leap in imagination for me to swallow. He then mentions Lamphere’s introduction to Gardner, and how the FBI agent was fortuitously able, in the autumn of 1947 [sic: not 1948], to have his FBI colleagues send stacks of the plaintext messages that had been photographed by the FBI in 1944 during a break-in of Amtorg. He takes this story directly from Lamphere.

Lastly, I believe that Batvinis, as do others, mixes up the demise of VENONA with the Black Friday events. The Soviet systems that were suddenly made more secure and impenetrable were not VENONA, and were known by such names as Operation Taber and Operation Shamrock. Weisband had not passed on much of significance on VENONA but he had indeed been the source of intelligence to Moscow that the Americans were tracking and interpreting its domestic communications systems. The fact is that the new OTPs had already closed off all diplomatic traffic (with the exception of Australia) from Western eyes, and Australia was the last outlier to fall in August 1948.

• Academic articles:

A sprinkling of articles published between 1997 and 2002 provides a mixture of revelations – as much because of what they fail to say, as to what new insights they bring.

1. ‘Venona and beyond: Thoughts on work undone’ by Michael Warner and Robert Louis Benson (Intelligence and National Security, Volume 12, Number 3, July 1997)

This is a somewhat supererogatory offering by the authors of Book 5, above (with the names of the authors transposed). At the time of publication, Warner was Acting Chief of the CIA History Staff, while Benson merely ‘served with the Office of Security of the National Security Agency’. The article brings very little new to the table, but it does pose a lot of unanswered questions. Indeed, the point of it seems to be to urge researchers to apply close scrutiny to the many loose ends uncovered. In particular, they mention the desire to have the identities of far more covernames revealed. They also express bewilderment over the errors made by the Soviet cryptologists, but conclude their rhetoric merely by saying that ‘the answers presumably reside somewhere in the former Soviet Union’, an assertion at which anyone familiar with the events could have arrived.

One fresh observation they make is to praise Lamphere’s book on the FBI response, without commenting on the more controversial of that author’s claims. They speculate whether Elizabeth Bentley’s revelations to the FBI in 1946 did more damage to Soviet counter-intelligence than did VENONA. Rather disturbingly, they attempt to debunk the theory that the acquisition of a KGB  codebook ‘captured in Finland and purchased in 1944 by the OSS’ has any merit, claiming (again) that the cryptanalytic breakthrough predated that acquisition (and thus implicitly rejecting what Lamphere and others wrote). Here they hedge a little: “General William J. Donovan surely [sic] copied the material of 1,500 pages before ‘returning’ them to Andrey Gromyko”. (The material was never ‘owned’ by the Soviet Union in that form.) They express ignorance as to what happened with those records thereafter. “It is entirely possible that the 1,500 pages that OSS reputedly gave American cryptanalysts contained significant information about Soviet codes and ciphers, but information that was not useful with the particular systems read by the Venona program”, they write. Yet they refer to a 1946 document recently released by the NSA that does claim that among the material handed over by the Finns was a four-digit system used by the NKVD between 1941 and 1943? Could it be the POBEDA codebook familiar to the Venona cryptologists? One might expect the pair to answer this question, but they leave it hanging there.

They conclude by raising a number of historical questions concerning Stalin’s intentions and abilities if no successful decryption had taken place, and spies had not been unmasked. All very well, but hardly valuable fodder. “  . . . serious research with the Venona translations may take a generation to produce its deepest and most lasting insights”, they write. Yet, in truth, almost thirty years later, very little new has evolved – apart from those brief insights from the Soviet archives, which mainly served to confirm what figures like Klehr, Haynes and Weinstein had derived from their studies.

• ‘Debris from Stella Polaris: A footnote to the CIA-NSA account of Venona’ by C. G. McKay (Intelligence and National Security, Volume 14, Number 2, Summer 1999)

Two years later, C. G. McKay responded to the above article in a short piece. McKay is not described, but I know him as joint author of a work on Swedish Signals Intelligence, and he published a perceptive essay titled British Sigint and the Bear in 1997. His intention was to show how the recent declassification of some papers held by the NSA shed light on the STELLA  POLARIS operation. He highlights Major Hallamaa’s ingenuity in succeeding in selling his trove to the Japanese, the Americans, the British, and, later, the French. He refers to the material sold to the French (which he calls Document F) as that which Benson and Warner described, but he categorizes as the ‘star of the show’ a series of progress reports from Finnish Radio Intelligence dealing with several Soviet Union and NKVD codes (Document G). He claims that the original G was a part of the STELLA POLARIS archive – unlike F – but he puzzles over the relationship between G and the Finnish original, and when it came in the hands of the Americans. He wonders whether G was acquired in another later transaction, i.e. not the one where Stettinius intervened. For some reason, McKay seems to think that that distinction is very important, but I admit I failed to understand the significance.

• ‘From Petsamo to Venona Intelligence: Services in the Nordic Countries from Hot War to Cold War’ by Tore Pryser (Scandinavian Journal of History, Volume 24 (1), 1999)

Pryser, having access to insider information, makes some startling statements. The three assailants at the Petsamo consulate, were in fact Norwegians working for the Abwehr, and were on a mission to retrieve Abwehr [sic] archival material stored there. (That enigma is not explored.) Thus the codebook they discovered, which the Soviets did not have time to burn, was immediately available to the Abwehr unit on the north front, and it used that codebook to crack Soviet messages. What is even more extraordinary, according to Pryser, is that Edmund Sala, who led the Abwehr unit, in 1944 retreated to Lillehammer and then traded with Swedish intelligence, the C- Bureau, giving them what he had obtained at Petsamo. The Swedes then added it to what Finnish intelligence had gathered to create the STELLA POLARIS material, and then horse-traded with the Germans behind the backs of the Norwegians and the British. The result of this was that German specialists on the Soviet Union were transferred to the OSS in Frankfurt. Of course, this account explains much more clearly how the Germans knew about the Soviet codes, and Pryser’s Footnotes indicate that Sala must have passed them on to Berlin, but it is a remarkably different tale, with its emphasis on the Swedes initiating the transactions, from the conventional one that highlights the achievements of the Finns.

Pryser cites Pavel Sudoplatov (author of Special Tasks, the name of the unit the KGB officer headed during the war) as saying that the codebook was subsequently changed, writing: “He therefore claimed many years later that it would have been virtually impossible for the FBI to infiltrate the Soviet agent network in the United States with the aid of the codebook found in Petsamo.” Yet Pryser attempts to disprove that statement, writing that ‘the NSA confirmed that the Petsamo material was decisive for being able to decipher the coded telegrams sent to Soviet GRU and NKVD agents in the USA in the years 1943–1945.’ Nowhere does NSA state that: quite the contrary, in fact, since they dismiss the whole STELLA POLARIS trove as inconsequential in their efforts. Pryser misses the subtlety in the careful NSA statement: the NSA wanted to credit the TICOM expedition, rather than the OSS operation, for bringing in the Petsamo trove. In fact, Pryser cites Peter Wright in Spycatcher for this insight that the find had ’enormous significance’ for the identification of Soviet agents: while Wright referred to a codebook retrieved on the battlefield (a clear pointer to Tiltman’s loot), Pryser, who knew nothing about Tiltman, assumes that the battlefield codebook and the Petsamo codebook must be the same entity.

Incidentally, I do not know why Pryser inserted ‘many years later’ into his text. Page 218 of Special Tasks lays out all of Sudoplatov’s thoughts on the matter, including his disbelief that codebreaking played any role in the unmasking of spies! Pryser does also remind his readers that the 1993 book, Deadly Illusions, by John Costello and Oleg Tsarev, disclosed that Arvid Harnack (CORSICAN) alerted his Moscow bosses that the codebooks had fallen into enemy hands, and he quotes a message from Anatoly Gurevich (KENT), received by the GRU in November 1941 that included the sentence: “The Germans possess the USSR’s diplomatic cypher, which was captured in Petsamo, however, the cypher has reportedly not yet yielded to the extent that there is an opportunity to decypher any large volume of Soviet documents.” Yet, if some messages had been deciphered, that should have caused a large dent in Sudoplatov’s confidence.

• ‘Tore Pryser’s Article ‘From Petsamo to Venona’ by Olav Riste (Scandinavian Journal of History, Volume 24 (3-4), 1999)

Olav Riste, in a short follow-up the same year, argues that Pryser overstates the importance of the Petsamo codebook to VENONA. He merely echoes the NSA official story, however, namely that Arlington Hall’s 1944-1946 achievement was a purely analytic accomplishment, and that the Petsamo codebook, although it came into American hands in 1945, was applied to the project not until 1953, after the VENONA project had unmasked so many atomic spies. Riste never mentions TICOM, or the extraordinary conundrum of the Americans’ sitting on such valuable material for eight years. Instead he points to the Soviets’ dilatoriness in not replacing the codebook until 1943, even though, in his short article, he had beforehand explained that access to a codebook would solve nothing if the OTPs were used properly. He does not seem to be aware of the duplicated pages problem the Soviets had. In other words, Riste merely stirs the soup. I do not know whether any correspondence was exchanged afterwards, but both Pryser and Riste were obviously much confused.

• ‘“Venona” – what we really knew during the cold war’ by Nigel West (The RUSI Journal, 146:1, 2000)

This article is based on a presentation that West made at RUSI (The Royal United Services Institute) in 2000. Again, it presents very little new information. Who ‘we’ were is not stated, nor is the precise time to which West is referring: the paper seems more designed to promote his book. What he does divulge is that MI5 required him never to mention VENONA, to which he replied that a retired officer ‘living in Tasmania’ would soon be writing a book about it. His interlocutor assured him that Peter Wright would be doing no such thing, and he then ordered West to remove Wright’s name from his book as well. So it is evident where much of West’s material came from. And that is why he had to refer to VENONA as ‘U Traffic’.

There is little new in what follows. West does dedicate a long passage covering the Australian angle, and how the ‘contemporaneous’ decryption of the Canberra to Moscow traffic enabled ‘huge’ counter-intelligence operations disclosing the existence of ‘a gigantic Soviet espionage network’ in Australia. He misrepresents the failure of the cover story, however, claiming that the Australians accepted it, and that they did not discover until years later that the source was SIGINT. He does have some useful input on how VENONA was circulated, and how the decrypted texts gradually improved, and reminds us again that that process has made it impossible for historians to track the process of decryption of individual cables. He expresses surprise that the United States lost interest in the project in the 1950s, and states that it was British tenacity that pursued it, resulting in their successful decryption of some London to Moscow GRU traffic in the 1960s. He provides some evidence of his sleuthing to pin down the identities of INTELLIGENTSIA (J. B. S. Haldane) and NOBILITY (Ivor Montagu), but those disclosures had already appeared in his book. He draws attention to the fact that Mitrokhin identified TINA as Melita Norwood, and, like Warner and Benson, implores his audience to study the VENONA traffic in more depth to uncover further spy names.

• ‘“Stella Polaris” and the secret code battle in postwar Europe’ by Mathew Aid, Intelligence and National Security, Volume 17, Number 3, Autumn 2002

This is a seventy-page article (including 327 Endnotes) that explores in depth the STELLA POLARIS events. Aid (who died in 2018) was an American military historian. It consists of a comprehensive analysis of the September 1944 retreat into Sweden of the Finnish Intelligence Group, including, most importantly, Major Hallamaa and his Sigint Section, the operation known as STELLA POLARIS. I shall not attempt to summarize the whole story, but I do remark that the entrepreneurial Hallamaa succeeded in causing some havoc among several foreign intelligence organizations.

The story does not start auspiciously. Aid echoes the anecdote whereby Roosevelt forbad OSS’s Donovan from retaining ‘Russian code materials’ obtained from the Finns [sic, not ‘Swedes’], and required him to return them to the Soviet Embassy. Again, I point out that these were not all native Soviet documents that inadvertently ended up with the Americans. They were a complex set of materials derived from multiple sources: they would have been a massive eye-opener for Moscow. Aid then continues with the arresting claim: “The documents [declassified materials from the post-war Strategic Services Unit, or SSU, the intelligence section peeled off from the OSS in September 1945 before the CIA was created] reveal that throughout World War II, the OSS secretly obtained volumes of sensitive intelligence information concerning America’s wartime ally, the Soviet Union, from agents within the Finnish intelligence service.” This comment is, again, troublesome. World War II started in September 1939. The USA entered the war in November 1941. The OSS was not created until June 13, 1942. A strong assertion is weakened by a sloppy rendering of the chronological framework. This error is repeated twice later, where Aid claims that the OSS had been spying on the Soviet Union throughout the war. Aid also explains that most of the intelligence that the OSS gained came from Swedish sources, not Finnish ones – at least until early August 1944, when Hallamaa made overtures to Tikander.

When Hallamaa, soon after his arrival in Sweden, began seeking purchasers for the trove of intelligence material he had brought with him, the OSS’s 75-man station in Stockholm, led by Wilho ‘Ty’ Tikander, was a prime target. He also pointed out that the Finns had been decrypting US State Department traffic, and he then explained that most of their effort had been directed on Soviet codes, ‘of which he claimed they had broken over a thousand’. He showed the OSS some specimens. (This traffic obviously went beyond diplomatic cables, and covered much internal army, police, and other activity within the Soviet Union.) The OSS was keen to acquire the material, but the State Department got wind of it, and protested. Yet much of the intelligence reached Roosevelt’s eyes before the deal was abandoned.

What the OSS did not know was that Hallamaa was selling the same material to the Swedes and the Japanese, i.e. not just to neutrals but to enemies of the Allies. (Finland was technically an enemy country under control of the Nazis, but also mostly fearful of Soviet repercussions in the wake of the Finnish-Soviet war of 1940.) And the Japanese of course informed the Germans. What the OSS had not done was to demand an exclusive purchase – which would have come at a higher price, no doubt. Moreover, when, on December 11, 1944, Donovan defied the State Department edict, and authorized Wikander to purchase the Russian cryptographic material, a remarkable action took place. The local OSS office photostated the material before sending it on to Washington. Thus, irrespective of whether Donovan in Washington photocopied it before handing it over to Gromyko, the Americans retained it. (Aid also writes that ‘SSA officials reportedly photocopied all 1,500 pages of the code materials’, citing a presentation made by Donovan’s executive officer, Edwin Putzell, in 2002), I do not believe this claim about OSS Stockholm’s initiative has been made elsewhere.

A glimpse of what the package contained was gained from interception of wireless traffic from the Japanese military attaché in Stockholm, the notorious General Onodera (who gave many secrets away). Aid reports that the five codes given to the Japanese were ‘a five digit military codebook called 091-A; a five-digit NKVD codebook that was still in use as of November 1944; a four-digit military codebook used by Russian tank units on the Leningrad Front opposite Finland; and a four-digit military codebook used by Russian anti-aircraft units on the Leningrad Front.” This statement is troublesome, however. Aid refers to a ‘five-digit codebook that was still in use in November 1944’, presumably suggesting that the POBEDA book (the victim of Petsamo) endured until then. Yet all other sources indicate that POBEDA was replaced by JADE  in 1943. It seems more probable that Aid accurately identified the proprietorship of the codebook, but was mistaken about its withdrawal.

There is other evidence to support that hypothesis. In Special Tasks Pavel Sudoplatov wrote:

I do not think that American codebreakers played the decisive role in unmasking our espionage effort. In December 1941, our agent Senior Warrant Officer in Berlin reported that the Germans had seized our codebook in Petsamo, Norway, and were trying to decipher our cables. Naturally, we changed our codebooks. By the time the Petsamo book had fallen into American hands we had stopped using it entirely.

Thus one might conclude from Aid’s analysis that the cryptologists at Arlington Park did indeed obtain the full [POBEDA] codebook, soon after the STELLA POLARIS acquisition, but that it was of use to them only when its deployment overlapped with the introduction of the duplicate OTPs. While the Soviets had known of the codebook exposure back in 1941 (and then taken a while to remedy the problem), they would in 1945 have received confirmation that the POBEDA codebook had been revealed, but, more seriously, concluded that all their domestic systems were in a dire shape, as far as security was concerned. That may have been the prompt to set ‘Black Friday’ in motion.

There is obviously much more to tell, and I recommend that avid students locate Aid’s article, to learn more about the deceptions and plotting going on behind the scenes, including episodes involving the abscondment of the Hallamaa tribe to France and, inevitably, Wilfred Dunderdale of MI6. I note, in conclusion, some sentences that Aid writes about the British access to the STELLA POLARIS material:

What MI6 did get was full and complete [??] access to the ‘Stella Polaris’ archives and the intelligence reports generated by the ‘Stella Polaris’ team in Paris. When the ‘Stella Polaris’ microfilm files were removed to London later in 1946, GCHQ took custody of the ‘Stella Polaris’ cryptographic materials. GCHQ, in turn, passed a copy of the Russian codebooks and related materials that it got from the Finns to the Army Security Agency at Arlington Hall Station in Virginia in the spring of 1946, apparently without knowledge of the SSU. The British referred to the material obtained from the Finns by the designator Source 267.

So, even if the ASA did not get the materials from the OSS, they got them from GCHQ.

• Intermediate Summary

Despite the overwhelming fog, some framework of a story has gradually evolved. Yet it is remarkable how smoothly the more official histories have simply ignored the earlier anecdotal accounts, with their suggestions of assistance from acquired codebooks. One might expect a more coherent story gradually to be built up over time, but the NSA/CIA volumes have not been updated, and the later publications fail to address the obvious paradoxes in the literature. I present a temporary précis of the less controversial aspects:

The Japanese traffic was highly useful in providing hints to message indicators. The serendipity discovery of duplicate OTPs in 1944 was the first major breakthrough. Trade and diplomatic traffic shared the same codebook, as well as faulty OTPs. Formulaic trade messages assisted in the process of building an embryonic codebook, and Gouzenko’s insights helped Arlington Hall understand the generic structure of codebooks and the use of ‘begin-spell’ techniques. Gardner’s knowledge of Russian sped up the process of filling out the codebook. The 1943-1946 traffic was easier to decrypt because it used a one-part codebook, introduced in 1943 after the Soviets learned in late 1941 that the POBEDA codebook had been compromised. Cribs dramatically improved the process during 1946-1948. Earlier traffic could not be addressed so well until the codebook acquired from the Finns/Germans in the TICOM operation was exploited – surprisingly late – in 1952. Most traffic after 1945 was undecipherable, with the Canberra-Moscow/Moscow-Canberra channels becoming a hold-out until the summer of 1948.

One could also lay out five phases of Codebook/OTP coexistence, according to my modification to the method by which Budiansky delineated them. (He does not register my 1C phase):

1A: Pre-1939 (not described)

1B: 1939-1942 ‘POBEDA’ two-part codebook and proper OTPs

1C: May 1942-November 1943 ‘POBEDA’ codebook, and introduction of duplicate OTPs

2A: November 1943-1945 ‘JADE’ codebook and some duplicated OTPs

2B: 1945-1948 ‘JADE’ codebook (probably), and new OTPs (except for Australia)

One has to map onto these the probable timing of the Soviets’ reactions to evidence or rumour that their systems were under stress, namely the 1943 story that a copy of the POBEDA codebook had been obtained, the intelligence gleaned from Currie, Philby and Weisband about progress in decryption, the break-in at the consulate(s) in New York (and Washington) in 1944, the 1945 ‘return’ of the STELLA POLARIS material, the surrender to the FBI by Elizabeth Bentley in November 1945, and the indication in 1948 that the Canberra-Moscow channel was exposed.

Yet questions remain. Why did Tiltman apparently not offer his Petsamo codebook to the team? Was Gardner really able to make such spectacular progress in 1946 without any other help? Can the unofficial stories be trusted? Which codebook did Gardner have displayed in his office? Why has the matter of his codebook, seen by Lamphere, been largely overlooked? Who initiated the consulate break-in(s), and why were their troves not exploited for so long? Were any other cribs overlooked? Why are the accounts of the retrieval of codebooks so confused? Why did the NSA ignore the STELLA POLARIS material from OSS and take so long to process the TICOM trove? Why did the KGB introduce a weaker codebook in 1943? When did Moscow discover that it had issued OTPs in duplicate?

• What special factors contributed to, or hindered, the decryption of Soviet traffic?

I see three major threads of activity, but it is not clear how they interact with each other, or how they influence events. The first thread consists of the activities of the incidental personnel who contributed through the supply of codebooks at various times (whether charred, with bullet-holes, or intact), and when actions were taken (or not taken) on such events. Much of this is not explicit, and the accounts are contradictory, so it is difficult to determine (for example) exactly what books were available to whom at what time. The second thread is the progress and milestones of the NSA (and GCHQ) cryptanalysts: when they started work on certain traffic, what they accomplished, what aids they received, in the form of cribs, legitimately acquired in messages, or through purloined material. The problem is that the texts of the telegrams that were (if only partially) decrypted, and were published in 1995, give no indication of the evolution of those translations, so it is impossible to determine (unless one of the official accounts makes it explicit) what intelligence was gained to pursue leads at what time. The third thread (which I analyze as a separate section) is the activities of the Soviets: when they heard stories about unauthorized acquisition of cryptographic materials, or rumours of cryptological breakthroughs, what the substance of the warnings was, how they reacted, when they concluded that they had reused OTPs, when they changed codebooks and when pads. The primary insights into these matters come from defectors (i.e. Gouzenko and Petrov), but the story is inevitably very murky.

1. Codebooks

The first puzzle is the matter of the codebooks. Codebooks were an essential part of the cryptographic process. When a message was being prepared for encryption, the cipher clerk would look up words and expressions in a dictionary that gave a five-number (or, in some versions ‘four-number’) code for the item to be represented. If a word did not appear in the codebook, and had to be spelled out, the codebook provided an entry (sometimes multiple alternate entries) for each letter, especially if it were not part of a Russian word. Thus the use of a codebook was the first step in concealment of the text: even if the eventual enciphered message (namely, adding a number randomly generated to the stream created from the codebook) were somehow broken, it would be very difficult for an analyst to work out what the resulting series of numbers meant unless a substantial part of the codebook had been constructed – or was available by other means.

Yet the NSA continues to maintain that any availability of codebooks had no effect on the cryptanalytical successes of VENONA. Benson and Phillips wrote in their original ‘Umbra’ report: “The fundamental cryptanalytical discoveries and the decryptions through 1952 were not aided by our side having any KGB or GRU code book from anyone.” And Cecil Phillips wrote: “The 1944-46 messages – which yielded the early translations and the bulk of all translations – were recovered over a period of years by Arlington Hall cryptanalysts and decoded from a ‘codebook’ that crypto-linguist Meredith Gardner reconstructed by using classic codebreaking techniques.” Those are bold claims that suggest that there was nothing of use in the codebooks that were retrieved, or that, by the time codebooks were obtained, it was very late in the day. Moreover, it directly contradicts the testimony that Lamphere in particular offered.

It is difficult for a layman to digest these claims. I have no idea what ‘classic codebreaking techniques’ means, but, to my mind, if a cryptanalyst has no plaintext to work from, and does not have access to any skeleton codebook entries, and is faced with an incomprehensible series of numbers that has to have the additive removed before the underlying text can be analyzed, when there no apparent clue to the subject matter exists, and one cannot even apply conventional traffic analysis to the cables, since they all issue from an immobile station, the task would be impossible. It would be like contemplating an empty crossword grid with no clues given. Moreover, the frequent references in the literature to cribs and aids (including a codebook that was – of course – not part of VENONA) belie the whole essence of the Benson/Phillips propaganda. ‘Classic codebreaking techniques’ to me would not include help gained from analogous trade messages sharing similar characteristics, or insights deriving from a defector giving valuable tips as to how special techniques were deployed in creating codebooks.

According to the various accounts, as many as three codebooks had been retrieved from Soviet sources. The first was by senior GC&CS cryptologist John Tiltman. As Keith Jeffery wrote in the authorized history of MI6, of Tiltman’s visit to Finland in April 1940 (according to an internal GC&CS history, actually in early March): “Carr immediately replied in the affirmative and it was arranged that Colonel John Tiltman of GC&CS should travel out to Finland, where he was presented by Hallamaa with a Red Army code-book taken off a dead Russian officer and which ‘bore the marks of a bullet’. GC&CS noted afterwards that it had been ‘of real assistance’ to their cryptographers”. The official report of Tiltman’s visit (at HW 3/151 at the National Archives) states: “It transpired that the Finns were prepared to supply Russian intelligence provided the British paid for their equipment. Tiltman’s trip was followed up by a visit of Admiral Godfrey, the Director of British Naval Intelligence, who visited Finland in September 1940. This was judged to yield ‘very satisfactory’ results.” In the NSA tribute to Tiltman (‘A Giant Among Cryptanalysts’) appears the following paragraph: “In addition to his work coordinating efforts with the French, in March 1940 Tiltman was sent to Finland during the last two weeks of the Russo-Finnish War. He was not able to provide the Finns with much cryptologic help. His sole contribution was to tell them that the Soviet submarines operating in the Baltic were using onetime pads; hence, their messages were unbreakable as long as the system was used correctly, as it was. For his part, he received from the Finns a number of captured Soviet naval codebooks, copies of which would later be provided to the Americans.” (The NSA tribute does not mention that the Finns requested wireless equipment, of which they were very short, as part of the deal, and that Tiltman succeeded in getting the equipment shipped. The statement above from HW 3/151 about ‘the British paying for their equipment’ is bizarre.)

Note that the first account refers to ‘a Red Army code-book’ and the last to ‘Soviet naval codebooks’, an alarming anomaly. The GC&CS history states that the Finns were able to provide the British ‘the current Naval Baltic Codes and some Military codes, the reconstruction of which was sufficiently advanced to make the traffic readable’. Irrespective of those discords, how had the retrieval helped? The contradictions increase, however. Robert Louis Benson of the NSA wrote: “We have seen that GC&CS had a body of Russian military and NKVD crypto material, obtained from the Finns by Colonel Tiltman in 1940. Russian Diplomatic traffic to and from London, which included the KGB and GRU traffic, was passed on international commercial circuits and, from later 1940, on national circuits too.” We now can add ‘military and NKVD’ to the classes of material obtained. While Benson’s second sentence seems inconsequential from the first, it might have relevance.  Benson specifically refers to ‘military and NKVD’, which suggests either a close similarity between the material, or else separate packages holding their own value.

Later in his report, Benson relates how the American Captain Sinkov visited Bletchley Park in 1941, when the British gave him a host of information about Russian systems, such as details on weather ciphers, the major army and air force signalling systems, and call-sign and radio procedures. In a hand-written note to his summary report on Russian systems, Sinkov concluded: “The Russian secret systems utilize a one part code book. These code books are super enciphered using additive, or special tables which vary from day to day”. How did Tiltman acquire that knowledge? We are not told. Yet Sinkov is surely wrong: the codebook in use at the time was a two-part codebook. He obviously knew the difference, and his statement is bewildering. On the other hand, his testimony hints that a comprehensive codebook was shared by all systems, presumably diplomatic, GRU (military), GRU (naval), and NKVD, and that the Americans should have known all about this in 1941. (In the TICOM expedition, as Randy Rezabek informs us in TICOM: the Hunt for Hitler’s Codebreakers, the Germans at HSL Ost told the Americans that the same five-figured codes were indeed used by the Army, Air Force and the NKVD!) Tiltman himself spent a month in Washington in March-April 1942, so surely would have added his personal testimony.

I move on to the second codebook, the item obtained from the Petsamo consulate in 1941, and presumably an integral component of the STELLA POLARIS material. While Tiltman’s codebook ‘bore the marks of a bullet’, that obtained at Petsamo was ‘charred’. Martin confused Petsamo with the battlefield, as did Wright. Johnson was also imprecise: he suggested that some Petsamo materials were charred, but that the codebooks were in good shape. Yet he continued by stating that the ‘charred codebook fragments’ were handed over to Hallamaa of the Finnish Sigint Service. Johnson then suggests that the AFSA (when he must have meant ASA) was handed the codebooks by the British in 1945 or 1946: Aid confirms the fact of that transfer, since he uses the same identifier, ‘Source 267’ (as does Nigel West), and specifically names the beneficiary as the ASA. Benson and Warner acknowledge the existence of the Petsamo codebook, but restrict their comments to the fact that the Soviets were alerted to the Germans’ familiarity with it in 1943. Haynes and Klehr confirm that the copy that the Germans owned had been obtained from the Finns. Batvinis is simply confused.

The evidence is overall clear that a (possibly damaged) version of the POBEDA (two-part) codebook that was in use in 1941, but suspended in 1943, should have been made available to Arlington Hall through the offices of Sinkov (1941), Tiltman (1942), the OSS (1945) and GCHQ (1946). And everything that Lamphere wrote about the charred codebook in Gardner’s office would tend to confirm that supposition. Lamphere linked it to Petsamo, and stated that Gardner, even though the codebook was not ‘current’, had found it ‘immensely useful’ in his efforts to construct the [JADE] codebook. No doubt, for even though the codes themselves would surely have changed, the entries in the codebook, and the principle of representation of letters to be invoked through ‘beginspell’ routines, would have constituted an enormous boost to Gardner’s work.

So was the third codebook really a different animal? This was, of course, the item retrieved through one of the TICOM expeditions. I wrote earlier that Johnson had written that the OSS-Hallamaa set was not the same as the set that the TICOM team acquired shortly after the end of the war, and I had added: “Of course it may have been the same material, simply being the copy that landed up with the Germans.” Moreover, the full TICOM package apparently contained many other items, as well as the valuable codebook itself. Budiansky remarks that ‘most of the codebook’ was retrieved in the TICOM exploit, but he echoes the puzzling standard assertion (as articulated by Aldrich) that the ASA/NSA was not able to start exploiting it until 1953 or so. Budiansky similarly hinted at the alarming lethargy when he wrote (p 69):

The Kod Pobeda book found by the TICOM team was applicable only to NKGB traffic sent before November 1943, and as of 1947 Gardner remained unaware the book had ever been recovered. Thus the effort at Arlington Hall at this point still relied upon pure cryptanalysis and ‘book breaking’ to reconstruct both the one-time-pads and the underlying codebooks, all sight unseen of the originals.

It is odd that he did not pick up Lamphere’s statement about Gardner and his charred codebook, and ignored the fact that the ASA had received a copy of the materials from GCHQ.

A study of the TICOM files by Randy Rezabek in his 2017 book TICOM: the Hunt for Hitler’s Codebreakers makes the behaviour of the ASA look even more erratic. Rezabek shows how German signals intelligence was wastefully deployed among several competing organizations. He describes how more than one copy of the Petsamo trove found its way into the hands of the Germans, specifically the OKW/Chi (the Oberkommando der Wehrmacht/Chiffrierabteilung), the OKW being the Nazi-controlled unit that bypassed the traditional services command, and the OKH/GdNA (the Oberkommando des Heeres/General der Nachrichten Aufklärung). OKW/Chi was never able to make inroads into Soviet diplomatic traffic, because of the OTPs, but Rezabek reported that the interrogation of Oberleutnant Schubert of the OKH on May 24, 1945

  . . . revealed that the Germans had received a captured 5-figure codebook from the Finns and that the Russians had reused a one-time pad, allowing the system to be read for a while. They also had success with three and four figured codebooks re-enciphered with conversion tables. Later, when questioned about Soviet military and agents’ systems, he gave details of codes used by the NKVD and their counter-intelligence SMERSH system.

The implication is clearly that it was the Germans, not the Finns, who had made that breakthrough, and it appears that the OKH had mirrored the experience of ASA in determining the use of duplicate pads, and its temporary existence, yet it is also very clear about codes used by the NKVD, and it is to me astonishing that this intelligence never filtered back to ASA at the time. After all, as Rezabek informs us, the TICOM teams had ASA representation on its groups, including the Bletchley Park ‘notable’ Major John Seaman, described as ‘US Army Security Agency liaison officer to GC&CS’. He also writes that ‘scooped up in the troves of documents from Burgscheidungen, the codebook ended up in the archives of the Army’s ASA and later, at the newly established NSA.’ Moreover, Colonel Bicher, who was prominently engaged with TICOM, had, by November 1945, taken up his duties as Deputy Chief of the Army Security Agency, and Rezabek adds that ‘ASA continued to collect and evaluate its TICOM materials, and in May 1946 published a nine-volume study, European Axis Intelligence in World War II as Revealed by TICOM Investigations  . . .’. Why did no one pick up this valuable material until 1953?

Nevertheless, the official accounts remain very coy about TICOM. It might be because of the absurdly long time it took for the haul to be handed over to the NSA (or for the NSA to investigate it properly), and a reluctance to describe how inter-governmental rivalries prevented its dissemination. Another reason could be that the authorities did not want to admit that the ASA had had the same material for several years already, and preferred to exaggerate the skills of their cryptanalysts – a role that Cecil Phillips played to the hilt, and a story that was earnestly reinforced by Benson. The evidence seems clear to me that Tiltman gave Gardner his captured codebook, and that the ASA, out of its dislike for Bill Donovan and the OSS, tried to bury the contribution he made.

1. Cribs

As I have stated before the notion of ‘cribs’ can be embarrassing to some. It suggests ‘cheating’, as if that had any relevance in cryptanalysis. Yet there is a criminal aspect. Gardner had mentioned to Lamphere that progress in decryption would be very slow without cribs, and then, almost miraculously, the results of the raids conducted by the FBI of Soviet consulates came to light, and they delivered useful material by which Gardner could compare plain texts with their encrypted equivalents. Nigel West emphasized the importance of cribs in his book Sigint Secrets, stating that the best short-cut was always the duplicate text sent in plain language or a known cipher. Yet he also asserted in that book that the break-ins (n.b. plural) in New York ‘helped with low-grade traffic but were not much help with NKVD texts’. As always with West, one has to be guarded: how did he know that? And what does ‘not much help’ mean? That they provided some help? In what way? Very few plaintexts? I do not know.

In VENONA, West wrote, ‘the issue of “black-bag” jobs remains very sensitive’. Indeed. It is very hard to gather reliable information about these operations. Who authorized them? And how did the raiders know how to find the relevant material? Did they leave any traces? Did the Soviets realize what had happened, and did they complain, or were the locals too embarrassed to admit to their superiors in Moscow that their security had been lax? Were the raids limited to New York, or did they extend to Washington? And why did the FBI sit on their hoard for so long, if it had been a carefully targeted attack, and requested by some other government department? So many questions the answers to which will probably never be obtained.

The most notable, and heavily publicized, crib was the 1946 report drafted by Brigadier Francis Curtis, the Director of Post-Hostilities Plans, which set out British defence policy in the Mediterranean, and, in Nigel West’s words ‘effectively presaged the creation of NATO’. What happened at the Canberra end when this report was stolen and photographed by Ian Milner can be found in David Horner’s Spycatchers: the Official History of ASIO, 1949-1963. So eager was the local KGB station to get the news to Moscow that, instead of placing their copy in the diplomatic bag, they encrypted it word for word, and, when the original document was tracked down, the cryptanalysts at GCHQ and Arlington Hall had a field day.

• What were the Soviet reactions? Why were they so sluggish?

As I suggested earlier, I see ten distinct occasions when a Soviet response might have been expected, namely:

1. The 1941 disclosure that a copy of the POBEDA codebook had been obtained;
2. The intelligence gleaned from Currie in 1944 that the Americans were ‘solving the Russian codes’;
3. Philby in 1944;
4. The break-in(s) at the consulate(s) in New York (and Washington) in 1944;
5. Weisband’s access at Arlington Park between 1945 and 1948;
6. The February 1945 ‘return’ of the STELLA POLARIS material;
7. Gouzenko’s defection in September 1945;
8. The surrender to the FBI by Elizabeth Bentley in November 1945;
9. The indication in 1948 that the Canberra-Moscow channel was exposed;
10.  Philby in 1949.

The Pobeda Codebook:

(Photograph from the National Cryptologic Museum at NSA)

When the Soviets learned, in November 1941, that the POBEDA codebook had been obtained by the Germans, they were not greatly alarmed. They believed that the strong protection of OTPs would mean any discoveries of code tables would be irrelevant. Nevertheless, they decided to introduce a new one (075-B; JADE, to the Americans). It took them about two years to complete the exercise, namely to devise new codes, print them, and get them distributed to all worldwide stations. (Pryser criticized them for dilatoriness, as if the replacement were a delayed reaction.) And yet this exercise introduced a weaker, one-part code, that proved to be more easily ‘broken’ than the previous one. (It would presumably have taken them even longer to produce a new two-part codebook.) Moreover, Sudoplatov later expressed his scepticism that Gardner would have been able to learn much from the POBEDA codes when trying to build the JADE book. Of course, the Soviets did not, at this stage, believe that the Americans had started to engage in a serious effort to decipher their diplomatic telegrams, and given that the Petsamo material had been stolen by the Germans, would not have imagined that it could have fallen into the hands of the Americans.

Lauchlin Currie:

According to Benson and Warner, Currie informed his contacts in the spring of 1944 that the Americans were on the brink of ‘breaking the code’, something that Elizabeth Bentley got to hear, and about which she later told her FBI interrogators. Currie, who did not really understand these matters, may well have overdramatized the situation. Klehr and Haynes write that ‘he may well have heard an overly optimistic report sent to the White House about the early Venona effort’ (p 47). In any event, on May 1, 1944, a circular was issued to all stations to use a new message-starting point when they encrypted cables. This was a straightforward change in procedure to effect, but it backfired, since Cecil Phillips soon noticed that the first five-digit cipher group was the indicator to which page of the OTP should be used. (The cable that described the change can be seen as Number 26 in the Benson-Warner work, on page 259.)

Kim Philby (1):

According to Genrikh Borovik in The Philby Files, pages 235-236, Philby’s London controller reported to Moscow on August 29, 1944, that Philby (STANLEY) had told him that GCHQ had people working on Soviet cyphers:

‘S’ also reports that according to Cowgill there are fifteen people working at the ‘Resort’ [i.e. Bletchley Park] on our ciphers. Menzies proposes to add many more people at the ‘Resort’ to work on deciphering. ‘S’ tells me that this should be avoided, since the ‘Resort’ has a lot of experience in deciphering.

Moscow Centre did not appear to be unduly concerned about this news. It sounds as if Phiby was indeed referring to the VENONA project: he was intimately familiar with the ISCOT operation, but that was being undertaken in London, tackling communications between Moscow and communist groups in satellite countries. Philby had presumably already informed his bosses of ISCOT, since it had begun in June of 1943. In any event, that news was overshadowed by the other news that Philby revealed on that occasion – that he was probably going to be appointed chief of the new Soviet counter-intelligence section in MI6. No particular response has been recorded.

Consulate Break-Ins:

These events (apparently in 1944) are most mysterious, and very little has been written about them, as Nigel West intimated. Thus we do not know: i) at whose request the break-ins occurred, ii) how many of them there were, with New York and Washington being mentioned; iii) when they occurred; iv) how the FBI knew what to find, where, and whether they managed to hide their traces; v) why the FI apparently did nothing afterwards; and vi) whether the Soviets discovered the break-ins, and if so, why they did not make a vigorous protest. All we have is Lamphere’s not very convincing account, which I critiqued in detail earlier. If, indeed, plaintexts had been discovered alongside their encrypted versions, and the Soviets had suspected that they had been photographed, it should have constituted a very sharp warning. Yet they apparently did not respond.

William Weisband:

Weisband has been blamed for the disclosure of the VENONA project to his Moscow masters. Yet the records show that he was out of contact for much of the critical period, and that what he passed on mainly concerned internal Soviet military, police and other traffic, which was transmitted in plaintext. After a long gap in communications, he made contact with Feklisov on February 9, 1945, when he had fresh duties at Arlington Hall, and gained some high-level knowledge of the VENONA program. Yet he did not gain a permanent job there until August 26, 1946, when he was put to work on internal Soviet communications, and it was only in June 1947 that his team was transferred to the diplomatic messages. Again, he had been out of touch with his Soviet minders, and only in January 1948 did he make contact with Bruslov. Batvinis reports that, the following month, Weisband informed Bruslov on progress on VENONA, and plaintext systems such as TABER, and SHAMROCK. These disclosures may have alerted Moscow to the exposures still rife in Australia, but, even then, the Soviets did not react with authority or urgency, and it was in any case too late to protect anything by then. Weisband was interrogated by the FBI in 1950, but he denied everything.

The STELLA POLARIS Material:

One has to suspend belief over some of the claims made about the STELLA POLARIS material. First, that the trove handed over to the OSS and that discovered by the TICOM expedition were essentially identical. Second, that Donovan was forced to hand it all over to Andrey Gromyko, under pressure from Eleanor Roosevelt and Secretary of State Stettinius, since they claimed that it belonged to the Soviets. Third, that the NSA declined to acknowledge that it had received anything from the OSS, and failed to acknowledge that GCHQ had copied a portion, and had sent it to them. Fourth, that the NSA somehow managed to overlook the significance of the TICOM resource until 1952 or 1953, when they realized that it contained a useful codebook.

So what did the two collections contain? Batvinis claimed that TICOM shipped seventy-three file cabinets of material, containing 300,000 documents, which may explain why NSA took so long to find that charred codebook. (Although it does not explain why a complete inventory was not taken immediately.) On the other hand, Warner and Benson described the 1,500 pages that were ‘returned’ to Gromyko in February 1945, and Costello and Tsarev, in Deadly Illusions, and Andrew and Gordievsky, in KGB: The Inside Story, echo that number. Yet, as former GCHQ historian Tony Comer has sagely pointed out to me, no codebook would have been that large and unmanageable: the documents must have included ancillary information, such as reciphering tables, or other analysis.

The TICOM haul was obviously much bigger. Budiansky wrote of ‘eight tons’ of material (including technology) dug up at Rosenheim, and of the fact that the TICOM 3 expedition brought back 300,000 pages in one hundred-and-seventy steel file cases. The TICOM haul was much more expansive than that of STELLA POLARIS. Alvarez and Mark inspected the NSA archives, and learned that the records contained a ‘long list of Soviet codes, ciphers, and other cryptographic materials under the heading ‘STELLA POLARIS/Source 267’ [namely GCHQ]. Yet the 1944-45 fiscal report for the ‘Special Problems [Russian] Section’ did not mention any Finnish material. “If the code breakers had received product from the STELLA POLARIS group, the annual report probably would have said so”, they write, trustingly. But maybe not: if some facts had to be buried, then well-interred they were. Again, a weaselly use of words could also conceal much: ‘OSS’, when the information was received from GCHQ. ‘Finnish sources’, when they could better have been described as Swedish, or even German. Moreover, if the ASA had been able to gain so much useful intelligence from the TICOM interrogations of the German personnel in 1945 (see Johnson, above), why were they so slow in exploiting the documentation that came along with the exploit?

Thus it is likely that only a small subset of the trove was handed over to Gromyko. It would have been absurd for Donovan to have presented a gift of intelligence that the Soviets had never seen before – even though their spies may have got wind of it. The repetitive term of ‘returned’ suggests that only items that were clearly viewed as having been in the original possession of the NKVD were presented, and Donovan probably minimized the scale and scope of the STELLA POLARIS material when dealing with the White House. Bradley Smith has offered the most detailed description of the process: he writes generally about ‘codes and documentary materials’ that were assembled and turned over, but also mentions that Colonel Buxton, the former OSS chief in Stockholm, had unnecessarily put his oar in, and informed the State Department that 15,000 sheets of material were involved. But if these ‘documentary materials’ were additive to the codes, whence did they spring? Were they German or Finnish commentary or analysis, which the Soviets were absolutely not entitled to see?

Smith declared that the most surprising aspect of this transfer (overlooking the pressure from Red Eleanor) had been the decision by the U.S. government to send the codes back, ‘for this virtually guaranteed that the Russians would change their ciphers’. Yet this was a matter of codebooks, not ciphers, and the Soviet has long replaced the POBEDA codebook of which the details would have been in the package. So his conclusion that ‘the really irresponsible gamble had been taken by Dovonan when he purchased the codes in the first place’ is off the mark. Gromyko may not have been overwhelmed or surprised by his gift, and it did not provoke any changes in Soviet cipher procedures.

A dissenting voice comes from Andrew and Gordievsky. They write that Donovan, after Buxton’s intervention, had to send a message to Colonel Fitin, head of the Foreign Directorate (INU) in Moscow, in which he disingenuously stated that ‘he had made no study of the material’. At Fitin’s request, the charred code book (and nothing else?) was passed over to Gromyko in Washington. Fitin was of course not deceived, although he was impressed by the naïveté of Roosevelt and Stettinius. The authors then state that the NKVD subsequently changed their codes in May 1945, and that ‘a copy of the charred 1944 code book was used from 1948 to help decrypt some NKVD/NKGB communications during the last year of the war’. They are doubly mistaken. The codebook was from 1941, not 1944, and there is no evidence that the Soviets changed their codes again, having done so at the end of 1943. Why the Arlington Hall team had to wait until 1948 before exploiting the codebook is not explained.

Gouzenko’s Defection:

Moscow was severely alarmed by the defection of Gouzenko in September 1945, but more because of the threats to the safety and concealment of its spy networks than by any cryptological revelations that Gouzenko might have passed on. Gouzenko belonged to the GRU, not the KGB, of course, and he brought no relevant material with him, just the knowledge of procedures, some messages that were of little use, and information on how codebooks were constructed. Budiansky states that the GRU used a completely different codebook from that the KGB deployed. (Some writers, such as John Bryden in Best-Kept Secret, and Phillip Knightley in The Master Spy, have exaggerated Gouzenko’s contribution.) Thus, instead of reacting with operational changes, the KGB just put all agents on alert, and ordered contacts to be reduced in frequency. Ironically, Weisband was affected by this edict as well, so it meant that any disclosures he had about activities at Arlington Hall were held over.

Elizabeth Bentley:

Bentley confessed to the FBI two months later, and revealed much about the extent of Soviet espionage in the United States. Her statements just reinforced the reduction of meetings.

The Exposure of the Canberra-Moscow Channel:

Remarkably, the channel between Canberra and Moscow was still using the faulty duplicate OTPs in 1948. Unfortunately, as I described before, the records to which I have access show only one-way traffic, so I cannot read how Makarov in Canberra responded to instructions and queries from Moscow. Yet those messages do provide insights into the way that the Soviets eventually worked out that their cables – especially from 1946 – had been exposed.

On March 19, 1946, Makarov (EFIM) reported to Fitin (VIKTOR) in Moscow that Clayton (KLOD or CLAUDE) had obtained the documents ‘Security and the Indian Ocean’ and ‘Security in the Western Mediterranean and the Eastern Atlantic’, prepared by the English Post-Hostilities Planning Staff, dated May 19, 1945. Makarov was able to have them photocopied in thirty-five minutes so that Clayton could return them to the Department of External Affairs. Makarov concluded his message with the following: “  . . .it would be necessary to transmit immediately to MOSCOW but this will require copious telegraphic correspondence. We urgently request your instructions.”

How Moscow responded is not available, but Makarov evidently encrypted the complete documents verbatim, which provide the VENONA teams with an excellent crib, and gave a large boost to their efforts to decrypt more traffic, with the result that they were processing 1948 messages ‘in real time’. Thus, when rumours of leakages surfaced in April 1948 that confidential papers had been passed on to the Soviets (with the cover story that a defector had informed the British) John Burton, Evatt’s chief adviser in the Ministry, who knew nothing about VENONA, began to ask questions about the source of the information, in order that he could try to track down the leakages. Shedden, who had little trust in Burton (according to Ball and Horner) essentially ignored him, which was a strange reaction if Burton had been charged with finding the source of the leak. Burton then provided cover for the obvious culprit, Ian Milner.

Thus news soon reached Moscow that a very delayed reaction to the theft had taken place. It is possible that the Soviets then realized that their ciphers had been broken. Yet relevant inquiries from Moscow had already been sent. A message of March 26 asks Makarov to ‘state how and from whom KLOD received the information reported by you in [unrecovered]’. They had earlier suggested that KLOD was acting incautiously, and had asked Makarov to stop receiving documents from him, all of which suggests they had already received a warning about signals security. For several weeks, however, the cables show no alarm, and Moscow discussed routine matters such as bringing in new recruits. Only on June 2 does a note of urgency arrive: “Please confirm urgently the receipt of code 0177-B which was sent with post number  . . . on 8 January 1948.” That message was sent by Colonel Aleksey Shchekoldin, head of the KGB cipher section. No cables were decrypted after June 5.

What is puzzling is why Canberra did not adopt the new pads (for that is how I interpret ‘code 0177-B’) in January. It was obviously not an order, since it had not been carried out. It was a preparatory measure, as the Cipher Section, maintaining a parallel set, knew that the pads would be running out soon. Makarov was not disciplined for this oversight, as he stayed in office right until the Petrov years. It all proves to me that Moscow had no idea about the duplicate OTPs up until this time, as they would have otherwise mandated an immediate world-wide change. Every other station – or, certainly, Washington, New York, and San Francisco – had used up their supply in 1945, and Moscow had had no inkling that there had been an exposure in its traffic with those cities. Bringing in new pads immediately, as a safety precaution, was obviously a good idea, but it does not prove to me that they recognized what the source of the problem had been at this juncture. How long it took them to work out what had happened may never be known, and those responsible would have done all they could to cover it up, knowing what their fate would be if detected.

Lastly, the closedown of the outlying KGB channel had nothing to do with ‘Black Friday’. The covers had been put on diplomatic and intelligence traffic in June 1948. ‘Black Friday’ was provoked by what Moscow learned from Weisband about their highly insecure internal communications systems.

Kim Philby (2)

When Philby was posted to Washington in October 1949, he quickly picked up from Arlington Park the progress being made on VENONA, and the recognition of cryptonyms concealing the identities of spies. On a return to London in March 1950 (ironically a recall to discuss the Fuchs affair), Philby passed to Modin, the handler of Burgess and Blunt, the now infamous message (reported in West’s and Tsarev’s Crown Jewels, p 182):

 . . . . the Americans and the British had constructed a deciphering machine which in one day does ‘the work of a thousand people in a thousand years’. Work on deciphering is facilitated by three factors; (1) A one-tie pad was used twice; (2) Our cipher resembles the cipher of our trade organization in the US: (3) A half-burnt codebook has been found in Finland and passed to the British and used to decrypt our communications. They will succeed within six to twelve months.

Philby then referred to the CHARLES (Fuchs) case, and he went on to describe the threats to himself. Korovin, the rezident, admitted to Modin that ‘mistakes had been made in Soviet cipher procedures’, which may suggest that the Soviets had recognized the OTP problem before Philby brought it up.

Hayes and Klehr picked up the story in their book, and added a Footnote (51, p 401) that reflected the views of Arlington Hall, but failed to clear up the confusion:

Venona project cryptanalysts comment that Philby’s explanation, although adequate for warning Moscow that its cable traffic had been broken into, reflected a layman’s muddled understanding of cryptanalysis in that it exaggerated the assistance provided by early computers to sorting messages for duplications, overplayed how the burnt code book (which was in American, not British, hands) had been used up to that time (only as a model of Soviet code-making), and understood the way Trade messages assisted solutions.

The comments about exaggeration were indeed, but Philby had clearly picked up from Gardner the intelligence about the charred Tiltman codebook, something that these non-laypersons definitely wanted to conceal.

• Why has no further progress on VENONA messages been made since 1980? Why does there continue to be such secrecy over the programme?

Last June, I wrote the following on coldspur, lamenting the lack of progress:

I see a number of opportunities. First of all, a renewed attack on partially deciphered messages, using much faster computers, and probably advanced AI techniques, could surely reveal much more about the traffic and persons involved than was decrypted decades ago. Second, an integrative approach to the interpretation of information would be highly desirable since records released during the past twenty years for the Foreign Office, MI5, and GCHQ, as well as resources like the Mitrokhin Archive, would probably point to conflicting missions, and oversights in analytical opportunities. Third, much of the material that has been published has been redacted because of old sensitivities to living persons, and also contains errors or partial information that could be easily corrected based on intelligence that is now available. With the passage of time, and the deaths of such persons, such names should be restored. One of the most frustrating aspects of VENONA decrypts is that it has been impossible to determine what breakthroughs were made, when, which has complicated the task of historical interpretation.

Given the previously mentioned appeals for continuing research into identifying the persons still hidden behind the cryptonyms, it is rather shocking that, in 2025, the state of the archives is in worse shape than it was twenty years ago. The decisions made by the NSA in obscuring a large part of the trove, and making the records unsearchable, are particularly egregious. In addition, no general housekeeping appears to have been made on the transcripts that are available, such as correcting obvious mistakes, sorting and identifying them correctly, or making emendations based on intelligence that has come to light since. A reconciliation of versions from different sources is sorely needed. An integrated view of the US, London and much neglected Swedish traffic should be made.

Valiant efforts have been made by John Earl Haynes, but he is now in the process of moving his archive at the Wilson Center at Stanford University to the Davis Center at Harvard University and to the National Security Archives at George Washington University. What that transfer means technologically, I do not know: nor do I know whether the exercise will attempt to incorporate other files, or to engage in a clean-up of some sort. All this means, however, is that, even if there were a will to start a project of further investigation, it would by no means be a simple task.

Following up on my comments above about computer analysis, it puzzles me why highly parallel processing, perhaps accompanied by AI techniques, has not been deployed to attack some of the obvious gaps in the transcripts. For instance, what about that one-way traffic from Moscow to Canberra from August 1946 onwards? The fact that no messages issued from Canberra to Moscow appear in any form can surely not be attributable to the fact that, for two years, Moscow always used duplicated pads, while Canberra used pristine ones? Given the nature of the posited exchanges, with, for example, Makarov being asked questions that required an answer, one might imagine that necessary phrases from the codebook would be used in the following cable? Also, when Makarov asked for guidance over the transmission of the British papers in 1946, why is there no response in the record? One can only conclude that GCHQ and the NSA have lost interest.

And that conclusion leads to the second part of my query: whether there has been a serious decision made to hush things up, lest further embarrassing secrets be unveiled, such as the names of hitherto unidentified spies who had infiltrated government establishments. Again, from last summer, I reproduce what I wrote:

Yet I could never understand why such an attempt at secrecy was necessary. William Weisband, a linguist at Arlington Hall, had informed his Soviet masters of the project, and by 1948 the Soviets were able to undertake a total overhaul of their encryption procedures. Kim Philby also informed them of the progress made on the exercise. Yet the Foreign Office (who admitted to being controlled by MI5’s demands) stubbornly insisted that there was a security risk. As late as September 28, 1953, Talbot de Malahide (yes, he!), responding to a request by Patrick Dean as to why the Office was against releasing all our knowledge of the Maclean/Burgess affair, wrote:

The argument roughly is that it is most important to conceal from the Russians our knowledge of Bride [i.e. VENONA] material. They cannot, of course, now prevent us from extracting what we can from it. But if they knew we were doing this, they could take defensive action which would probably ruin any chance we still have of making use of the knowledge we obtain in this way. [FCO 158/126]

Dean annotated: ‘Thank you! I agree.’, thus endorsing the code of silence. Yet why Malahide and co. thought that the Soviets would not already be taking ‘defensive actions’, based on their knowledge of the exercise, rather than waiting for the British to declare to the world what they had discovered, defies explanation. Of course, those illusions would shortly be shattered by the Petrov revelations a few months later.

For some reason, American institutions also decided to try to keep the details about VENONA secret until writers like Chapman Pincher and Robert Lamphere started leaking details in the 1980s. It was not until 1995 that an admission was made, and a bi-partisan commission started releasing materials. From my study of the archives, I would conclude that the professed anxiety about admitting the VENONA programme to the public was attributable more to the embarrassment over the way that British institutions had been infiltrated, and to the decisions made about re-instituting Burgess and Maclean in prominent positions, than it was to the concern about divulging damaging secrets to the Soviets.

While there was a justifiable conviction that trying to use the transcripts themselves as evidence in any criminal trial, because of the use of cryptonyms and the lack of transparency in how the decryptions themselves had been made, it seems to me that a substantial propaganda coup could have been made by explaining the stunning achievements of the exercise. It was not that it would have alerted the Soviets: they had made the necessary adjustments as soon as they learned of the exposure. It was not like the secrecy over the ENIGMA project, and the corresponding British Type-X equipment, which had been supplied to other countries after the war, and thus might have provoked embarrassing questions. This was a once-off example of a lapse in procedure, and a spectacular effort to exploit it. Chrsitopher Andrew wrote: “The value of VENONA as a counter-espionage tool was diminished, sometimes seriously, by the extreme secrecy with which it was handled.” (Defend the Realm, p 380)

I regard it as absurd to maintain that there is a security risk in explaining openly how the VENONA project succeeded, how some of the breakthroughs were made, what the relationship between the various codebooks (diplomatic and trade; one-part and two-part – POBEDA and JADE) was, how Tiltman, STELLA POLARIS and TICOM individually contributed to the success, where exactly Gardner acquired his codebook with the inkblot or bullethole, etc. etc. The technology of 2026 is vastly different from that of 1946. The details about ULTRA and the Enigma machine have been discussed in great detail ever since Frederick Winterbotham broke the news in 1974. Why cannot the same openness be granted to VENONA, a phenomenon of equal significance? And why should attempts not be made to take a fresh stab at identifying the persons behind all those unresolved Russian cryptonyms? Nigel West wrote in the entry on VENONA in his Historical Dictionary of Signals Intelligence (2012): “While  the American policy appears to have given a measure of protection to the living, being those suspected Soviet sources who were never positively identified or confronted with the allegations, their British partners seem to have adopted political embarrassment as their principal criterion for eliminating sensitive names.” Yet too much time has passed. There are no longer any surviving Soviet agents from that time, and politicians from two or three generations back should not be protected from embarrassment.

Conclusions

1) Overall, the CIA/NSA have behaved dishonorably and deceptively in their quasi-official histories of the project. With the apparent goal of exaggerating the skills of its domestic cryptanalysts, and minimizing the contributions made by GCHQ, cribs from foreign channels, the acquisition of codebooks from various sources, or even assistance from computer techniques, Benson, Warner, Phillips and others have distorted what happened, and have even tried to reformulate memoir to fit the story. Even more remarkably, no one has risen to the task of challenging their accounts.

2) The treatment of codebooks, and the explanation of their exploitation, are particularly egregious. Tiltman’s contribution, and the way in which Gardner unsubtly revealed to Lamphere how the codebook with the bullethole had helped him, has been blankly denied. At the same time, efforts have been made to blur the distinction between the Petsamo codebook and the battlefield version. The prevarications over the STELLA POLARIS incidents and the TICOM retrievals are simply ingenuous.

3) The role of cribs has been vastly understated. There were probably similarities between the Trade codebooks and procedures and the Diplomatic equivalents that have not been described properly. It is difficult to understand how Gardner could have made such quick progress in 1946 without such help. The incidents of the consular break-ins, and their results, have been clumsily finessed. The role of the documents transmitted verbatim from Canberra in 1946 has been largely ignored.

4) The disdain for the OSS, and Donovan in particular, shown by the ASA was probably warranted. Donovan was a pushy individual, and his organization was riddled with communists. Yet, for the histories to diminish entirely the contribution of the OSS in gaining material from the Finns/Swedes/Germans was foolish, especially since the records show that GCHQ off its own bat sent the same valuable material to Arlington Park soon after it was received.

5) The stumbles of the KGB (which was increasingly controlling the GRU at this time, as well as being the department responsible for the production of OTPs) are remarkable. Even though it believed in the absolute security of its OTP system, in 1943 it made theoretically unnecessary changes to its codebooks that actually weakened its systems, and it changed the indicator process to make its traffic more exposed. It showed an irresponsible carelessness in not enforcing the use of new OTPs around the globe, thus allowing the dramatic disclosures of the Canberra-Moscow exchanges, which were a damaging phenomenon in many ways.

6) The contributions to the demise of VENONA made by Currie, Weisband and Philby have been exaggerated. The KGB did not respond appropriately to early warnings, and Weisband was out of contact at the critical time. Weisband’s knowledge had a far greater effect on the Soviet Union’s domestic systems, and ‘Black Friday’ was in response to his alerts concerning those communications, not to the success of the decryptions of VENONA, which was not able to work on any traffic after June 1948. Philby’s revelations were too late to protect many Soviet spies from identification, although his knowledge did contribute to plans for the exfiltration of Burgess and Maclean.

7) The obstructiveness of such as Eleanor Roosevelt and Minister of State Stettinius is regrettable, but nor surprising. There were many in prominent positions who, even if not actual fellow-travellers or agents of influence, still regarded the Soviet Union as an ally, and foolishly believed that by appeasing Stalin they would encourage him to behave more generously to the USA – and to the countries of Eastern Europe. The process of ‘returning’ material to Gromyko in 1945 has an element of farce about it.

8) The woeful state of the VENONA materials, and the lack of any initiative to clean them up, or apply fresh techniques to resolving many unanswered questions, is extremely regrettable. Unfortunately there appears to be no pressure being applied by even those who have had an investment in studying VENONA. The project is now eighty years old: the USA stopped intensive work on it in the 1950s, and it is now almost ancient history. It needs another eager academic with a youthful cadre of undergraduates to resuscitate the project.

9) The state of totally undeciphered messages is unknown. Whether there is a pool of all worldwide traffic in its original format of five-number groups held centrally (probably by the NSA or GCHQ) is currently indeterminable. Tiltman went on record as saying that Soviet traffic gathered by UK interception in the early years of the war was later destroyed. Likewise, it is impossible to tell whether the medium on which they were stored – and inspected as late as the 1980s – is still viable and readable forty years later. A statement from both organizations would be appropriate.

(I thank coldspur correspondent Ian Wraith for his comments on an earlier draft of this article. Any mistakes I have made are of course mine, not his.)

(Recent Commonplace entries can be seen here.)