Analysis

Which cipher is it?

Somebody hands you FMXVEDKAPHFERBNDKRXRSREFMORUDSDKDVSHVUFEDKAPRKDLYEVLRHHRH. Vigenère? An agent will say Vigenère, it always does. Ask first:

ciphers guess "FMXVEDKAPHFERBNDKRXRSREFMORUDSDKDVSHVUFEDKAPRKDLYEVLRHHRH"

Family guess (57 characters, 57 A-Z letters, lang=en, IoC 0.0627 (en plaintext ~0.065, uniform random ~0.038)), most likely first:

1. substitution-multiplicative, substitution-shift, substitution-reflection (medium): affine, caesar, atbash
   IoC 0.0627 is close to en plaintext, one alphabet throughout, and affine a=3 b=5 makes the letters fit en.
   Next: ciphers_decode with cipher affine and a=3, b=5.

Not a Vigenère at all. It's Example 1.10 from Stinson, and a=3, b=5 is his key. The test suite checks that too.

The layout goes first. Dots and dashes are Morse. Digits 1 to 5 in pairs are Polybius, single digits tap code. Numbers up to 26 with something between them are A1Z26. Only A, D, F, G, V and X is ADFGVX. Two letters in groups of five is Bacon. Hex goes by length: whole 16-byte blocks point at AES and its friends, 8-byte ones at DES and its friends, anything else at a stream. And U2FsdGVkX1 is the Salted__ header CryptoJS writes. Those are rules, not statistics, so they come back sure of themselves.

Letters get statistics. Do they fit the language as they stand? Then they were moved, not replaced, so rail fence, columnar or a route. IoC near plaintext? One alphabet. The guess then tries all 311 affine keys, Caesar and Atbash included, and names the one that reads. Near 0.038? Several alphabets, and period is next. An even count with no J and no doubled pair smells like Playfair. Add a doubled pair and it's four-square, which bifid can pass for. English pairs peeking through in plain sight? That's a two-square getting lazy, and it moves ahead of four-square. Flat letters with a J, no spaces and whole blocks of 2 or 3? Could be Hill. Numbers from 22 to 110, none of them ending in 1? Two square cells added up, so Nihilist.

How sure is it? I encrypted 40 English and 40 Polish samples with fourteen letter ciphers, Caesar to Enigma. From 200 letters the right family came first in at least 37 of 40, for each of them. At 30 letters a Caesar lands on top about half the time. So a short text comes back low and keeps the other side on the list. Bifid is the stubborn one. A keyed square can leave the letter counts looking English, and counting single letters can't see past that.

import { guessFamily } from "@agntn/ciphers";

const guess = guessFamily(ciphertext, "en");
guess.candidates[0].ciphers; // ["affine", "caesar", "atbash"]
guess.candidates[0].options; // { a: 3, b: 5 }

Candidates, not a verdict. The decode that reads is the verdict. --lang pl or ja changes the language the letters should fit.

Brute force

A Caesar has 25 keys. The attack is all of them, then read:

ciphers brute "KHOOR ZRUOG"

shift= 3 -> HELLO WORLD
shift= 6 -> EBIIL TLOIA
shift=14 -> WTAAD LDGAS
...

There is no dedicated function for this in the library, and there shouldn't be. It's create("caesar").decode(text, { shift }) in a loop from 1 to 25 - that's what the command, the ciphers_caesar_brute tool and the landing panel run. Shift 0 isn't on the list. It would be the input, and the library refuses it as a shift anyway.

The command and ciphers_caesar_brute sort the same 25, the most English line first. With --lang pl on the command, or lang: "pl" on the tool, the most Polish one leads. ja does the same for Japanese romaji. A model then reads the answer on line one instead of wading to shift 17.

So how do you tell HELLO WORLD from EBIIL TLOIA? Not by letters. Both are made of common ones, and counted one by one the junk wins. English goes by letter pairs inside words instead, that's pairFit. HE is one English pair in 63, II one in 5800. On 10 characters of Pride and Prejudice or Frankenstein, pairs put the right shift first 92% of the time, single letters 72%. Polish and Japanese have no pair table yet, so they still rank by fit. A lone short word can still lose, and that's why all 25 come back.

The tool cuts them, though. Only its top line comes back whole, the rest stop at 80 characters with a …. Past that length and with the right lang, the ranking put the right shift first in all 6000 English and Polish samples I threw at it, and in 2000 Japanese ones. And 24 full copies of a 2000 character ciphertext are about 23 000 tokens of wrong answers. Eighty characters is plenty to see that a lower line reads. For the whole of it, decode with that shift. The command prints every line whole, a terminal has no context window to save.

Frequency

import { analyzeFrequency } from "@agntn/ciphers";

const analysis = analyzeFrequency("DWWDFN DW GDZQ", "en");
analysis?.counts; // [["D", 4], ["W", 3], ["F", 1], ...] sorted by count
analysis?.reference; // "ETAOINSHRDLCUMWFGYPBVKJXQZ"
analysis?.ic; // index of coincidence
analysis?.referenceIc; // what plaintext in the language scores, 0.065 for English
analysis?.fit; // how much the letters look like English, higher is closer
analysis?.pairFit; // the same for letter pairs inside words, English only

analyzeFrequency(text, language) uppercases, drops everything outside A to Z and counts. No letters at all - undefined, not an empty table, so check before you read. The language picks the reference order, en, pl or ja. The English order is the Wikipedia letter frequency table. The Polish one is PWN's count of the IPI PAN corpus, base letters only, because the diacritics are gone before anything gets counted: ż counts for nothing, not for z.

Japanese is ja, romaji in Hepburn: shi, chi, tsu, fu, ji. I didn't find a published letter table for it, only keystroke counts from typing in the Kunrei style, where C and J never show up. So the table is built from the hiragana counts of Chikamatsu et al. (2000), a year of newspaper text. Each kana is spelled the way Japan's passport Hepburn table does it, きょ as kyo, and っ doubles the next consonant. L, Q, V and X get almost nothing instead of zero, so a loanword costs some fit but doesn't wipe it out. It only knows kana, not kanji readings. That's why it expects less u, s and h than real text has.

Reading the index of coincidence

The IC is the chance that two letters picked from the text are the same. English sits near 0.065, Polish near 0.057 because its letters spread more evenly, uniform random near 0.038. Japanese romaji goes the other way. Five vowels carry about half the text, so it sits high: Kokoro and Rashomon measure 0.082, the kana table says 0.088. That one number tells you which family you are looking at before you try a single key:

• Monoalphabetic substitution (Caesar, Atbash, Affine) keeps the histogram shape and only relabels the bars. IC stays high, and the tallest ciphertext letter is probably E.
• Polyalphabetic (Vigenère, Alberti) spreads one plaintext letter over several ciphertext letters. IC drops toward 0.038 and the histogram goes flat. Time to look for a period.
• Transposition (rail fence, columnar, route) moves letters without changing them. The histogram is the plaintext's, only the words are gone.

The Playground draws the histogram for any text, expected order next to actual.

The CLI

ciphers frequency "DWWDFN DW GDZQ"
ciphers frequency "ZAŻÓŁĆ GĘŚLĄ JAŹŃ" --lang pl
ciphers frequency "KIMIGAYO WA CHIYO NI" --lang ja

One bar per letter, expected and actual order, then the IC with the two reference values next to it so you don't have to remember them.

Key length

The IC dropped to 0.038. Now what? A Vigenère key repeats, and that's the whole weakness. Take every fifth letter under a five-letter key and you're holding one Caesar. Get the length right and every column reads like plaintext again.

ciphers period "CHREEVOAHMAERATBIAXXWTNXBEEOPHBSBQMQEQERBWRVXUOAKXAOSXXWEAHBWGJMMQMNKGRFVGXWTRZXWIAKLXFPSKAUTEMNDCMGTSXMXBTUIADNGMGPSRELXNJELXVRVPRTULHDNQWTWDTYGBPHXTFALJHASVBFXNGLLCHRZBWELEKMSJIKNBHWRJGNMGJSGLXFEYPHAGNRBIEQJTAMRVLCRREMNDGLXRRIMGNSNRWCHRQHAEYEVTAQEBBIPEEWEVKAKOEWADREMXMTBHHCHRTKDNVRZCHRCLQOHPWQAIIWXNRMGWOIIFKEE"

  length  5  IoC 0.0666  key JANET
  length 10  IoC 0.0663  key JANETJANET
  length 15  IoC 0.0693  key JANETJANTTJANET
  length 20  IoC 0.0604
  length  3  IoC 0.0466
  ...
Kasiski: 12 distances between repeated trigrams; factors dividing the most: 5 (10), 2 (4), 10 (4), 7 (3), 3 (2).

That's Example 1.11 from Stinson's Cryptography: Theory and Practice. The test suite checks it. Stinson gets the same column IoC by hand. Five, then decode:

ciphers decode vigenere "CHREEVOAHM..." --key JANET   # THEALMONDTREEWASINTENTATIVEBLOSSOM...

How does it know? For each length from 2 to 20, the letters get dealt into columns. Each column gets its IoC, the length gets their mean. Wrong length, every column mixes alphabets and sits near 0.038. Right length, it jumps to plaintext level. Here 0.047 at 3 against 0.067 at 5. Hard to miss.

Multiples are the trap. Ten columns under a five-letter key are still one alphabet each. They have fewer letters, so their IoC wobbles, often upward. Sort by IoC alone and 20 beats 5 on a short text. So a multiple goes right behind a divisor that reads almost as well. 5, then 10, 15, 20. I ran 300 random keys over README prose. At 200 letters the right length came first 90% of the time. At 400, every time. Plain IoC sorting managed 65%. Under 100 letters it's a coin toss either way.

Kasiski is the second opinion. It finds every trigram that repeats and measures the distance back to its last copy. Then it counts how many distances each factor divides. Why does that work? THE under the same key letters gives the same ciphertext. Those copies sit a multiple of the key length apart. Chance repeats add noise. And 2 always looks good, half of all numbers are even.

The top three lengths get a key. Each column takes the Caesar shift that reads most like the language. Same fit the brute force uses. --lang pl or ja changes the language, --max-period the longest length.

import { estimatePeriod } from "@agntn/ciphers";

const analysis = estimatePeriod(ciphertext, "en", 20);
analysis.periods[0]; // { period: 5, ic: 0.0666, key: "JANET" }
analysis.factors[0]; // { factor: 5, distances: 10 }

The key is Vigenère math. Beaufort has the same length but shifts the other way, and recover below handles that. Autokey has no length at all. Its key runs on into the plaintext and never repeats. Nothing to find, sorry.

Recovering the key

period guesses each key letter from its own column alone. On a short text a letter or two comes out wrong. recover goes the rest of the way. Name the cipher and get keys back, best first:

ciphers recover "CHREEVOAHM..." --cipher vigenere --limit 2

Vigenère keys for 313 letters (lang=en), best fit first. Fit is the mean log probability per quadgram: English plaintext scores about -9.44, random letters -13.95.

1. key JANET, fit -9.82: "THEALMONDTREEWASINTENTATIVEBLOSSOMTHEDAYSWERELONGER..."
2. key MTE, fit -13.41: "QONSLRCHDAHAFHPPPWLESHUTPLACWDPZXETMSXAFISFCTIVWYEWCZTLDAOOXKNFATMAUGUYBJNTKANNE"…

Stinson again, JANET again, no decode needed. Fit is the mean log probability of a quadgram, four letters in a row. English sits near -9.4 and random letters near -14. JANET reads. MTE is noise. That gap is the whole verdict.

How does it get there? It tries the five likeliest lengths. Each column first takes the shift whose letter counts fit best by chi-squared. Then quadgrams take over: change one key letter, then two neighbours at once, keep whatever reads better. Why pairs? A quadgram spans neighbouring columns, so two wrong letters can hold each other in place. --cipher beaufort and --cipher variant-beaufort run the same search with the shift going the other way. Know the length already? --period skips the guessing.

A substitution has no length, just 26! alphabets. So it climbs. Swap two letters, keep the swap if the text reads better, stop when no swap helps. Then start over from a random alphabet, up to a hundred times. Short texts get the most restarts, they're cheap.

ciphers recover "Nih. Sfwhlm yilftsg rm gsv gizb zg vrtsg. Slonvh zgv mlgsrmt, ivzw gsv mlgv gdrxv zmw gsivd rg rmgl gsv uriv drgslfg z dliw." --cipher substitution --limit 2

1. key ZYXWVUTSR??NBMLO?IHGF?D???, fit -9.21: "Lrs. Hudson brought in the tram at eight. Hoples ate nothing, read the note twice and threw it into the fire without a word."
2. key ZYXWVUTSR??OBML??IHGFND???, fit -9.23: "Vrs. Hudson brought in the tram at eight. Holves ate nothing, read the note twic"…

That's Atbash on 97 letters. Close, not perfect. M, L and Y show up once or twice each, and "tram" reads as well as "tray". Your eyes fix the rest. More text fixes it for you. On ten 150-letter passages from Sherlock Holmes, every one came back with at least 95% of its letters right. The key gives the cipher letter for each plaintext letter A to Z, and ? marks a letter the text never uses. Holes and all, it goes straight into ciphers decode substitution --key.

Columnar is plain brute force. Every column order of every width from 2 to 8, scored on the start of the text, about a tenth of a second. --key-length 9 adds 362,880 more orders and takes under a second.

ciphers recover "ru gn  i eeh  nt t nhrt rMHnuieye.mtt,d  e witiito.srttrtho nnehtinrto  o .sdbh tagHs irtowahiteehad oo ha tlaogaeecde  fwuw" --cipher columnar --limit 1
# 1. key BADCE, fit -9.30: "Mrs. Hudson brought in the tray at eight. Holmes ate nothing, ..."

The text went in under BAKER. BADCE is BAKER in disguise: columnar only cares about the alphabetical order of the letters, and that order is the same. ciphers decode columnar "<text>" --key BADCE takes it as is.

The quadgrams come from ten Project Gutenberg novels, Dickens to Tolstoy, 7.86 million of them with spaces dropped. The 16,000 most common made the cut. The full Google Books table counts inside words only, and it solved none of twelve test texts once the spaces were gone. Puzzles drop spaces all the time.

Polish or Japanese romaji? There's no quadgram table for those yet, so --lang pl scores single letters. That's enough for Vigenère and both Beauforts, every column is its own Caesar. A longer key pays ln 26 per letter there, or twelve columns beat six by pure chance. Substitution and columnar need English.

import { recoverKey } from "@agntn/ciphers";

const recovery = recoverKey(ciphertext, { cipher: "vigenere" });
recovery.candidates[0]; // { key: "JANET", fit: -9.82…, text: "THEALMONDTREE..." }

ciphers_key_recover does the same for a model. Only the top text comes back whole, the rest stop at 80 characters.

Which passphrase settings?

A blob that starts with U2FsdGVkX1 is a passphrase blob. You have the passphrase. You don't have the rest. MD5 or SHA-256? 128 bits or 256? One hash pass or ten thousand? The header won't tell you. So try them all:

ciphers probe "U2FsdGVkX18HYWQDuJcJTh2NoqzqwZ9pWaEBkXGGu54=" --key secret --iterations '[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]'

Passphrase probe (salt 07616403b897094e, 1 block, 144 tries: digests md5, sha1, sha256 × keyLength 128, 192, 256 × iterations 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16):
Padding held in 2 of 144 tries, about 0.56 expected by chance. Most printable first:

  digest sha256, keyLength 256, iterations 1: pad 2, 100% printable, "ATTACK AT DAWN"
  digest md5, keyLength 192, iterations 13: pad 1, 33% printable, not UTF-8, hex 4232d77f248ed9f9e107f42fc46ab8

A wrong setting passes the padding check about once in 255 tries, ...

Two hits. One of them is real. The blob comes from openssl enc -md sha256, the same one the test suite opens. The MD5 line is luck. A wrong key ends in a valid 01 pad once in 256 tries. Add the rarer 02 02 and longer, and it's one in 255. Run 144 tries and half a hit falls out of thin air. Run a few hundred and you get a couple. That's why the count sits next to the hits.

How do you tell them apart? The real one reads. Noise is bytes that aren't UTF-8, usually less than half of them printable, and its pad is 1 nearly every time. The list goes most printable first, so the answer is on top.

It's cheap, too. Each digest and iteration count runs EVP_BytesToKey once, and every key length is a slice of that same output. Only the last block gets decrypted until its padding holds. Without flags you get md5, sha1 and sha256 at 128, 192 and 256 bits, one iteration. Nine tries. --digests and --key-lengths take JSON arrays too, like '["md5","sha256"]'. The iteration counts add up to at most a million hash passes across the digests.

Nothing at all? Wrong passphrase, or a setting outside the grid. Or openssl enc -pbkdf2, which writes the same header but derives the key with PBKDF2. This format doesn't take that.

import { probePassphrase } from "@agntn/ciphers";

const probe = probePassphrase(blob, "secret", { iterations: [1, 1000, 10000] });
probe.hits[0]; // { digest: "sha256", keyLength: 256, iterations: 1, padLength: 2, printable: 1, text: "ATTACK AT DAWN", ... }
probe.expected; // 0.106 for 27 tries

Crib dragging a reused key

One key, two messages. That's the end of the one-time pad. XOR the two ciphertexts and the key cancels out. What's left is one plaintext XOR the other. Guess a word in one, and the other shows through under it.

The guess is the crib. Dragging means trying it at every byte. Here are three ciphertexts from Dan Boneh's Many Time Pad exercise, all under one key:

ciphers crib '["315c4eeaa8b5f8aa…","234c02ecbbfbafa3…","32510ba9babebbbe…"]' --crib " the " --limit 5

Crib " the " across 3 ciphertexts (lang=en), 299 places, most readable first:

1. #0 at 70 (score -2.76): #1 " of c", #2 "ore t"
2. #2 at 60 (score -2.89): #0 "lso f", #1 "rner "
3. #1 at 35 (score -3.21): #0 "arucm", #2 " oude"
4. #2 at 70 (score -3.32): #0 "ore t", #1 "oike7"
5. #0 at 13 (score -3.32): #1 "robab", #2 "ssage"
(294 more places ranked lower)

#0 at 70 puts the crib in ciphertext 0 from byte 70. Under that guess the other two read " of c" and "ore t". That's English. Lines 2 and 5 too. Line 3 is junk, nobody writes arucm. The score is the mean log probability of the revealed bytes as text: spaces and common letters score high, control bytes sink. Ciphertexts and bytes count from 0.

Found one that reads? Accept it:

ciphers crib '["315c4eea…","234c02ec…","32510ba9…"]' --crib "ssage" \
  --known '[{"message":0,"offset":70,"text":" the "}]'

Accepted places fix key bytes. A place that disagrees with them drops out, and so does one that only repeats them. Every call prints the key so far and each plaintext, with · where nothing is known yet. Then ssage turns out to sit in ciphertext 2 at 13, and ciphertext 0 reads the there. A plaintext grows a few bytes per call, and a half known word begs for a longer crib.

Got only two ciphertexts? Then every place shows up twice. The crib in #0 reveals #1, the crib in #1 reveals #0, same bytes. Which side is it on? Two can't tell you. A third one can, the wrong side reveals junk there.

ciphers_crib_drag does the same for a model. known is an argument, so the drag lives in the calls, not in a server. One accepted place per call.

import { dragCrib, showCribBytes } from "@agntn/ciphers";

const drag = dragCrib([first, second, target], { crib: " the ", limit: 5 });
drag.candidates[0]; // { message: 0, offset: 70, score: -2.76…, reveals: [...] }
showCribBytes(drag.candidates[0].reveals[1].bytes); // "ore t"

showCribBytes prints bytes the way the command does: text as text, anything else as \xNN, unknown as ·. This is why a one-time pad gets used one time.

Hidden in plain sight

Some messages aren't encrypted at all. They sit in the text, one letter per line or per word, and the text around them is a decoy. Here's a German telegram from 1917, the one Kahn tells in The Codebreakers:

ciphers hidden "PRESIDENT'S EMBARGO RULING SHOULD HAVE IMMEDIATE NOTICE. GRAVE SITUATION AFFECTING INTERNATIONAL LAW. STATEMENT FORESHADOWS RUIN OF MANY NEUTRALS. YELLOW JOURNALS UNIFYING NATIONAL EXCITEMENT IMMENSELY."

Hidden text readings (68 tried, 57 with 5 or more A-Z letters, lang=en), most like the language first:

   0.19  pick=word letter=1 -> PERSHINGSAILSFROMNYJUNEI
  -0.07  pick=every-letter every=8 start=4 -> SMLLMOVITRLEERNLJUNENE
  -0.08  pick=every-letter every=9 start=7 -> EGHINVONTSFWALOIITE
  ...

Pershing sails from NY June 1. First letter of every word.

Without --pick the command tries the usual places. The first, second and last letter of each line, word, sentence and paragraph. Both diagonals. Every nth letter for n up to 10, from every start. That's 68 readings. A reading with fewer than five letters A to Z is dropped, a short one fits anything by chance. The rest get ranked.

How? Not by single letters. Every second letter of English prose has the same letters as English, just in the wrong order. So the ranking looks at pairs. It takes how often a pair shows up in English, against how often its two letters would meet by chance. The mean over the reading is the number on the left. Above 0 the pairs read like English. Near 0 or below, the letters were picked at random. The opening of A Tale of Two Cities hides nothing, and its best reading stays below 0.

Polish and Japanese have no pair table yet. With --lang pl or ja the readings rank by fit, single letters, and the number goes away. Read past the top line there.

Every nth word is never ranked. Words taken out of a text read like the language whether they hide something or not. So that one you ask for:

ciphers hidden "SALUDOS LOVED ONE SO TODAY I HEARD FROM UNCLE MOE ..." --pick every-word --every 5 --start 5
# TODAY MOE TOLD ME HE TESTED POSITIVE FOR METHAMPHETAMINES

That's a prison letter the FBI published in 2011. --pick reads one place and prints only that. line, word, sentence and paragraph take --letter, 1 for the first, -1 for the last. every-letter and every-word take --every and --start. diagonal takes --start and --direction down-left, which counts from the end of each line.

Only letters count. Spaces, digits and punctuation are never picked, and a line with no letters is skipped. That matters for Poe's A Valentine. The first letter of line one, the second of line two and so on spell Frances Sargent Osgood, but only if you count letters, not characters. And the closing poem of Through the Looking-Glass spells Alice Pleasance Liddell down its lines, which the ranking puts first.

import { readHiddenText, rankHiddenText } from "@agntn/ciphers";

readHiddenText(poem, { pick: "diagonal" }); // "Francessargentosgood"
rankHiddenText(carroll).candidates[0].text; // "ALICEPLEASANCELIDDELL"