Wednesday, 12 October 2016

CODZ ciphers, testing RC4 in python

Some of the CODZ ciphers consist of binary data, and it is not clear what sort of cipher has created them. Notably 2,3,7 and 10 are base64 converted binary data. Ciphers 4,5,11, and 14 are hex data. I will be dealing with 2,3,7 and 10 in this post.

Cipher 2 decrypts to: dd3ed3b56ff21a7751ebbbaf6ffd1086f190a6b29b49c2a47e656bcc91768346df0321db23505874cb9f8a71d978490e and 3a411e6479e27ee31d601d486c7c807dbd20d85273ad599f90a7126caae9406be1434fdbbeeefa45d9c6cbf5ad4fb4f7 depending on wether you read the base64 data forwards or backwards. Either way you get exactly 384 bits of binary, which is suspiciously similar to the SHA384 hash length. I tried running john the ripper on the two hashes for a day or so, but nothing turned up. Either it is not SHA384 or we're gonna need some better word lists.

Ciphers 3,7 and 10 are longer. I have tested their entropy after breaking them up into different length segments, and the entropy is pretty much always equal to random, this means they are highly unlikely to have been generated by a classical cipher. More likely something modern like RC4. I have tried decrypting the 3 ciphers as RC4 using a wordlist for the keys, but no luck.

Code for testing entropy:

from base64 import decodestring
from bitarray import bitarray
from binascii import b2a_hex
from math import log

ctext = 'iW9cXmzOU7ZuZBtW40b3ng...' # I cut the rest of the cipher off to save space
data = decodestring(ctext)

a = bitarray(endian='little')

for bitlen in range(1,16):
  code = {}
  for i in range(2**bitlen):
    string = format(i,'0'+str(bitlen)+'b')
    code[i] = bitarray(string)
  num = a.decode(code)
  freq = {}
  for i in num:
      if i in freq: freq[i] += 1.
      else: freq[i] = 1.
  N = len(num)
  en = 0
  for i in freq.values():
    p = i/N
    en -= p*log(p)
  print bitlen,' entropy: ',en,' if random: ',-log(1./2**bitlen)

For decrypting RC4, we keep the key corresponding to the decrypt with the lowest entropy, if the wrong key is used something high entropy will result, but if it decrypts to text we should get something easily identifiable. Python code for decrypting RC4 with a word list:

data = base64.b64decode(ctext)

def rc4(data,key):
    S = range(256)
    j = 0
    out = []

    #KSA Phase
    for i in range(256):
        j = (j + S[i] + ord( key[i % len(key)] )) & 0xFF
        S[i] , S[j] = S[j] , S[i]

    #PRGA Phase
    i = j = 0
    for char in data:
        i = ( i + 1 ) & 0xFF
        j = ( j + S[i] ) & 0xFF
        S[i] , S[j] = S[j] , S[i]
        out.append(chr(ord(char) ^ S[(S[i] + S[j]) & 0xFF]))
    return out
# find a good word list and use it here
keylist = open("C:\Users\james\Desktop\cipher_stuff\simplesub_word\\count_1w.txt")

besten = 10e10
bestkey = ""
bestout = ""
for count,key in enumerate(keylist):
  key = key.split()[0].strip()
  for i in range(3):
    if i == 0: key = key[0].upper() + key[1:].lower()
    elif i == 1: key = key.upper()
    else: key = key.lower()
    out = rc4(data,key)
    # compute entropy of the decoded text
    freq = {}
    for i in out:
        if i in freq: freq[i] += 1.
        else: freq[i] = 1.
    N = len(out)
    en = 0
    for i in freq.values():
        p = i/N
        en -= p*log(p)
    if en < besten:
        besten = en
        bestkey = key
        bestout = ''.join(out[:20])
  if count % 500 == 0: print count, key, bestkey,besten,bestout
print count, key, bestkey,besten,bestout

Sunday, 9 October 2016

CODZ 8. sha_paper

I havn't actually played the game, I just like breaking codes.

Apparently there are some new CODZ ciphers, the transcripts can be seen here:

This post will look at the following cipher:

bx re yh zy bf lm kt ut yg se tb sx ky co jh km aq we tx wx
cy ji ut vt kn vc gx aw ij av qn lg ef fj uq bd kn sv cx fn
je wr rk kn cg aw xq vn zf li fh vz wt ta ia ij zf eh uf tj 
qm yg hl yq cx ij vw ig de qz tg nj rs er vk tm sa yv tw hr 
hs lt vy kr qc tv gh hb jn yb qh er ut gk et cs wv jl rh xo 
wr ex hr xt zi kc xs qs fd wd cm ku ah fh fj lf ui ly sh vf 
au xm hx qw dl gi cx vb dh wt xm kv un ej kt kt ye cq jd ef 
eh zv xt he uz tg cl jw nr tw ur vo jt jo ru iq iy rz ey ho 
gd nq yn bq ul ai fh bu ji ho nw qg yg vj if yv zu id jc gh 
ke xr qf cq ra it gw dl fc gq ti iu qu ny vr gy sj rh iu hi 
wr mv ym zi lk re vk xu ry uq gs ve gd yn bq ch ky er qh jr 
ho ya el ky zj ei hz cb if dk

There are 460 characters total, 25 distinct characters, IC is 0.0418. Counts:

a: 12, c: 18, b: 11, e: 23, d: 12, g: 20,
f: 19, i: 24, h: 29, k: 21, j: 23, m: 9,
l: 14, o:  7, n: 15, q: 23, s: 12, r: 24,
u: 20, t: 26, w: 18, v: 23, y: 25, x: 19,
z: 13

Since this cipher uses pairs of characters, and there are 25 distinct characters, it is highly likely to be foursquare, bifid, playfair or some other digraphic substitution cipher based on a 5 by 5 grid. Usually I and J are combined, but it looks like in this cipher P is missing, so It was combined with something else.

Other ciphers that use a 25 letter alphabet include BAZERIES, BIFID, CADENUS, CHECKERBOARD, CM BIFID, FOURSQUARE, PHILLIPS, PHILLIPS-C, PHILLIPS-RC, PLAYFAIR, SERIATED PLAYFAIR, TRI-SQUARE, TWIN BIFID and TWO-SQUARE. Unfortunately this is quite a lot, and trying them all will be time consuming.

I am trying 4 variants of the above the cipher, the original cipher: bkreyh..., the reversed cipher: kdfibc..., forward but with each pair reversed: xberhy..., and each pair forwards, but the pairs from last to first: dkifcb... .

BAZERIES is a substitution + transposition, which means its IC would be that of normal text. Since the IC of this text is ~0.04, I am confident in saying it is not BAZERIES . It also can't be CADENUS because the message length is not a multiple of 25. CHECKERBOARD uses a 5x5 key square, but can at maximum use 20 characters in the ciphertext.

Possibilities after thinking a bit: BIFID, CM BIFID, TWIN BIFID, FOURSQUARE, PHILLIPS, PLAYFAIR, SERIATED PLAYFAIR, TRI-SQUARE, and TWO-SQUARE. The next piece of information is that none of the letter pairs in the ciphertext contain a doubled letter e.g. AA, BB, CC etc. This is highly unlikely to occur with most ciphers, but it is a peculiarity of PLAYFAIR. So this puts playfair near the top of the list.

I have tried the playfair cryptanalysis program over here on all four variants and it hasn't worked. I did a test sentence of the same length which it cracked in just a few iterations, so I am fairly confident in saying it is not a vanilla playfair, or at least if it is then something else is going on as well (such as some sort of route cipher).

For bifid, there is no obvious period identifiable (see here for how to compute it), and some quick simulated annealing didn't turn anything up.

Regarding the possibility of routes, 460 has a lot of factors: 2x230,4x115,5x92,10x45,20x23. This means lots of routes around the rectangles when the characters are lined up right. Exploring this would add a lot of extra time to the cracking effort.

The quest continues...

Saturday, 8 October 2016

CODZ cipher 13. kin_paper_torn straddle checkerboard?

I haven't actually played the game, I just like breaking codes.

Apparently there are some new CODZ ciphers, the transcripts can be seen here:

This post will look at the following cipher:



EDIT: I'm beginning to think that this is not the final decryption, that they have used a slightly modified method which results in some extra characters. There is too much decrypted for me to think that it is totally wrong, but I believe there is an extra step I'm missing somewhere in the middle. </EDIT>

The ciphertext is a straddle checkerboard. Converting the 10 symbols to digits, and reversing the string, we get (using @=0, C=1, B=2, ?=3, >=4, F=5, A=6, G=7, = is 8, <=9, Note that this mapping is arbitrary, and any other mapping would work just as well):

320143238736103135367632361414143975932181021439731321 83607321012710183138733341010341033711143635014343210

See the link for a description of the straddle checkerboard. Using this key:


The message decrypts to OCTOBERNSAREFORTTTHEYFOUNDTHESOURCEONVENUSBEGINNINGEXTRACTION. with spaces: OCTOBER NSA REPORT T THEY FOUND THE SOURCE ON VENUS BEGINNING EXTRACTION. There are likely a couple characters wrong still, or the slight mistakes are intentional.

My approach

There are 10 different characters, 107 in total. The counts for each character are as follows: A: 6, @: 11, C: 24, B: 9, G: 8, F: 3, =: 5, <: 3, ?: 28, >: 10. It so happens that 10 characters can be converted to digits, so we can look at existing ciphers that use digits e.g. the straddle checkerboard, GRANDPRE, MONOME-DINOME, MORBIT, NIHILIST SUBSTITUTION, POLLUX or TRIDIGITAL. There are probably also more.

Converted to digits it is: 0123434105363411173301430101433378313810172101237063812313 7934120181239579341414163236763531301637832341023. If you reverse it, you get 320143238736103135367632361414143975932181021439731321 83607321012710183138733341010341033711143635014343210. I'll be talking about the non reversed ciphertext, but I'll be applying all the steps to both just in case. Using the above ciphertext and this calculator, we can compute the most likely cipher algorithms, and it has concluded that it is most likely a nihilist substitution. Unfortunately, it can't be nihilist because it has an odd number of characters, and nihilist requires even. It can't be grandpre for the same reason. CryptoCrack (from here) can't break it as GRANDPRE, MORBIT, NIHILIST SUBSTITUTION, POLLUX or TRIDIGITAL.

That really only leaves straddle checkerboard as the last candidate from the ciphers I can think of. Assuming it is a straddle checkerboard, we first have to find to location of the blanks in the key. If we can find those locations, we can decrypt in with any letters and we will have a plain substitution cipher. There are 10 choose 2 = 45 ways of putting the first 2 blanks, and 20 choose 2 = 190 ways of putting the second. This means there are 45*190 = 8550 possible configurations of putting the blanks. Some of these will be invalid though, e.g. the key:

   0 1 2 3 4 5 6 7 8 9
   f k m   c p d   y e
3: h b i g q r o s a z
7: l u t j n w v x    

can't have 38, 39, 78 or 79 in the ciphertext. If these numbers do exist in the ciphertext, then this is not a possible set of blanks. After trying to decrypt our ciphertext with all 8550 keys, only 3340 are actually valid blank positions. In addition to this, most of the blank positions result in identical outputs to other blank positions, so we can discard all the duplicate outputs, leaving only 45 substitution ciphers that we have to try.

After trying to decrypt the resulting 45 substitution ciphers from the forward cipher and another 45 from the reversed cipher, one of the decrypts, namely SALSYFWIRVWFESWLLLZFDHSOIBLZFRSOWAFSIKFIORYFTUIIUITFJLWVALUSI, can be decrypted to OCTOBERNSAREFORTTTHEYWOUNDTHESOURCEONMENUSBEGINNINGEXTRACTION. There are likely a couple characters wrong still, or the slight mistakes are intentional.

This is a bit of python code that will spit out the possible substitution ciphers for a given straddle checkerboard ciphertext (Please excuse the rough code, I wrote everything pretty quick):

I have actually since discovered there is a much shorter way of getting to the final candidates, but I'll leave this code here. To rank the final candidates, use index of coincidence, the correct one is almost always the one with the IC closest to 0.07.

import random
import sys
from itertools import combinations

ctext = '6909746723099383772753870703607230943837727093872638757438333832743772974928387272384175943874720383270'

''' decrypt a straddle checkerboard cipher ctext given a key
 key should look like e.g. 'fkm.cpd.yehbigqrosazlutjnwvx..' 
 it should be 30 chars in length, and have 4 dots. 
 2 dots in the first 10 chars, and 2 in the last 20.
 - if it returns 0 in the first result, the key was invalid
def scdecrypt(ctext,key):
    dotpos = []
    for i,k in enumerate(key): 
        if k == '.': dotpos.append(i)
    output = ""        
    if dotpos[0] > 9: return 0,output
    if dotpos[1] > 9: return 0,output
    if dotpos[2] < 10: return 0,output
    if dotpos[3] < 10: return 0,output
    flag = 0
    for cc in ctext:
        c = int(cc)
        if key[c] != '.' and flag == 0: 
            output += key[c]
        elif flag == 1: 
            if key[10+c] == '.': return 0,""
            output += key[10+c]
            flag = 0
        elif flag == 2: 
            if key[20+c] == '.': return 0,""        
            output += key[20+c]
            flag = 0
        elif c == dotpos[0]: flag = 1     
        elif c == dotpos[1]: flag = 2
            return 0,output
    return 1,output

''' see if word and ct are possibly the same substitution cipher 
returns 0 in the first part of the result if they are not '''        
def canstart(ct,word):
    key = {}
    if len(word) > len(ct): return 0,key
    for i,char in enumerate(word):
        if ct[i] not in key: 
            if char not in key.values():           
                key[ct[i]] = char
            else: return 0,key
        elif key[ct[i]] == char: pass
        else: return 0,key
    return 1,key

# get a list of all the possible blank spot permutations
dp = []
a = combinations(range(10),2)
for i in a:
    b = combinations(range(20),2)
    for j in b:
        dotpos = list(i)

# we have all possible blank spot positions, try decrypting with each and see if we can discard any
count = 0
texts = []
for d in dp:
    for pos in d: key.insert(pos,'.')
    bool,ptext = scdecrypt(ctext,key)
    if bool == 0: continue
    count += 1

# now we have a heap of valid decrypts, discard duplicates
lst = []
for i in range(len(texts)):
    cs = False
    for j in range(len(lst)):
        if canstart(lst[j],texts[i])[0]: cs = True
    if not cs: lst.append(texts[i])

# print the unique possible decrypts, they must be solved as substitution ciphers
for i,j in enumerate(range(len(lst))):
    print lst[j]

Thursday, 29 September 2016

The CODZ Mob of the dead ADFGX cipher

The ADFGX cipher from CODZ has been unsolved for quite a while, even with many people trying to break it. I thought I would document some of my progress on this page. Other Discussions:

There are of course other discussions around, but I'm not sure how much useful information is in them. Apparently the developers of the game have said that this was one of the early ciphers they made, and it may be too difficult to break. But we will have a go anyway.

EDIT: later thoughts

This paragraph has been added later, but I have decided that most of the analysis on this page is useless because I can't crack length 34 substitution ciphers using all the knowledge and code I have (which is mostly over here). I have made a couple of my own length 34 substitution ciphers, and I can't get anywhere even close to cracking them (If it were e.g. 100 characters it would be totally possible). So I might think about how to solve that problem. But until then, there is little point in the analysis below, because even without the ADFGX keyword steps it is impossible to solve (for me at the moment).

Some Observations

An ADFGX cipher with 68 characters means the plaintext has only 34 characters. Unfortunately this is close to the unicity distance for substitution ciphers, so it may be unbreakable from that perspective unless some extra information can be found. Most substitution cipher solvers can't break text this short as there are incorrect solutions that will score higher (using ngram scorers) than the true decryption. The ciphertext is as follows:


There are two components to an ADFGX key: a keyword e.g. 'GERMAN' and a substitution key for the Polybius square. Much of the discussion on various forums is about finding the ADFGX keyword, however I think this is unneccessary. With a length 6 keyword there are only 6! = 720 possible keys, and only 48 possibilities if we assume the last two columns are the first 2 characters of the keyword (this is a reasonable assumption because they are dangling at the end like that).

This means we can decrypt the cipher using each of the possible 48 keyword permutations (this gives us 48 Polybius square ciphers), then use an arbitrary Polybius square key e.g. "ABCDEFGHIKLMNOPQRSTUVWXYZ" to get a 48 ciphers which are just a substitution cipher. From here, ideally, it would just take a good substitution cipher solver applied to each of the 48 candidates to determine which can be broken resulting in English text. It turns out that, due to symmetry of the polybius square, there are actually only 24 possible substitution ciphers, which makes our job easier. Unfortunately 34 characters is just too short for most substitution cipher crackers, or alternatively one of the previous assumptions is wrong.

Of course I am not the first to think of this, and many people have tried this exact procedure to no avail.

The Repeated Rows

There are two identical rows in the ciphertext: FDGFFG FDGFFG. This means, regardless of the keyword, there will be 6 letters in the plaintext that consists of 3 letters repeated e.g. THETHE. How often does this sort of thing occur in English? Quite often actually. I have a corpus of around 50 million English sentences from news websites and wikipedia which can be used to determine the frequency of this sort of occurrence.

It so happens that in every million english sentences, you would expect around 34,000 to have a repeated triple like this, consisting of around 1100 distinct sequences (i.e. many of the 34,000 are repeats of previously seen sequences). In total after 50 million sentences 6164 distinct 6-character sequences were found, occurring 1.89 million times in total. The most common are: THATHA (235167), ANDAND (176355), INGING (122567), THETHE (89908), REAREA (67514), NTONTO (42511), ETHETH (36691), SSESSE (31542), NDINDI (31259), ASSASS (29754), followed by many other rarer ones.

To utilise this information, it may be possible to fix the letters in the substitution key so that the repeated 3-letter combo is e.g. fixed to THATHA. This might make it easier for an n-gram scorer to identify the rest of the letters if some are known. I haven't tried this yet.

A Dictionary attack on the Polybius bit

Apparently past ciphers in the game have used keywords to generate keys for ciphers. This means we may be able to use a dictionary attack to generate polybius keys for decrypting the adfgx. This means we don't have to crack the substitution component, which would make things easier.

After doing this, the best decryption I could get from the 48 keyword permutations was VOGANHEMOWIGHHEHHENHEKANZEWESINHAC with a 4-gram score of -166. So this means either the polybius key was not a dictionary word (my dictionary has about 500k words in it), the adfgx keword is not length 6, or something other assumption we have made is wrong.

Friday, 23 October 2015

Installing RNNLIB on Ubuntu 14.04

I have previously looked at CURRENNT, but now I'll go through how to use RNNLIB, which is a different rnn library. RNNLIB doesn't use cuda, but it can do multi-dimensional rnns and also can use the CTC loss function which can be handy. Let's get right into it. Download RNNLIB from sourceforge here.

You'll have to install the netcdf library:

sudo apt-get install libnetcdf-dev

You should also install boost: sudo apt-get install libboost-all-dev. I have libboost1.55 installed already.

There is an installation guide over here if you want some additional pointers. First run configure and make:


Fixing the errors

The first error I got was:

Mdrnn.hpp:239:6:   required from here
Helpers.hpp:311:41: error: ‘range_min_size’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
  size_t size = range_min_size(r1, r2, r3);

To fix this one you need to put templates code above the code that uses them. Moving the 4 templates for range_min_size to the top of the file should fix all the issues in Helpers.hpp. I cut lines 532-547 and pasted the lines to line 163 of Helpers.hpp.

Running make again I got: Container.hpp:157:24: error: ‘resize’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation. To fix this one change on line 157 of containers.hpp "resize" to "this->resize."

That was all I needed to do to get this version installed.

Thursday, 22 October 2015

Running the CURRENNT rnn library

Previously I have made posts about installing Currennt and writing data files for Currennt. This post will be about actually running it assuming it is installed and the data files have been written.

The first thing you'll need to specify is the config file, which I called config.cfg. Its contents look like this:

max_epochs_no_best   = 20
max_epochs           = 100
learning_rate        = 1e-5
network              = network.jsn 
train                = true
train_file           =
val_file             =
weights_dist         = normal
weights_normal_sigma = 0.1
weights_normal_mean  = 0
stochastic           = true
validate_every       = 1
parallel_sequences   = 20
input_noise_sigma    = 0
shuffle_fractions    = true
shuffle_sequences    = false

Many of the options are described on the currennt wiki. The network.jsn is a file that I will describe down below and contains the network configuration e,g, number of layers, layer types etc. The files and were created using the MATLAB script in a previous post. If you set parallel sequences too high you'll run out of memory on your GPU, but higher means it will train faster.

The network configuration

Info on the network configuration can be found over here. The network I used looks like this:

    "layers": [
            "size": 20,
            "name": "input",
            "type": "input"
            "size": 200,
            "name": "level_1",
            "bias": 1.0,
            "type": "blstm"
            "size": 200,
            "name": "level_2",
            "bias": 1.0,
            "type": "blstm"
            "size": 4,
            "name": "output",
            "bias": 1.0,
            "type": "softmax"
            "size": 4,
            "name": "postoutput",
            "type": "multiclass_classification"

actually running stuff

Now to run things, just do:

/path/to/currennt/build/current config.cfg

and everything should work, just wait for it to finish training. This will output a file called trained_network.jsn which we will need during testing. For testing, you will need another HDF5 file just like the training one, except with the test sequences in it. For testing we have another config file which I called ff_config:

network          = trained_network.jsn
cuda             = 0
ff_output_format = csv
ff_output_file   = test_results
ff_input_file    =

Notice that cuda is off, I found it ran faster in feed forward mode without cuda. To do testing run:

/path/to/currennt/build/current ff_config.cfg

Now all you need to do is parse the output file test_results.csv if you need the individual predictions, it is in comma separated variable format so it is not hard to work out.

Writing HDF5 files for Currennt

In a previous post I explained how to install currennt. This post will deal with writing the data files. The next post will be about actually running things. Currennt uses data written to HDF5 files, so whatever format your data is in you have to write it to a HDF5 file then use currennt to process it. This post will use matlab to generate the HDF5 files, but e.g. python could easily do it as well.

Recurrent neural networks predict things about sequences, so for this example we'll be using protein sequences as the example. Predicting things like phonemes for speech files is exactly the same. For each amino acid in a protein, we'll be predicting the secondary structure, which can be C, E, H or X. The training information will be a L by 20 matrix, this means the protein is of length L and each amino acid has 20 features (we'll use PSSM features, of which there are 20, extracted using PSI-BLAST.). I will just be using 400 training proteins and 100 test proteins, it is often beneficial to try things out on small datasets because you get the results quicker and can figure out problems quicker.

HDF5 files

Currennt needs the data written to HDF5 files, but it needs a heap of extra parameters written to the files as well as the data, this section will specify what you need to write to them, then I'll put some actual MATLAB code in the next section.

The data we'll be using is protein data, there will be 400 training proteins, each of length L (the sequences can be anywhere from about 20 to 2000 in length). There are 20 features per amino acid, this means each protein is represented by a matrix of size 20 by L, and there are 400 matrices.

These parameters are for classification, if you want something else like regression you may have to use different dimensions and variables e.g. numLabels only makes sense for classification. HDF5 files have dimensions and variables, and they need to be specified separately. The dimensions that currennt needs specified in the HDF5 files are as follows:

numTimesteps: the total number of amino acids if you lined 
      up all the proteins end to end
inputPattSize: the number of features per amino acid, in this case 20.
numSeqs: the number of proteins
numLabels: the number of classes we want to predict, in this case 4
maxLabelLength: the length of the longest string used to hold 
      the class label name
maxTargStringLength: to be honest I'm not sure about this one, I 
      just set it to be 5000 and things seem to work
maxSeqTagLength: this is another one I'm not sure about. I set 
      it to 800 and it seems to work.

Now we have specified the dimensions, now we want to specify the variables i.e. the actual data we will be using for features and labels. They are specified like so:

seqTags,          size: maxSeqTagLength x numSeqs
numTargetClasses: size: 1
inputs,           size: inputPattSize x numTimesteps
seqLengths,       size: numSeqs
targetClasses,    size: numTimesteps
labels,           size: maxLabelLength x numLabels

You then just have to write the relevant data to each of the variables and run currennt. Some MATLAB code that does all this stuff is provided below. Note that this matlab file will just create one file for e.g. training. If you want a separate dataset for e.g. testing (strongly recommended) then you will need to run the code a second time changing the file names and protein dataset.