Hack You CTF 2014 - Crypto 100 - Easy One - [Team SegFault]

Source file of the encryption algorithm was given. Also, we have plain text and cipher text combination for one message

 FILE* input  = fopen(argv[1], "rb");
 FILE* output = fopen(argv[2], "wb");
 char k[] = "CENSORED";
 char c, p, t = 0;
 int i = 0;
 while ((p = fgetc(input)) != EOF) {
     c = (p + (k[i % strlen(k)] ^ t) + i*i) & 0xff;
     t = p;
     i++;
     fputc(c, output);
 }

Find the key:

[*] Algorithm coverts one byte of plain text to one byte of cipher text using equation of the form c = (p + (k[i % len(k)] ^ t) + i*i) mod 256
[*] Using the plain text - cipher text combination, the equation can be written as k[i] = ((c[i] - (i*i) - p[i]) ^ t) & 0xff to find the key. Here c,i,p and t are known values

The key used is VeryLongKeyYouWillNeverGuess. Once the key is found, the decryption algorithm is straight forward. Below is the code

#!/usr/bin/env python

plain_text = open('msg001','r').read().strip()
cipher_text = open('msg001.enc','r').read().strip()

plain_text = [ord(i) for i in plain_text]
cipher_text = [ord(i) for i in cipher_text]

t = 0
key = ''

for i in range(len(plain_text)):
    c = ((cipher_text[i] - (i*i) - plain_text[i]) ^ t) & 0xff
    key += chr(c)
    t = plain_text[i]
#print key

cipher_text = open('msg002.enc','r').read().strip()
key = 'VeryLongKeyYouWillNeverGuess'

key= [ord(i) for i in key]
cipher_text = [ord(i) for i in cipher_text]

t = 0
plain = ''

for i in range(len(cipher_text)):
    c = (cipher_text[i] - (key[i % len(key)] ^ t) - i*i) & 0xff
    plain += chr(c)
    t = c
print plain

Flag for the challenge is CTF{6d5eba48508efb13dc87220879306619}