{"id":13715241,"url":"https://github.com/0b01/SimGAN-Captcha","last_synced_at":"2025-05-07T04:30:43.345Z","repository":{"id":80974761,"uuid":"98011560","full_name":"0b01/SimGAN-Captcha","owner":"0b01","description":"Solve captcha without manually labeling a training set","archived":false,"fork":false,"pushed_at":"2024-04-24T11:00:11.000Z","size":14872,"stargazers_count":435,"open_issues_count":7,"forks_count":80,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-04-18T09:55:32.408Z","etag":null,"topics":["captcha-solving","gan","generative-adversarial-network","keras","neural-network","simgan"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/0b01.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-22T06:23:20.000Z","updated_at":"2025-01-07T14:00:59.000Z","dependencies_parsed_at":"2024-04-26T16:31:18.510Z","dependency_job_id":null,"html_url":"https://github.com/0b01/SimGAN-Captcha","commit_stats":null,"previous_names":["rickyhan/simgan-captcha"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0b01%2FSimGAN-Captcha","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0b01%2FSimGAN-Captcha/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0b01%2FSimGAN-Captcha/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0b01%2FSimGAN-Captcha/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/0b01","download_url":"https://codeload.github.com/0b01/SimGAN-Captcha/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252813611,"owners_count":21808359,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["captcha-solving","gan","generative-adversarial-network","keras","neural-network","simgan"],"created_at":"2024-08-03T00:00:56.367Z","updated_at":"2025-05-07T04:30:38.117Z","avatar_url":"https://github.com/0b01.png","language":"Jupyter Notebook","funding_links":[],"categories":["Łamanie","Jupyter Notebook"],"sub_categories":["Ogólne"],"readme":"### Capsolver\n\n\u003e [![image](https://github.com/0b01/SimGAN-Captcha/assets/1768528/4313c9cc-11e8-412b-aee4-3dbc09bd19a3)](https://www.capsolver.com)\n\u003e \n\u003e [Capsolver.com](https://www.capsolver.com/?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha) is an AI-powered service that specializes in solving various types of captchas automatically. It supports captchas such as [reCAPTCHA V2](https://docs.capsolver.com/guide/captcha/ReCaptchaV2.html?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha), [reCAPTCHA V3](https://docs.capsolver.com/guide/captcha/ReCaptchaV3.html?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha), [hCaptcha](https://docs.capsolver.com/guide/captcha/HCaptcha.html?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha), [FunCaptcha](https://docs.capsolver.com/guide/captcha/FunCaptcha.html?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha), [DataDome](https://docs.capsolver.com/guide/captcha/DataDome.html?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha), [AWS Captcha](https://docs.capsolver.com/guide/captcha/awsWaf.html?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha), [Geetest](https://docs.capsolver.com/guide/captcha/Geetest.html?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha), and Cloudflare [Captcha](https://docs.capsolver.com/guide/antibots/cloudflare_turnstile.html?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha) / [Challenge 5s](https://docs.capsolver.com/guide/antibots/cloudflare_challenge.html?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha), [Imperva / Incapsula](https://docs.capsolver.com/guide/antibots/imperva.html?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha), among others.\n\u003e \n\u003e For developers, Capsolver offers API integration options detailed in their [documentation](https://docs.capsolver.com/?utm_source=github\u0026utm_medium=banner_github\u0026utm_campaign=SimGAN-Captcha), facilitating the integration of captcha solving into applications. They also provide browser extensions for [Chrome](https://chromewebstore.google.com/detail/captcha-solver-auto-captc/pgojnojmmhpofjgdmaebadhbocahppod) and [Firefox](https://addons.mozilla.org/es/firefox/addon/capsolver-captcha-solver/), making it easy to use their service directly within a browser. Different pricing packages are available to accommodate varying needs, ensuring flexibility for users.\n\n\n# SimGAN-Captcha\n\nWith simulated unsupervised learning, breaking captchas has never been easier. There is no need to label any captchas manually for convnet. By using a captcha synthesizer and a refiner trained with GAN, it's feasible to generate synthesized training pairs for classifying captchas.\n\n## Link to paper: SimGAN by Apple \n\n[PDF](https://arxiv.org/pdf/1612.07828v1.pdf)\n[HTML](https://machinelearning.apple.com/2017/07/07/GAN.html)\n\n![SimGAN](http://www.fudzilla.com/images/stories/2016/December/apple-simgan-generative-adversarial-networks.jpg)\n\n# The task\n\n[HackMIT Puzzle #5](https://captcha.delorean.codes/u/rickyhan/).\n\nCorrectly label 10000 out of 15000 captcha or 90% per character.\n\n## Preprocessing\n\n### Download target captcha files\n\nHere we download some captchas from the contest website. Each batch has 1000 captchas. We'll use 20000 so 20 batches.\n\n\n```python\nimport requests\nimport threading\nURL = \"https://captcha.delorean.codes/u/rickyhan/challenge\"\nDIR = \"challenges/\"\nNUM_CHALLENGES = 20\nlock = threading.Lock()\n```\n\n\n```python\ndef download_file(url, fname):\n    # NOTE the stream=True parameter\n    r = requests.get(url, stream=True)\n    with open(fname, 'wb') as f:\n        for chunk in r.iter_content(chunk_size=1024): \n            if chunk: # filter out keep-alive new chunks\n                f.write(chunk)\n                #f.flush() commented by recommendation from J.F.Sebastian\n    with lock:\n        pass\n        # print fname\n\n\nts = []\nfor i in range(NUM_CHALLENGES):\n    fname = DIR + \"challenge-{}\".format(i)\n    t = threading.Thread(target=download_file, args=(URL, fname))\n    ts.append(t)\n    t.start()\nfor t in ts:\n    t.join()\nprint \"Done\"\n```\n\n    Done\n\n\n### Decompression\n\nEach challenge file is actually a json object containing 1000 base64 encoded jpg image file. So for each of these challenge files, we decompress each base64 strs into a jpeg and put that under a seprate folder.\n\n\n```python\nimport json, base64, os\nIMG_DIR = \"./orig\"\nfnames = [\"{}/challenge-{}\".format(DIR, i) for i in range(NUM_CHALLENGES)]\nif not os.path.exists(IMG_DIR):\n    os.mkdir(IMG_DIR)\ndef save_imgs(fname):\n    with open(fname) as f:\n        l = json.loads(f.read())\n\n    for image in l['images']:\n        b = base64.decodestring(image['jpg_base64'])\n        name = image['name']\n        with open(IMG_DIR+\"/{}.jpg\".format(name), 'w') as f:\n            f.write(b)\n\nfor fname in fnames:\n    save_imgs(fname)\nassert len(os.listdir(IMG_DIR)) == 1000 * NUM_CHALLENGES\n```\n\n\n```python\nfrom PIL import Image\nimgpath = IMG_DIR + \"/\"+ os.listdir(IMG_DIR)[0]\nimgpath2 = IMG_DIR + \"/\"+ os.listdir(IMG_DIR)[3]\nim = Image.open(example_image_path)\nim2 = Image.open(example_image_path2)\nIMG_FNAMES = [IMG_DIR + '/' + p for p in os.listdir(IMG_DIR)]\n```\n\n\n```python\nim\n```\n\n\n\n\n![png](imgs/output_8_0.png)\n\n\n\n\n```python\nim2\n```\n\n\n\n\n![png](imgs/output_9_0.png)\n\n\n\n### Convert to black and white\nInstead of RGB, binarized image saves significant compute. Here we hardcode a threshold and iterate over each pixel to obtain a binary image.\n\n\n```python\ndef gray(img_path):\n    # convert to grayscale, then binarize\n    img = Image.open(img_path).convert(\"L\")\n    img = img.point(lambda x: 255 if x \u003e 200 or x == 0 else x) # value found through T\u0026E\n    img = img.point(lambda x: 0 if x \u003c 255 else 255, \"1\")\n    img.save(img_path)\n\nfor img_path in IMG_FNAMES:\n    gray(img_path)\n```\n\n\n```python\nim = Image.open(example_image_path)\nim\n```\n\n\n\n\n![png](imgs/output_12_0.png)\n\n\n\n### Find mask\n\nAs you may have noticed, all the captchas share the same horizontal lines. Since this is a contest, it was a function of participant's username. In the real world, these noises can be filtered out using morphological transformation with OpenCV.\n\nWe will extract and save the lines(noise) for later use. Here we average all 20000 captchas and set a threshold as above. Another method is using a bit mask (\u0026=) to iteratively filter out surrounding black pixels i.e.\n\n```\nmask = np.ones((height, width))\nfor im in ims:\n    mask \u0026= im\n```\n\nThe effectiveness of bit mask depends on how clean the binarized data is. With the averaging method, some error is allowed.\n\n\n```python\nimport numpy as np\nWIDTH, HEIGHT = im.size\nMASK_DIR = \"avg.png\"\n```\n\n\n```python\ndef generateMask():\n    N=1000*NUM_CHALLENGES\n    arr=np.zeros((HEIGHT, WIDTH),np.float)\n    for fname in IMG_FNAMES:\n        imarr=np.array(Image.open(fname),dtype=np.float)\n        arr=arr+imarr/N\n    arr=np.array(np.round(arr),dtype=np.uint8)\n    out=Image.fromarray(arr,mode=\"L\")\n    out.save(MASK_DIR)\n\ngenerateMask()\n```\n\n\n```python\nim = Image.open(MASK_DIR) # ok this can be done with binary mask: \u0026=\nim\n```\n\n\n\n\n![png](imgs/output_16_0.png)\n\n\n\n\n```python\nim = Image.open(MASK_DIR)\nim = im.point(lambda x:255 if x \u003e 230 else x)\nim = im.point(lambda x:0 if x\u003c255 else 255, \"1\")\nim.save(MASK_DIR)\n```\n\n\n```python\nim\n```\n\n\n\n\n![png](imgs/output_18_0.png)\n\n\n\n# Generator for real captchas\n\nUsing a Keras built in generator function `flow_from_directory` to automatically import and preprocess real captchas from a folder.\n\n\n```python\nfrom keras import models\nfrom keras import layers\nfrom keras import optimizers\nfrom keras import applications\nfrom keras.preprocessing import image\nimport tensorflow as tf\n```\n\n\n```python\n# Real data generator\n\ndatagen = image.ImageDataGenerator(\n    preprocessing_function=applications.xception.preprocess_input\n)\n\nflow_from_directory_params = {'target_size': (HEIGHT, WIDTH),\n                              'color_mode': 'grayscale',\n                              'class_mode': None,\n                              'batch_size': BATCH_SIZE}\n\nreal_generator = datagen.flow_from_directory(\n        directory=\".\",\n        **flow_from_directory_params\n)\n```\n\n# (Dumb) Generator\n\nNow that we have processed all the real captchas, we need to define a generator that outputs (captcha, label) pairs where the captchas should look almost like the real ones.\n\nWe filter out the outliers that contain overlapping characters.\n\n\n```python\n# Synthetic captcha generator\nfrom PIL import ImageFont, ImageDraw\nfrom random import choice, random\nfrom string import ascii_lowercase, digits\nalphanumeric = ascii_lowercase + digits\n\n\ndef fuzzy_loc(locs):\n    acc = []\n    for i,loc in enumerate(locs[:-1]):\n        if locs[i+1] - loc \u003c 8:\n            continue\n        else:\n            acc.append(loc)\n    return acc\n\ndef seg(img):\n    arr = np.array(img, dtype=np.float)\n    arr = arr.transpose()\n    # arr = np.mean(arr, axis=2)\n    arr = np.sum(arr, axis=1)\n    locs = np.where(arr \u003c arr.min() + 2)[0].tolist()\n    locs = fuzzy_loc(locs)\n    return locs\n\ndef is_well_formed(img_path):\n    original_img = Image.open(img_path)\n    img = original_img.convert('1')\n    return len(seg(img)) == 4\n\nnoiseimg = np.array(Image.open(\"avg.png\").convert(\"1\"))\n# noiseimg = np.bitwise_not(noiseimg)\nfnt = ImageFont.truetype('./arial-extra.otf', 26)\ndef gen_one():\n    og = Image.new(\"1\", (100,50))\n    text = ''.join([choice(alphanumeric) for _ in range(4)])\n    draw = ImageDraw.Draw(og)\n    for i, t in enumerate(text):\n        txt=Image.new('L', (40,40))\n        d = ImageDraw.Draw(txt)\n        d.text( (0, 0), t,  font=fnt, fill=255)\n        if random() \u003e 0.5:\n            w=txt.rotate(-20*(random()-1),  expand=1)\n            og.paste( w, (i*20 + int(25*random()), int(25+30*(random()-1))),  w)\n        else:\n            w=txt.rotate(20*(random()-1),  expand=1)\n            og.paste( w, (i*20 + int(25*random()), int(20*random())),  w)\n    segments = seg(og)\n    if len(segments) != 4:\n        return gen_one()\n    ogarr = np.array(og)\n    ogarr = np.bitwise_or(noiseimg, ogarr)\n    ogarr = np.expand_dims(ogarr, axis=2).astype(float)\n    ogarr = np.random.random(size=(50,100,1)) * ogarr\n    ogarr = (ogarr \u003e 0.0).astype(float) # add noise\n    return ogarr, text\n\ndef synth_generator():\n    arrs = []\n    while True:\n        for _ in range(BATCH_SIZE):\n            arrs.append(gen_one()[0])\n        yield np.array(arrs)\n        arrs = []\n```\n\n\n```python\ndef get_image_batch(generator):\n    \"\"\"keras generators may generate an incomplete batch for the last batch\"\"\"\n    img_batch = generator.next()\n    if len(img_batch) != BATCH_SIZE:\n        img_batch = generator.next()\n\n    assert len(img_batch) == BATCH_SIZE\n\n    return img_batch\n```\n\n\n```python\nimport matplotlib.pyplot as plt\nimarr = get_image_batch(real_generator)[0, :, :, 0]\nplt.imshow(imarr)\n```\n\n\n\n\n    \u003cmatplotlib.image.AxesImage at 0x7f160fda74d0\u003e\n\n\n\n\n![png](imgs/output_25_1.png)\n\n\n\n```python\nimarr = get_image_batch(synth_generator())[0, :, :, 0]\nprint imarr.shape\nplt.imshow(imarr)\n```\n\n    (50, 100)\n\n\n\n\n\n    \u003cmatplotlib.image.AxesImage at 0x7f160fdd4390\u003e\n\n\n\n\n![png](imgs/output_26_2.png)\n\n\n# What happened next?\n\nPlug all the data in an MNIST-like classifier and call it a day. Unfortunately, it's not that simple.\n\nI actually spent a long time fine-tuning the network but accuracy plateued around 55% sampled. The passing requirement is 10000 out of 15000 submitted or 90% accuracy or 66% per char. I was facing a dilemma: tune the model even further or manually label x amount of data: \n```\n0.55 * (15000-x) + x = 10000\n                   x = 3888\n```\n\nObviously I am not going to label 4000 captchas and break my neck in the process.\n\nMeanwhile, there happened a burnt out guy who decided to label all 10000 captchas. This dilligent dude was 2000 in. I asked if he is willing to collaborate on a solution. It's almost like he didn't want to label captchas anymore. He agreed immediately.\n\nUsing the same model, accuracy immediately shot up to 95% and we both qualified for HackMIT.\n\n/aside\n\nAfter the contest, I perfected the model and got 95% without labelling a single image. Here is the model for SimGAN:\n\n![SimGAN](http://www.fudzilla.com/images/stories/2016/December/apple-simgan-generative-adversarial-networks.jpg)\n\n# Model Definition\n\nThere are three components to the network:\n\n### Refiner\n\nThe refiner network, Rθ, is a residual network (ResNet). It modifies the synthetic image on a pixel level, rather than holistically modifying the image content, preserving the global structure and annotations.\n\n### Discriminator\n\nThe discriminator network Dφ, is a simple ConvNet that contains 5 conv layers and 2 max-pooling layers. It's abinary classifier that outputs whether a captcha is synthesized or real.\n\n### Combined\n\nPipe the refined image into discriminator.\n\n\n```python\ndef refiner_network(input_image_tensor):\n    \"\"\"\n    :param input_image_tensor: Input tensor that corresponds to a synthetic image.\n    :return: Output tensor that corresponds to a refined synthetic image.\n    \"\"\"\n    def resnet_block(input_features, nb_features=64, nb_kernel_rows=3, nb_kernel_cols=3):\n        \"\"\"\n        A ResNet block with two `nb_kernel_rows` x `nb_kernel_cols` convolutional layers,\n        each with `nb_features` feature maps.\n        See Figure 6 in https://arxiv.org/pdf/1612.07828v1.pdf.\n        :param input_features: Input tensor to ResNet block.\n        :return: Output tensor from ResNet block.\n        \"\"\"\n        y = layers.Convolution2D(nb_features, nb_kernel_rows, nb_kernel_cols, border_mode='same')(input_features)\n        y = layers.Activation('relu')(y)\n        y = layers.Convolution2D(nb_features, nb_kernel_rows, nb_kernel_cols, border_mode='same')(y)\n\n        y = layers.merge([input_features, y], mode='sum')\n        return layers.Activation('relu')(y)\n\n    # an input image of size w × h is convolved with 3 × 3 filters that output 64 feature maps\n    x = layers.Convolution2D(64, 3, 3, border_mode='same', activation='relu')(input_image_tensor)\n\n    # the output is passed through 4 ResNet blocks\n    for _ in range(4):\n        x = resnet_block(x)\n\n    # the output of the last ResNet block is passed to a 1 × 1 convolutional layer producing 1 feature map\n    # corresponding to the refined synthetic image\n    return layers.Convolution2D(1, 1, 1, border_mode='same', activation='tanh')(x)\n\ndef discriminator_network(input_image_tensor):\n    \"\"\"\n    :param input_image_tensor: Input tensor corresponding to an image, either real or refined.\n    :return: Output tensor that corresponds to the probability of whether an image is real or refined.\n    \"\"\"\n    x = layers.Convolution2D(96, 3, 3, border_mode='same', subsample=(2, 2), activation='relu')(input_image_tensor)\n    x = layers.Convolution2D(64, 3, 3, border_mode='same', subsample=(2, 2), activation='relu')(x)\n    x = layers.MaxPooling2D(pool_size=(3, 3), border_mode='same', strides=(1, 1))(x)\n    x = layers.Convolution2D(32, 3, 3, border_mode='same', subsample=(1, 1), activation='relu')(x)\n    x = layers.Convolution2D(32, 1, 1, border_mode='same', subsample=(1, 1), activation='relu')(x)\n    x = layers.Convolution2D(2, 1, 1, border_mode='same', subsample=(1, 1), activation='relu')(x)\n\n    # here one feature map corresponds to `is_real` and the other to `is_refined`,\n    # and the custom loss function is then `tf.nn.sparse_softmax_cross_entropy_with_logits`\n    return layers.Reshape((-1, 2))(x)\n\n# Refiner\nsynthetic_image_tensor = layers.Input(shape=(HEIGHT, WIDTH, 1))\nrefined_image_tensor = refiner_network(synthetic_image_tensor)\nrefiner_model = models.Model(input=synthetic_image_tensor, output=refined_image_tensor, name='refiner')\n\n# Discriminator\nrefined_or_real_image_tensor = layers.Input(shape=(HEIGHT, WIDTH, 1))\ndiscriminator_output = discriminator_network(refined_or_real_image_tensor)\ndiscriminator_model = models.Model(input=refined_or_real_image_tensor, output=discriminator_output,\n                                   name='discriminator')\n\n# Combined\nrefiner_model_output = refiner_model(synthetic_image_tensor)\ncombined_output = discriminator_model(refiner_model_output)\ncombined_model = models.Model(input=synthetic_image_tensor, output=[refiner_model_output, combined_output],\n                              name='combined')\n\ndef self_regularization_loss(y_true, y_pred):\n    delta = 0.0001  # FIXME: need to figure out an appropriate value for this\n    return tf.multiply(delta, tf.reduce_sum(tf.abs(y_pred - y_true)))\n\n# define custom local adversarial loss (softmax for each image section) for the discriminator\n# the adversarial loss function is the sum of the cross-entropy losses over the local patches\ndef local_adversarial_loss(y_true, y_pred):\n    # y_true and y_pred have shape (batch_size, # of local patches, 2), but really we just want to average over\n    # the local patches and batch size so we can reshape to (batch_size * # of local patches, 2)\n    y_true = tf.reshape(y_true, (-1, 2))\n    y_pred = tf.reshape(y_pred, (-1, 2))\n    loss = tf.nn.softmax_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n\n    return tf.reduce_mean(loss)\n\n\n# compile models\nBATCH_SIZE = 512\nsgd = optimizers.RMSprop()\n\nrefiner_model.compile(optimizer=sgd, loss=self_regularization_loss)\ndiscriminator_model.compile(optimizer=sgd, loss=local_adversarial_loss)\ndiscriminator_model.trainable = False\ncombined_model.compile(optimizer=sgd, loss=[self_regularization_loss, local_adversarial_loss])\n```\n\n# Pre-training\n\nIt is not necessary to pre-train GANs but it seems pretraining makes GANs converge faster. Here we pre-train both models. For the refiner, we train by supplying the identity. For the discriminator, we train with the correct real, synth labeled pairs.\n\n\n```python\n# the target labels for the cross-entropy loss layer are 0 for every yj (real) and 1 for every xi (refined)\n\ny_real = np.array([[[1.0, 0.0]] * discriminator_model.output_shape[1]] * BATCH_SIZE)\ny_refined = np.array([[[0.0, 1.0]] * discriminator_model.output_shape[1]] * BATCH_SIZE)\nassert y_real.shape == (BATCH_SIZE, discriminator_model.output_shape[1], 2)\n```\n\n\n```python\nLOG_INTERVAL = 10\nMODEL_DIR = \"./model/\"\nprint('pre-training the refiner network...')\ngen_loss = np.zeros(shape=len(refiner_model.metrics_names))\n\nfor i in range(100):\n    synthetic_image_batch = get_image_batch(synth_generator())\n    gen_loss = np.add(refiner_model.train_on_batch(synthetic_image_batch, synthetic_image_batch), gen_loss)\n\n    # log every `log_interval` steps\n    if not i % LOG_INTERVAL:\n        print('Refiner model self regularization loss: {}.'.format(gen_loss / LOG_INTERVAL))\n        gen_loss = np.zeros(shape=len(refiner_model.metrics_names))\n\nrefiner_model.save(os.path.join(MODEL_DIR, 'refiner_model_pre_trained.h5'))\n\n```\n\n    pre-training the refiner network...\n    Saving batch of refined images during pre-training at step: 0.\n    Refiner model self regularization loss: [ 0.05277019].\n    Saving batch of refined images during pre-training at step: 10.\n    Refiner model self regularization loss: [ 4.2269813].\n    Saving batch of refined images during pre-training at step: 20.\n    Refiner model self regularization loss: [ 0.76108101].\n    Saving batch of refined images during pre-training at step: 30.\n    Refiner model self regularization loss: [ 0.28633648].\n    Saving batch of refined images during pre-training at step: 40.\n    Refiner model self regularization loss: [ 0.19448772].\n    Saving batch of refined images during pre-training at step: 50.\n    Refiner model self regularization loss: [ 0.16131182].\n    Saving batch of refined images during pre-training at step: 60.\n    Refiner model self regularization loss: [ 0.11931724].\n    Saving batch of refined images during pre-training at step: 70.\n    Refiner model self regularization loss: [ 0.11075923].\n    Saving batch of refined images during pre-training at step: 80.\n    Refiner model self regularization loss: [ 0.10888441].\n    Saving batch of refined images during pre-training at step: 90.\n    Refiner model self regularization loss: [ 0.10765313].\n\n\n\n```python\nfrom tqdm import tqdm\nprint('pre-training the discriminator network...')\ndisc_loss = np.zeros(shape=len(discriminator_model.metrics_names))\n\nfor _ in tqdm(range(100)):\n    real_image_batch = get_image_batch(real_generator)\n    disc_loss = np.add(discriminator_model.train_on_batch(real_image_batch, y_real), disc_loss)\n\n    synthetic_image_batch = get_image_batch(synth_generator())\n    refined_image_batch = refiner_model.predict_on_batch(synthetic_image_batch)\n    disc_loss = np.add(discriminator_model.train_on_batch(refined_image_batch, y_refined), disc_loss)\n\ndiscriminator_model.save(os.path.join(MODEL_DIR, 'discriminator_model_pre_trained.h5'))\n\n# hard-coded for now\nprint('Discriminator model loss: {}.'.format(disc_loss / (100 * 2)))\n```\n\n    pre-training the discriminator network...\n    Discriminator model loss: [ 0.04783788].\n\n\n# Training\n\nThis is the most important training step in which we refine a synthesized captcha, then pass it through the discriminator and backprop gradients.\n\n\n```python\nfrom image_history_buffer import ImageHistoryBuffer\n\n\nk_d = 1  # number of discriminator updates per step\nk_g = 2  # number of generative network updates per step\nnb_steps = 1000\n\n# TODO: what is an appropriate size for the image history buffer?\nimage_history_buffer = ImageHistoryBuffer((0, HEIGHT, WIDTH, 1), BATCH_SIZE * 100, BATCH_SIZE)\n\ncombined_loss = np.zeros(shape=len(combined_model.metrics_names))\ndisc_loss_real = np.zeros(shape=len(discriminator_model.metrics_names))\ndisc_loss_refined = np.zeros(shape=len(discriminator_model.metrics_names))\n\n# see Algorithm 1 in https://arxiv.org/pdf/1612.07828v1.pdf\nfor i in range(nb_steps):\n    print('Step: {} of {}.'.format(i, nb_steps))\n\n    # train the refiner\n    for _ in range(k_g * 2):\n        # sample a mini-batch of synthetic images\n        synthetic_image_batch = get_image_batch(synth_generator())\n\n        # update θ by taking an SGD step on mini-batch loss LR(θ)\n        combined_loss = np.add(combined_model.train_on_batch(synthetic_image_batch,\n                                                             [synthetic_image_batch, y_real]), combined_loss)\n\n    for _ in range(k_d):\n        # sample a mini-batch of synthetic and real images\n        synthetic_image_batch = get_image_batch(synth_generator())\n        real_image_batch = get_image_batch(real_generator)\n\n        # refine the synthetic images w/ the current refiner\n        refined_image_batch = refiner_model.predict_on_batch(synthetic_image_batch)\n\n        # use a history of refined images\n        half_batch_from_image_history = image_history_buffer.get_from_image_history_buffer()\n        image_history_buffer.add_to_image_history_buffer(refined_image_batch)\n\n        if len(half_batch_from_image_history):\n            refined_image_batch[:batch_size // 2] = half_batch_from_image_history\n\n        # update φ by taking an SGD step on mini-batch loss LD(φ)\n        disc_loss_real = np.add(discriminator_model.train_on_batch(real_image_batch, y_real), disc_loss_real)\n        disc_loss_refined = np.add(discriminator_model.train_on_batch(refined_image_batch, y_refined),\n                                   disc_loss_refined)\n\n    if not i % LOG_INTERVAL:\n        # log loss summary\n        print('Refiner model loss: {}.'.format(combined_loss / (LOG_INTERVAL * k_g * 2)))\n        print('Discriminator model loss real: {}.'.format(disc_loss_real / (LOG_INTERVAL * k_d * 2)))\n        print('Discriminator model loss refined: {}.'.format(disc_loss_refined / (LOG_INTERVAL * k_d * 2)))\n\n        combined_loss = np.zeros(shape=len(combined_model.metrics_names))\n        disc_loss_real = np.zeros(shape=len(discriminator_model.metrics_names))\n        disc_loss_refined = np.zeros(shape=len(discriminator_model.metrics_names))\n\n        # save model checkpoints\n        model_checkpoint_base_name = os.path.join(MODEL_DIR, '{}_model_step_{}.h5')\n        refiner_model.save(model_checkpoint_base_name.format('refiner', i))\n        discriminator_model.save(model_checkpoint_base_name.format('discriminator', i))\n\n```\n\n    Step: 0 of 1000.\n    Saving batch of refined images at adversarial step: 0.\n    Refiner model loss: [ 2.46834831  0.01272553  2.45562277].\n    Discriminator model loss real: [  2.27849432e-07].\n    Discriminator model loss refined: [  1.63936726e-05].\n    Step: 1 of 1000.\n    Step: 2 of 1000.\n    Step: 3 of 1000.\n    Step: 4 of 1000.\n    Step: 5 of 1000.\n    Step: 6 of 1000.\n    Step: 7 of 1000.\n    Step: 8 of 1000.\n    Step: 9 of 1000.\n    Step: 10 of 1000.\n    Saving batch of refined images at adversarial step: 10.\n    Refiner model loss: [ 27.00968537   0.11238954  26.8972959 ].\n    Discriminator model loss real: [  1.26835085e-10].\n    Discriminator model loss refined: [  4.44882481e-08].\n    Step: 11 of 1000.\n    Step: 12 of 1000.\n    Step: 13 of 1000.\n    Step: 14 of 1000.\n    Step: 15 of 1000.\n    Step: 16 of 1000.\n    Step: 17 of 1000.\n    Step: 18 of 1000.\n    Step: 19 of 1000.\n    Step: 20 of 1000.\n    Saving batch of refined images at adversarial step: 20.\n    Refiner model loss: [ 26.89902883   0.10987803  26.78915081].\n    Discriminator model loss real: [  1.48619811e-07].\n    Discriminator model loss refined: [  4.60907181e-08].\n    Step: 21 of 1000.\n    Step: 22 of 1000.\n    Step: 23 of 1000.\n    Step: 24 of 1000.\n    Step: 25 of 1000.\n    Step: 26 of 1000.\n    Step: 27 of 1000.\n    Step: 28 of 1000.\n    Step: 29 of 1000.\n    Step: 30 of 1000.\n    Saving batch of refined images at adversarial step: 30.\n    Refiner model loss: [ 25.93090506   0.10890296  25.82200208].\n    Discriminator model loss real: [  3.96611703e-09].\n    Discriminator model loss refined: [  5.07067440e-08].\n    Step: 31 of 1000.\n    Step: 32 of 1000.\n    Step: 33 of 1000.\n    Step: 34 of 1000.\n    Step: 35 of 1000.\n    Step: 36 of 1000.\n    Step: 37 of 1000.\n    Step: 38 of 1000.\n    Step: 39 of 1000.\n    Step: 40 of 1000.\n    Saving batch of refined images at adversarial step: 40.\n    Refiner model loss: [ 28.67232819   2.33041485  26.34191332].\n\n\n# Results of SimGAN:\n\nAs you can see below, we no longer have the cookie-cutter fonts. There are quite a few artifacts that did not exist before refinement. The edges are blurred and noisy - which is *impossible* to simulate heuristically. And it is exactly these tiny things that renders MNIST-like convnet useless.\n\nNow the refined results are basically the original captchas.\n\n\n```python\nsynthetic_image_batch = get_image_batch(synth_generator())\narr = refiner_model.predict_on_batch(synthetic_image_batch)\n```\n\n\n```python\nplt.imshow(arr[200, :, :, 0])\n```\n\n\n\n\n    \u003cmatplotlib.image.AxesImage at 0x7f161417fa90\u003e\n\n\n\n\n![png](imgs/output_38_1.png)\n\n\n\n```python\nplt.imshow(get_image_batch(real_generator)[2,:,:,0])\n```\n\n\n\n\n    \u003cmatplotlib.image.AxesImage at 0x7f1614381690\u003e\n\n\n\n\n![png](imgs/output_39_1.png)\n\n\n# MNIST for Captcha\n\nNow we finish the puzzle by building an MNIST like convnet to predict captcha labels.\n\n\n```python\nn_class = len(alphanumeric)\n\ndef mnist_raw_generator(batch_size=128):\n    X = np.zeros((batch_size, HEIGHT, WIDTH, 1), dtype=np.uint8)\n    y = [np.zeros((batch_size, n_class), dtype=np.uint8) for _ in range(4)] # 4 chars\n    while True:\n        for i in range(batch_size):\n            im, random_str = gen_one()\n            X[i] = im\n            for j, ch in enumerate(random_str):\n                y[j][i, :] = 0\n                y[j][i, alphanumeric.find(ch)] = 1\n        yield np.array(X), y\n\ndef mnist_generator(batch_size=128):\n    X = np.zeros((batch_size, HEIGHT, WIDTH, 1), dtype=np.uint8)\n    y = [np.zeros((batch_size, n_class), dtype=np.uint8) for _ in range(4)] # 4 chars\n    while True:\n        for i in range(batch_size):\n            im, random_str = gen_one()\n            X[i] = im\n            for j, ch in enumerate(random_str):\n                y[j][i, :] = 0\n                y[j][i, alphanumeric.find(ch)] = 1\n        yield refiner_model.predict(np.array(X)), y\n\nmg = mnist_generator().next()\n\n# plt.imshow(mg[0][0,:,:,0]) # sanity check\n```\n\n\n```python\nfrom keras.layers import *\n\ninput_tensor = Input((HEIGHT, WIDTH, 1))\nx = input_tensor\nx = Conv2D(32, kernel_size=(3, 3),\n                 activation='relu')(x)\nfor _ in range(4):\n    x = Conv2D(128, (3, 3), activation='relu')(x)\n    x = MaxPooling2D(pool_size=(2, 2))(x)\nx = Dropout(0.25)(x)\nx = Flatten()(x)\nx = Dense(128, activation='relu')(x)\nx = Dropout(0.5)(x)\nx = [Dense(n_class, activation='softmax', name='c%d'%(i+1))(x) for i in range(4)]\n\nmodel = models.Model(inputs=input_tensor, outputs=x)\nmodel.compile(loss='categorical_crossentropy',\n              optimizer='rmsprop',\n              metrics=['accuracy'])\n```\n\n\n```python\nfrom keras.callbacks import History\nhistory = History()\nmodel.fit_generator(mnist_generator(), steps_per_epoch=1000, epochs=20, callbacks=[history])\n```\n\n    Epoch 1/20\n     341/1000 [=========\u003e....................] - ETA: 376s - loss: 2.7648 - c1_loss: 0.6493 - c2_loss: 0.6757 - c3_loss: 0.6681 - c4_loss: 0.7717 - c1_acc: 0.8199 - c2_acc: 0.8185 - c3_acc: 0.8197 - c4_acc: 0.7820\n\n\nObviously you will need to keep training as the per-character accuracy is only 80%\n\n# Let's test the trained model\n\n## Synthetic\n\n\n```python\ndef decode(y):\n    y = np.argmax(np.array(y), axis=2)[:,0]\n    return ''.join([alphanumeric[x] for x in y])\n\nX, y = next(mnist_generator(1))\nplt.title('real: %s\\npred:%s'%(decode(y), decode(y_pred)))\nplt.imshow(X[0, :, :, 0], cmap='gray')\nplt.axis('off')\n```\n\n\n    (-0.5, 99.5, 49.5, -0.5)\n\n\n\n\n![png](imgs/output_45_2.png)\n\n\n## Real\n\n\n```python\n\nX = next(real_generator)\nX = refiner_model.predict(X)\ny_pred = model.predict(X)\nplt.title('pred:%s'%(decode(y_pred)))\nplt.imshow(X[0,:,:,0], cmap='gray')\nplt.axis('off')\n```\n\n\n\n\n    (-0.5, 99.5, 49.5, -0.5)\n\n\n\n\n![png](imgs/output_47_1.png)\n\n\n\n```python\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0b01%2FSimGAN-Captcha","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F0b01%2FSimGAN-Captcha","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0b01%2FSimGAN-Captcha/lists"}