AI writing AI code🤐

It is 2021. And we have #AI writing #AI code. 🤪 It is quite interesting, but also can be quite boring once you get beyond the initial technology, and just think of it as one of the tools in your arsenal. And getting to that point is a good think.

As part of a think at work I recently started playing with GitHub Copilot, which is using GPT3 to be your pair programmer — helping write code. GPT3 has multiple models (called engines), and Copilot uses one of these family of engines called Codex. Codex is a derivative of the base GPT3 engine that is trained on billions of lines of code.

Using Copilot is quite simple; you install the Github Copilot extension, and it shows up in your IDE (VSCode in my example). We need to make sure we decompose the problem we are trying to solve – we should not think of this as helping write the complete program or all parts; but as it can help with different functions and pieces of code. To do this, we need to tell it what we are trying to do – these are done via prompts (code comments). For GPT models, prompt engineering is quite critical, and would be worth getting to details and understanding.

Starting simple, I create an empty python file and entered a prompt that outlines what I want to try and do. In this case as you can see in the screenshot below – I want to load an image from a file, and using our Vision Cognitive Services, run an image analysis, and auto-generate a caption for that image.

I started typing the definition of a function, and Copilot (via the add-in) understands the prompt I outlined, and the context of the code on what I am doing. Remember Codex builds on the base GPT3 and does have all that NLU capability.

Taking all of this in, it suggests completing the function for me. In terms of using this as an end-user (i.e. the developer) – the suggested code shows up as auto-complete and you can see it in the grey color. If I like that suggestion, I press tab and have it added to the file.

In this case you can see how it is reading the file from disk, calling a function called get_caption() and printing the caption to the stdout (console in this example).

There is also an option to cycle through different suggestions and then pick another one as shown in the screenshot below.

This variant of the suggested code is creating a function called image_caption() which takes the path to the image file to load. This also expects other required things for the Vision cognitive service to work – such as the subscription key to authenticate, the API end-point details to call, etc.

Typically, Copilot can synthesize up to 10 code options (Copilot calls these as Solutions), that one can cycle through and see if there is a better variant for the task at hand. The screenshot below shows this experience in VS Code.

The ask whilst simple, still involves a bit of code which needs to be written – reading from file, setting up the subscription details, and wiring that up to call this etc. And it is in these cases really where Copilot shines – it is your copilot picking up the ‘gunk work’ – freeing up your bandwidth, and cognition capacity on the more interesting and higher order bit of code and value to your business.

To get a flavor of our AI writing AI code, below is the full set of the nine suggestions the Copilot came back for what I was trying to do.

Synthesizing 9/10 solutions


def generate_caption(image_path):
    from import ComputerVisionClient
    from msrest.authentication import CognitiveServicesCredentials
    from import TextOperationStatusCodes
    from import TextRecognitionMode
    from import VisualFeatureTypes
    from msrest.exceptions import CognitiveServicesError
    from os.path import exists
    from os import remove
    from os import makedirs
    from os import listdir
    from os.path import isfile, join
    from time import sleep
    from sys import exit
    from random import randint
    from datetime import datetime

    # set up the connection to the cognitive services account
    subscription_key = "YOUR KEY HERE"
    endpoint = "https://YOUR ENDPOINT HERE"
    group_id = "YOUR GROUP ID HERE"
    client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))

    # set up the directory where the images will be downloaded
    download_dir = "./downloads/"
    if not exists(download_dir):

    # set up the directory where the images will be saved
    save_dir = "./images/"
    if not exists(save_dir):

    # set up the directory where the images will be saved
    caption_dir = "./captions/"
    if not exists(caption_dir):

    # get a list of the files in the download directory
    files = [f for f in listdir(download_dir) if isfile(join(download_dir, f))]
    if len(files) == 0:
        print("No files found in the download directory")

    # get the first file in the list
    file_name = files[0]


def generate_caption(image_path):
    from os import environ
    from import ComputerVisionClient
    from msrest.authentication import CognitiveServicesCredentials
    from import TextOperationStatusCodes
    from import TextRecognitionMode
    from import VisualFeatureTypes
    from msrest.exceptions import CognitiveServicesError

    # Add your Computer Vision subscription key to your environment variables.
        subscription_key = environ['COMPUTER_VISION_SUBSCRIPTION_KEY']
        print("\nSet the COMPUTER_VISION_SUBSCRIPTION_KEY environment variable.\n**Restart your shell or IDE for changes to take effect.**")

    # Add your Computer Vision endpoint to your environment variables.
    if 'COMPUTER_VISION_ENDPOINT' in environ:
        endpoint = environ['COMPUTER_VISION_ENDPOINT']
        print("\nSet the COMPUTER_VISION_ENDPOINT environment variable.\n**Restart your shell or IDE for changes to take effect.**")

    # Create an authenticated Computer Vision client.
    client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))

    # Open the local image file.
    with open(image_path, 'rb') as image_file:
        # Read the image into a byte array.
        image_data =

    # Set the features to return.
    features = [VisualFeatureTypes.categories, VisualFeatureTypes.description, VisualFeatureTypes.tags]

    # Analyze the local image.
        result = client.analyze_image(image=image_data, visual_features=features)


def generate_caption(image_file, subscription_key):
    from import ComputerVisionClient
    from msrest.authentication import CognitiveServicesCredentials
    from PIL import Image
    from io import BytesIO

    client = ComputerVisionClient(ComputerVisionClient.SUBSCRIPTION_KEY_ENDPOINT,

    image_stream = open(image_file, 'rb')
    image =
    image_bytes = BytesIO(image)

    analysis = client.analyze_image(image_bytes, ComputerVisionClient.DESCRIPTION)
    return analysis.description.captions[0].text


def generate_caption(image_filename):
    import os
    import json
    from import ComputerVisionClient
    from msrest.authentication import CognitiveServicesCredentials
    from PIL import Image
    from io import BytesIO
    import sys
    import requests

    # Set the environment variables for your subscription key and region
    # For example, for US West
    # "COMPUTER_VISION_SUBSCRIPTION_KEY=<subscription key>"

    # For example, for China East
    # "COMPUTER_VISION_SUBSCRIPTION_KEY=<subscription key>"

    # Set your subscription key and region
    subscription_key = os.environ['COMPUTER_VISION_SUBSCRIPTION_KEY']
    endpoint = os.environ['COMPUTER_VISION_ENDPOINT']

    # Set your image path
    image_path = image_filename

    # Set your region
    region = "eastus"

    # Create an instance of the Computer Vision client
    computer_vision_client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))

    # Load the image from disk
    with open(image_path, 'rb') as image_file:
        image_content =

    # Add the image to the list of images to analyze
    image_list = [image_content]

    # Make the analysis asynchronous
    analyze_result = computer_vision_client.analyze_image_in_stream(image_list, visual_features=['Categories', 'Description', 'Color'])

    # Get the analysis result
    analysis = analyze_result.result()

    # Display the image captions
    print("\nImage captions:")
    for caption in analysis.description.


def generate_caption(image_path):
    # load the image
    image = open(image_path, "rb").read()
    # encode the image into base64
    image = base64.b64encode(image).decode("utf-8")
    # create a dict with the image
    body = {"image": image}
    # create a dict with the subscription key
    headers = {"Ocp-Apim-Subscription-Key": "0d5a5f7f1e534d05a8e8d6f9e6e9c6d3"}
    # create a url for the api
    url = ",Tags"
    # send the request
    response =, headers=headers, json=body)
    # get the response
    # get the caption
    caption = response.json()["description"]["captions"][0]["text"]
    # return the caption
    return caption


def generate_caption(image_path):
    from os import environ
    from os.path import join
    from urllib.parse import urlparse
    from urllib.parse import parse_qs
    from http.client import HTTPSConnection
    from json import loads
    from base64 import b64encode

    # Load the image into a byte array
    image_data = open(image_path, "rb").read()
    image_data = b64encode(image_data).decode('utf-8')

    # Read the subscription key from the environment variables
    subscription_key = environ["AZURE_SUBSCRIPTION_KEY"]

    # Define the vision API endpoint
    vision_base_url = ""

    # Set the vision API endpoint to analyze the image
    vision_analyze_url = vision_base_url + "analyze"

    # Set the parameters
    parameters = {'visualFeatures': 'Categories,Description,Color'}

    # Prepare the request
    headers = {'Ocp-Apim-Subscription-Key': subscription_key,
               'Content-Type': 'application/octet-stream'}
    params = parse_qs(urlparse(vision_analyze_url).query)
    params['visualFeatures'] = 'Categories,Description,Color'
    params['language'] = 'en'

    # Send the request
    conn = HTTPSConnection('')
    conn.request("POST", vision_analyze_url, image_data, headers)
    response = conn.getresponse()
    data =

    # Load the JSON response into a Python dictionary
    parsed = loads(data)

    return parsed


def generate_caption(image_path):
    Generate a caption for an image.
    :param image_path: path to an image
    :return: the caption

    # Load the image
    from os.path import isfile
    from PIL import Image
    import requests

    if not isfile(image_path):
        raise FileNotFoundError(image_path)

    image =

    # Encode the image
    image_data = open(image_path, "rb").read()

    # Computer Vision parameters
    params = {'visualFeatures': 'Categories,Description,Color'}

    # Computer Vision API URL
    url = ""

    # Connect to Computer Vision
    headers = {'Ocp-Apim-Subscription-Key': 'd3f3e1b8e7e14d0b9d7d3f8c8f7e0b6f', 'Content-Type': 'application/octet-stream'}
    response =, headers=headers, params=params, data=image_data)

    # Load the response as JSON
    analysis = response.json()

    # Extract the word tags
    categories = [(category['name'], category['score'])
                  for category in analysis['categories']]
    for category in categories:

    # Extract the description
    descriptions = [(description['text'], description['score'])
                    for description in analysis['description']['captions']]
    for description in descriptions:

    # Display the color
        dominant_color = analysis['color']['dominantColorBackground']
        dominant_color = 'N/A'
    print('Dominant color: {}'.format(dominant_color))

    # Display the caption
    caption = '{}'.format(


def generate_caption(filename):
    import os
    import sys
    import json
    import requests
    import cognitive_services_keys
    import cognitive_services_key_names

    # create the header for the request
    headers = {'Ocp-Apim-Subscription-Key': cognitive_services_keys.get_key(cognitive_services_key_names.BING_KEY)}

    # the URL for the bing endpoint
    bing_url = ''

    # the parameters for the request
    params = {'q': 'image', 'count': 1, 'offset': 0, 'mkt': 'en-us', 'safeSearch': 'Moderate'}

    # read the contents of the file
    with open(filename, 'rb') as f:
        data =

    # build the request
    request =, headers=headers, params=params, data=data)

    # check for errors
    if request.status_code != 200:
        print('Error code: {}'.format(request.status_code))

    # process the response
    response = json.loads(request.text)
    if 'value' in response:
        caption = response['value'][0]['name']
        print('The caption is: {}'.format(caption))
        print('No caption found.')


def generate_caption(image_path):
    import os
    import json
    import requests

    subscription_key = "e9e9c8a8d1d9400e8d0b7f2d8f8e7c6e"

    assert subscription_key

    vision_base_url = ""

    analyze_url = vision_base_url + "analyze"

    image_data = open(image_path, "rb").read()

    headers    = {'Ocp-Apim-Subscription-Key': subscription_key,
                  'Content-Type': 'application/octet-stream'}
    params     = {'visualFeatures': 'Categories,Description,Color'}
    response =
        analyze_url, headers=headers, params=params, data=image_data)

    analysis = response.json()
    image_caption = analysis["description"]["captions"][0]["text"].capitalize()
    return image_caption